[lookit-research] stimulus sound not coming through on videos

Sat Mar 18 15:02:35 EDT 2023

Dear all -

Thanks so much to David for your question, and for the responses so far!  I
am going to recommend that people continue this conversation on our Slack
channel <http://lookit-mit.slack.com>, since there are some related
conversations (current & previous) that may be helpful to look through.
Just to make sure this reaches people who started reading this email
thread, here are some further thoughts on this issue (I will also post
these to Slack!):

Lisa, you've already said most of what I was planning to say regarding
timing & sound, so thank you!

For the specific recent "disappearing sound" issue, especially where
speech/sounds from the participant can be heard, but stimulus noises played
out of the participants cannot: This is a pattern that is more likely to be
related to something on the user's computer, and there haven't been any
recent codebase changes that are likely to explain them. There was actually
just a thread this week from Elizabeth Simmons on Slack - in her case, some
testing revealed that a change observed in the last several weeks may be
specifically connected to Mac computers.

More generally, especially if you care about precision of measurement under
a second, I would strongly recommend two general courses of action:

- First, make sure you're aware of *at least *two sources of information
that you can use to verify the timing of each individual session. Which of
these you choose will heavily depend on the design of your study and which
kinds of information are most critical for your analyses - for instance,
whether you are comparing times between two conditions to detect a
difference, versus measuring against an expected absolute timing.  Some of
these cues could include:

* A luminance or sound marker at the beginning of a trial (with the caveats
already mentioned above)
* The Lookit event time stamps
* The total expected versus actual length of a trial video (and considering
whether trial-by-trial recordings or a single session recording are the
best fit for your study)
* Eliciting a clap or similar from your participant/parent to help you
detect synchrony of the participant's audio & video

- Second, plan to conduct a few pilot sessions with actual participants and
then test your entire analysis pipeline before re-starting data collection
to ensure that you can verify these timings in the way you expect.  As
mentioned this won't catch everything, since some kinds of timing variation
are rarer, but this will also help you to establish with real data *what
kind of variation* your design can tolerate and when you will plan to
exclude a session from analysis. Can your design tolerate 10ms of variation
from the intended timing? 50ms? 500ms?

Finally, I want to assure the research community that this is an issue
we're keenly aware of and want to continue to improve so that Lookit can be
used for as many kinds of research as possible! In particular we are
exploring some ways to make it easier to get *more information* on event
timings in the future, even if we can't eliminate the parts of the problem
that come from our individual computer/internet settings.

Thanks again for this discussion, and for using Lookit!

- Melissa

On Sat, Mar 18, 2023 at 1:50 PM Lewkowicz, David <david.lewkowicz at yale.edu>
wrote:

> Hi Lisa,
>
>
>
> Thanks very much for sharing your experience with Lookit and for pointing
> out the limitations of what we can expect in terms of recorded sounds and
> timing.  The one good piece of news in all of this is that if you happen to
> pick visual stimuli that produce a relatively large change in the overall
> luminance of the scene, and if the camera recording the participant picks
> up that change, then you can use this as your cue that the trial has
> started. Of course, if your stimuli do not increase overall luminance then
> you’re back to the problem of figuring out when the trial starts and there,
> based on what you’re reporting, there isn’t a simple solution here. Oh
> well, one must be realistic about the limits of online testing.
>
>
>
> And, I suspect that the intractability of the timing issue is probably due
> to the variation in the type of connection that the testing computer has to
> the internet (e.g., wi-fi vs. ethernet) as well as the download speeds of
> the testing computer (which, of course, is partly determined by the ISP and
> the type of connection between the computer and the modem/router in the
> house).
>
>
>
> Best regards,
>
> David
>
>
>
> *From: *Lisa Oakes <lmoakes at ucdavis.edu>
> *Date: *Saturday, March 18, 2023 at 1:16 PM
> *To: *"Lewkowicz, David" <david.lewkowicz at yale.edu>
> *Cc: *Melissa Kline <mekline at mit.edu>, "lookit-research at mit.edu" <
> lookit-research at mit.edu>, "Aslin, Richard" <richard.aslin at yale.edu>
> *Subject: *Re: [lookit-research] stimulus sound not coming through on
> videos
>
>
>
> Hi David (and everyone):
>
>
>
> Online testing has such promise and yet has new frustrations.
>
>
>
> First, what you are experiencing is not a lookit problem.  Computers
> suppress what it interprets as background noise when recording. We have
> found that we hear the sound pretty reliably for some experiments but not
> for others, and sometimes we hear some sounds but not others. I suspect it
> depends on the nature of the sound and the volume. Bottom line: it doesn’t
> matter how you are recording, if you collect data online you can’t depend
> on hearing the sound of the stimuli.
>
> Moreover, even if you do hear sound, it is likely out of sync with the
> video. You may notice that when you see a child speak what you hear is out
> of sync with the video. Again, this is variable and there is no easy way to
> correct it. This is a known problem with video recording and with Lookit.
> You can find more info here:
> https://lookit.readthedocs.io/en/develop/researchers-lag-issues.html
> <https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Flookit.readthedocs.io%2Fen%2Fdevelop%2Fresearchers-lag-issues.html&data=05%7C01%7Cdavid.lewkowicz%40yale.edu%7C70d7851ac5dc41eb240a08db27d4790b%7Cdd8cbebb21394df8b4114e3e87abeb5c%7C0%7C0%7C638147565743157654%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=MiDTNZdQdqh4DV55IILN4R1btd3RJFMhOARUP5WKGQg%3D&reserved=0>
>
> A better solution is to use the timing info provided by lookit. However,
> this also is not exact. We (and others) have done extensive testing of the
> timing of stimulus presentations in lookit and have found that there is
> some variation in when stimuli start relative to when lookit says they
> start. I don’t think this has been solved and I don’t think it is known why
> it happens.
>
>
>
> We have tested hundreds of kids using online methods for the last 3 years
> and my recommendations are (1) don’t rely on sound for timing, and (2)
> online methods are best for stimuli that are static or if it doesn’t matter
> when the infant looks relative to what is happening in the stimulus.
>
>
>
> This probably isn’t the answer you wanted, but I hope it is helpful as you
> find a solution.
>
>
>
> Lisa
>
>
>
> Lisa M. Oakes
> Professor, Department of Psychology
> Faculty Researcher, Center for Mind and Brain
> University of California, Davis
> 267 Cousteau Place
> Davis, CA 95618
>
> phone:  530 754-4523 OR
>
> 530 754-8304
>
> Pronouns: she/her/hers
>
>
>
>
> On Mar 18, 2023, at 8:45 AM, Lewkowicz, David <david.lewkowicz at yale.edu>
> wrote:
>
>
>
> Dear Melissa,
>
>
>
> Recently, we have noticed that the participant videos recorded by Lookit
> seem to filter out the stimulus sounds that we normally present in our
> videos. We use these sounds to mark the beginning of a trial when we code
> the participant videos and, thus, their absence makes it impossible for us
> to reliably code the participant videos.  Are you aware of this and, if so,
> is there a fix for this problem?
>
>
>
> Thanks very much in advance for looking into this problem.
>
>
>
> Best regards,
>
> David & Dick
>
>
>
> __________________________________________
>
> David J. Lewkowicz, Ph.D.
>
> Senior Scientist
>
> Haskins Laboratories, New Haven, CT
>
> Yale Child Study Center
>
> Personal: https://haskinslabs.org/people/david-lewkowicz
> <https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fhaskinslabs.org%2Fpeople%2Fdavid-lewkowicz&data=05%7C01%7Cdavid.lewkowicz%40yale.edu%7C70d7851ac5dc41eb240a08db27d4790b%7Cdd8cbebb21394df8b4114e3e87abeb5c%7C0%7C0%7C638147565743157654%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=7FZIKL%2FYKMjMvxPvUWfBgaWMVeLcK15KBhmgRLpHrsY%3D&reserved=0>
>
> Lab: http://llamblab.haskins.yale.edu/
> <https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fllamblab.haskins.yale.edu%2F&data=05%7C01%7Cdavid.lewkowicz%40yale.edu%7C70d7851ac5dc41eb240a08db27d4790b%7Cdd8cbebb21394df8b4114e3e87abeb5c%7C0%7C0%7C638147565743157654%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=JOMur9zdTinqsmqpo3vcJ9Zh0N0IEj1ulApF8286GMc%3D&reserved=0>
>
>
>
>
>
> _______________________________________________
> lookit-research mailing list
> lookit-research at mit.edu
> https://mailman.mit.edu/mailman/listinfo/lookit-research
> <https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmailman.mit.edu%2Fmailman%2Flistinfo%2Flookit-research&data=05%7C01%7Cdavid.lewkowicz%40yale.edu%7C70d7851ac5dc41eb240a08db27d4790b%7Cdd8cbebb21394df8b4114e3e87abeb5c%7C0%7C0%7C638147565743157654%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=SVGsaYiANzE2bGmWyrkhdmz%2BF0GMFtoup7lmiIPT7cQ%3D&reserved=0>
>
>
>

-- 
Melissa Kline Struhl
Lookit Executive Director
Research Scientist, MIT Brain & Cognitive Sciences
mekline at mit.edu
https://lookit.mit.edu/ <http://lookit.mit.edu>
http://www.melissaklinestruhl.com/ <https://www.melissaklinestruhl.com>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.mit.edu/pipermail/lookit-research/attachments/20230318/14b010cc/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 13023 bytes
Desc: not available
URL: <http://mailman.mit.edu/pipermail/lookit-research/attachments/20230318/14b010cc/attachment-0001.png>