<div dir="ltr">Dear all - <div><br></div><div>Thanks so much to David for your question, and for the responses so far! I am going to recommend that people continue this conversation on our<a href="http://lookit-mit.slack.com"> Slack channel</a>, since there are some related conversations (current & previous) that may be helpful to look through. Just to make sure this reaches people who started reading this email thread, here are some further thoughts on this issue (I will also post these to Slack!):</div><div><br></div><div>Lisa, you've already said most of what I was planning to say regarding timing & sound, so thank you! </div><br class="gmail-Apple-interchange-newline">For the specific recent "disappearing sound" issue, especially where speech/sounds from the participant can be heard, but stimulus noises played out of the participants cannot: This is a pattern that is more likely to be related to something on the user's computer, and there haven't been any recent codebase changes that are likely to explain them. There was actually just a thread this week from Elizabeth Simmons on Slack - in her case, some testing revealed that a change observed in the last several weeks may be specifically connected to Mac computers.<div><div><br></div><div>More generally, especially if you care about precision of measurement under a second, I would strongly recommend two general courses of action: </div><div><br></div><div>- First, make sure you're aware of <i>at least </i>two sources of information that you can use to verify the timing of each individual session. Which of these you choose will heavily depend on the design of your study and which kinds of information are most critical for your analyses - for instance, whether you are comparing times between two conditions to detect a difference, versus measuring against an expected absolute timing. Some of these cues could include:</div><div><br></div><div>* A luminance or sound marker at the beginning of a trial (with the caveats already mentioned above)</div><div>* The Lookit event time stamps</div><div>* The total expected versus actual length of a trial video (and considering whether trial-by-trial recordings or a single session recording are the best fit for your study)</div><div>* Eliciting a clap or similar from your participant/parent to help you detect synchrony of the participant's audio & video</div><div><br></div><div>- Second, plan to conduct a few pilot sessions with actual participants and then test your entire analysis pipeline before re-starting data collection to ensure that you can verify these timings in the way you expect. As mentioned this won't catch everything, since some kinds of timing variation are rarer, but this will also help you to establish with real data <i>what kind of variation</i> your design can tolerate and when you will plan to exclude a session from analysis. Can your design tolerate 10ms of variation from the intended timing? 50ms? 500ms? </div><div> <br></div></div><div>Finally, I want to assure the research community that this is an issue we're keenly aware of and want to continue to improve so that Lookit can be used for as many kinds of research as possible! In particular we are exploring some ways to make it easier to get <i>more information</i> on event timings in the future, even if we can't eliminate the parts of the problem that come from our individual computer/internet settings. </div><div><br></div><div>Thanks again for this discussion, and for using Lookit!</div><div><br></div><div>- Melissa </div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sat, Mar 18, 2023 at 1:50 PM Lewkowicz, David <<a href="mailto:david.lewkowicz@yale.edu">david.lewkowicz@yale.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="msg2478473709005124023">
<div lang="EN-US" style="overflow-wrap: break-word;">
<div class="m_2478473709005124023WordSection1">
<p class="MsoNormal">Hi Lisa,<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Thanks very much for sharing your experience with Lookit and for pointing out the limitations of what we can expect in terms of recorded sounds and timing. The one good piece of news in all of this is that if you happen to pick visual
stimuli that produce a relatively large change in the overall luminance of the scene, and if the camera recording the participant picks up that change, then you can use this as your cue that the trial has started. Of course, if your stimuli do not increase
overall luminance then you’re back to the problem of figuring out when the trial starts and there, based on what you’re reporting, there isn’t a simple solution here. Oh well, one must be realistic about the limits of online testing.<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">And, I suspect that the intractability of the timing issue is probably due to the variation in the type of connection that the testing computer has to the internet (e.g., wi-fi vs. ethernet) as well as the download speeds of the testing
computer (which, of course, is partly determined by the ISP and the type of connection between the computer and the modem/router in the house).
<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Best regards,<u></u><u></u></p>
<p class="MsoNormal">David<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<div style="border-right:none;border-bottom:none;border-left:none;border-top:1pt solid rgb(181,196,223);padding:3pt 0in 0in">
<p class="MsoNormal"><b><span style="font-size:12pt;color:black">From: </span></b><span style="font-size:12pt;color:black">Lisa Oakes <<a href="mailto:lmoakes@ucdavis.edu" target="_blank">lmoakes@ucdavis.edu</a>><br>
<b>Date: </b>Saturday, March 18, 2023 at 1:16 PM<br>
<b>To: </b>"Lewkowicz, David" <<a href="mailto:david.lewkowicz@yale.edu" target="_blank">david.lewkowicz@yale.edu</a>><br>
<b>Cc: </b>Melissa Kline <<a href="mailto:mekline@mit.edu" target="_blank">mekline@mit.edu</a>>, "<a href="mailto:lookit-research@mit.edu" target="_blank">lookit-research@mit.edu</a>" <<a href="mailto:lookit-research@mit.edu" target="_blank">lookit-research@mit.edu</a>>, "Aslin, Richard" <<a href="mailto:richard.aslin@yale.edu" target="_blank">richard.aslin@yale.edu</a>><br>
<b>Subject: </b>Re: [lookit-research] stimulus sound not coming through on videos<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<p class="MsoNormal">Hi David (and everyone): <u></u><u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Online testing has such promise and yet has new frustrations.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<p class="MsoNormal" style="margin-bottom:12pt">First, what you are experiencing is not a lookit problem. Computers suppress what it interprets as background noise when recording. We have found that we hear the sound pretty reliably for some experiments
but not for others, and sometimes we hear some sounds but not others. I suspect it depends on the nature of the sound and the volume. Bottom line: it doesn’t matter how you are recording, if you collect data online you can’t depend on hearing the sound of
the stimuli. <br>
<br>
Moreover, even if you do hear sound, it is likely out of sync with the video. You may notice that when you see a child speak what you hear is out of sync with the video. Again, this is variable and there is no easy way to correct it. This is a known problem
with video recording and with Lookit. You can find more info here: <a href="https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Flookit.readthedocs.io%2Fen%2Fdevelop%2Fresearchers-lag-issues.html&data=05%7C01%7Cdavid.lewkowicz%40yale.edu%7C70d7851ac5dc41eb240a08db27d4790b%7Cdd8cbebb21394df8b4114e3e87abeb5c%7C0%7C0%7C638147565743157654%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=MiDTNZdQdqh4DV55IILN4R1btd3RJFMhOARUP5WKGQg%3D&reserved=0" target="_blank">https://lookit.readthedocs.io/en/develop/researchers-lag-issues.html</a><u></u><u></u></p>
<div>
<p class="MsoNormal">A better solution is to use the timing info provided by lookit. However, this also is not exact. We (and others) have done extensive testing of the timing of stimulus presentations in lookit and have found that there is some variation in
when stimuli start relative to when lookit says they start. I don’t think this has been solved and I don’t think it is known why it happens.<u></u><u></u></p>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal">We have tested hundreds of kids using online methods for the last 3 years and my recommendations are (1) don’t rely on sound for timing, and (2) online methods are best for stimuli that are static or if it doesn’t matter when the infant
looks relative to what is happening in the stimulus.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">This probably isn’t the answer you wanted, but I hope it is helpful as you find a solution.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Lisa<u></u><u></u></p>
</div>
<div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:9pt;font-family:Helvetica;color:black">Lisa M. Oakes<br>
Professor, Department of Psychology<br>
Faculty Researcher, Center for Mind and Brain<br>
University of California, Davis<br>
267 Cousteau Place<br>
Davis, CA 95618<br>
<br>
phone: 530 754-4523 OR<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:9pt;font-family:Helvetica;color:black">530 754-8304<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:9pt;font-family:Helvetica;color:black">Pronouns: she/her/hers<u></u><u></u></span></p>
</div>
</div>
<p class="MsoNormal"><span style="font-size:9pt;font-family:Helvetica;color:black"><br>
</span><img border="0" width="103" height="61" style="width: 1.0729in; height: 0.6354in;" id="m_2478473709005124023_x0031_27FB8DA-6EB0-4CD5-9C55-D527A07A7C43" src="cid:186f5ff78704cff311"><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><br>
<br>
<u></u><u></u></p>
<blockquote style="margin-top:5pt;margin-bottom:5pt">
<div>
<p class="MsoNormal">On Mar 18, 2023, at 8:45 AM, Lewkowicz, David <<a href="mailto:david.lewkowicz@yale.edu" target="_blank">david.lewkowicz@yale.edu</a>> wrote:<u></u><u></u></p>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div>
<p class="MsoNormal">Dear Melissa,<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Recently, we have noticed that the participant videos recorded by Lookit seem to filter out the stimulus sounds that we normally present in our videos. We use these sounds to mark the beginning of a trial when we code the participant videos
and, thus, their absence makes it impossible for us to reliably code the participant videos. Are you aware of this and, if so, is there a fix for this problem?<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Thanks very much in advance for looking into this problem.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Best regards,<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">David & Dick<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">__________________________________________<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">David J. Lewkowicz, Ph.D.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Senior Scientist<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Haskins Laboratories, New Haven, CT<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Yale Child Study Center<span class="m_2478473709005124023apple-converted-space"> </span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Personal:<span class="m_2478473709005124023apple-converted-space"> </span><a href="https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fhaskinslabs.org%2Fpeople%2Fdavid-lewkowicz&data=05%7C01%7Cdavid.lewkowicz%40yale.edu%7C70d7851ac5dc41eb240a08db27d4790b%7Cdd8cbebb21394df8b4114e3e87abeb5c%7C0%7C0%7C638147565743157654%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=7FZIKL%2FYKMjMvxPvUWfBgaWMVeLcK15KBhmgRLpHrsY%3D&reserved=0" target="_blank"><span style="color:rgb(5,99,193)">https://haskinslabs.org/people/david-lewkowicz</span></a><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Lab:<span class="m_2478473709005124023apple-converted-space"> </span><a href="https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fllamblab.haskins.yale.edu%2F&data=05%7C01%7Cdavid.lewkowicz%40yale.edu%7C70d7851ac5dc41eb240a08db27d4790b%7Cdd8cbebb21394df8b4114e3e87abeb5c%7C0%7C0%7C638147565743157654%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=JOMur9zdTinqsmqpo3vcJ9Zh0N0IEj1ulApF8286GMc%3D&reserved=0" target="_blank"><span style="color:rgb(5,99,193)">http://llamblab.haskins.yale.edu/</span></a><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<p class="MsoNormal"><span style="font-size:9pt;font-family:Helvetica">_______________________________________________<br>
lookit-research mailing list<br>
</span><a href="mailto:lookit-research@mit.edu" target="_blank"><span style="font-size:9pt;font-family:Helvetica">lookit-research@mit.edu</span></a><span style="font-size:9pt;font-family:Helvetica"><br>
</span><a href="https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmailman.mit.edu%2Fmailman%2Flistinfo%2Flookit-research&data=05%7C01%7Cdavid.lewkowicz%40yale.edu%7C70d7851ac5dc41eb240a08db27d4790b%7Cdd8cbebb21394df8b4114e3e87abeb5c%7C0%7C0%7C638147565743157654%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=SVGsaYiANzE2bGmWyrkhdmz%2BF0GMFtoup7lmiIPT7cQ%3D&reserved=0" target="_blank"><span style="font-size:9pt;font-family:Helvetica">https://mailman.mit.edu/mailman/listinfo/lookit-research</span></a><u></u><u></u></p>
</div>
</blockquote>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div>
</div>
</div>
</div></blockquote></div><br clear="all"><div><br></div><span class="gmail_signature_prefix">-- </span><br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr">Melissa Kline Struhl</div><div>Lookit Executive Director</div><div>Research Scientist, MIT Brain & Cognitive Sciences</div><div><a href="mailto:mekline@mit.edu" target="_blank">mekline@mit.edu</a></div><div><a href="http://lookit.mit.edu" target="_blank">https://lookit.mit.edu/</a></div><div dir="ltr"><div><a href="https://www.melissaklinestruhl.com" target="_blank">http://www.melissaklinestruhl.com/</a><br></div><div><br></div><div><br></div><div><br><br><div style="display:inline"></div></div></div></div></div></div></div></div>