From kimscott at mit.edu Tue Apr 7 12:27:22 2020 From: kimscott at mit.edu (Kim Scott) Date: Tue, 7 Apr 2020 12:27:22 -0400 Subject: [Baby-gaze-coding] Update on baby gaze coding efforts at MIT Message-ID: Hi folks interested in automated gaze coding for developmental research! *This is just an update for the curious, no action required.* Hope everyone's doing as ok as possible. I just wanted to give a quick update, since the last one was in October 2018. When we left off, we'd identified OpenGaze as the most promising starting point for gaze coding, and were going to submit a propose to the MIT Quest for Intelligence (QI) "Bridge" program, which exists to help address scientific problems using AI/ML (a great fit). QI did end up helping with trying out some approaches, but has hit a few roadblocks. Over the past year (~fall 2018 - fall 2019), Katherine Gallagher at QI has been working on this with the support of ~5-8 undergraduates, exploring a variety of approaches. They?ve confirmed that simply using direction of gaze as a proxy for gaze target (left/right) doesn't work well (as expected). One challenge is that what looks like the most promising existing system OpenGaze to implement the second "layer" of processing - the transformation from direction of gaze to target of gaze - has been really hard to get running. At all. Nine people have now tried and failed to get it to run, the senior engineer has spend >100 hours on it. It has been a year. (Does that sound like a challenge? PLEASE try installing it and let everyone know if you're able to get it running! And maybe Dockerize it after! :) ) So the current state is that there really needs to be a dedicated engineer to focus on developing a practical tool - not some more exciting version of this problem that might eventually have applications, not a proof of principle, a usable tool - for likely 1-2 years. MIT QI does not have the resources to take primary ownership of this project at this time. It's hard to pitch this to an academic lab, because much of this endeavor is not an exciting academic problem - e.g., it takes time to design and test a good UI. That said, the project would have tremendous impact and would open up new experimental techniques. It?s relatively low-risk - there are good starting points and even small improvements would be difference-making; there are plenty of extensions and improvements to explore; it would provide access to an essentially new type of data if someone were interested being involved with the first use case. Why hasn't this happened yet? One problem is that the problem has been persistently underestimated. People think it can?t possibly be that hard given the state of existing systems. They figure we just need to talk to so-and-so who has already solved it; and indeed existing output feels like since it?s capturing *something* relevant we must be pretty close. We have had ~5 student project groups (grad/undergrad) at MIT work on this with our support, because it is *so persistently underestimated that it gets assigned as a class project*. But now with QI's work I think there's some strong empirical evidence that even with a good starting point, there's a fair amount left to do to get a usable tool for developmental video. Note that QI conducted an updated literature review and still concluded that OpenGaze was the most promising approach. There is also some interest from Antonio Torralba?s group in CSAIL, which developed Gaze360, but they are interested in an adjacent scientifically-interesting proof-of-concept (a novel approach to gaze tracking relative to a screen) and not necessarily in careful evaluation or eventually UI development. This is on the back burner for me at the moment for obvious reasons, but if anyone's interested in applying for funding in this direction, feel free to get in touch with the group to coordinate. And budget for more than an undergrad for a term ;) best, Kim --- Kim Scott Research scientist | Early Childhood Cognition Lab | MI *T*W: www.mit.edu/~kimscott | Participate: https://lookit.mit.edu -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.mit.edu/pipermail/baby-gaze-coding/attachments/20200407/8dbe68fa/attachment.html