[Baby-gaze-coding] Update on baby gaze coding efforts at MIT

[Kim Scott]

Hi folks interested in automated gaze coding for developmental research!

*This is just an update for the curious, no action required.*

Hope everyone's doing as ok as possible. I just wanted to give a quick
update, since the last one was in October 2018.

When we left off, we'd identified OpenGaze as the most promising starting
point for gaze coding, and were going to submit a propose to the MIT Quest
for Intelligence (QI) "Bridge" program, which exists to help address
scientific problems using AI/ML (a great fit).

QI did end up helping with trying out some approaches, but has hit a few
roadblocks. Over the past year (~fall 2018 - fall 2019), Katherine
Gallagher at QI has been working on this with the support of ~5-8
undergraduates, exploring a variety of approaches. They’ve confirmed that
simply using direction of gaze as a proxy for gaze target (left/right)
doesn't work well (as expected). One challenge is that what looks like the
most promising existing system  OpenGaze to implement the second "layer" of
processing - the transformation from direction of gaze to target of gaze -
has been really hard to get running. At all. Nine people have now tried and
failed to get it to run, the senior engineer has spend >100 hours on it. It
has been a year. (Does that sound like a challenge? PLEASE try installing
it and let everyone know if you're able to get it running! And maybe
Dockerize it after! :) )

So the current state is that there really needs to be a dedicated engineer
to focus on developing a practical tool - not some more exciting version of
this problem that might eventually have applications, not a proof of
principle, a usable tool - for likely 1-2 years. MIT QI does not have the
resources to take primary ownership of this project at this time. It's hard
to pitch this to an academic lab, because much of this endeavor is not an
exciting academic problem - e.g., it takes time to design and test a good
UI. That said, the project would have tremendous impact and would open up
new experimental techniques. It’s relatively low-risk - there are good
starting points and even small improvements would be difference-making;
there are plenty of extensions and improvements to explore; it would
provide access to an essentially new type of data if someone were
interested being involved with the first use case.

Why hasn't this happened yet? One problem is that the problem has been
persistently underestimated. People think it can’t possibly be that hard
given the state of existing systems. They figure we just need to talk to
so-and-so who has already solved it; and indeed existing output feels like
since it’s capturing *something* relevant we must be pretty close. We have
had ~5 student project groups (grad/undergrad) at MIT work on this with our
support, because it is *so persistently underestimated that it gets
assigned as a class project*. But now with QI's work I think there's some
strong empirical evidence that even with a good starting point, there's a
fair amount left to do to get a usable tool for developmental video.

Note that QI conducted an updated literature review and still concluded
that OpenGaze was the most promising approach. There is also some interest
from Antonio Torralba’s group in CSAIL, which developed Gaze360, but they
are interested in an adjacent scientifically-interesting proof-of-concept
(a novel approach to gaze tracking relative to a screen) and not
necessarily in careful evaluation or eventually UI development.

This is on the back burner for me at the moment for obvious reasons, but if
anyone's interested in applying for funding in this direction, feel free to
get in touch with the group to coordinate. And budget for more than an
undergrad for a term ;)

best,
Kim

---
Kim Scott
Research scientist | Early Childhood Cognition Lab | MI
*T*W: www.mit.edu/~kimscott | Participate: https://lookit.mit.edu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/pipermail/baby-gaze-coding/attachments/20200407/8dbe68fa/attachment.html