[Baby-gaze-coding] Automated gaze coding update - October

[Kim Scott]

Hi folks,

Thanks to everyone for your interest in automated gaze coding for
developmental research! Here's a quick update on what's happening...

*TL;DR: What you can do (for developmental researchers):*

1. Please check out the proposed standard <https://osf.io/xh54d/> and
comment up the "How to contribute" document with
problems/questions/suggestions. If you can, send the group an example of a
video coding output file from your lab, to help inform choices about video
coding standards.

2. Take 15 minutes to look through current funding opportunities and send
any that might be good fits for gaze coding algorithm development to the
group!

3. We would still love to find someone who's interested in taking on the
coordination of data sharing. Email me or Kat if you're interested or think
this might make a good project for a student of yours!

*The technical side:*

Machine vision and HCI folks have been amazingly generous with their time.
After a bunch of meetings with people who have worked on eyetracking from
video (including developers of WebGazer, TurkerGaze, PyGaze, tracking.js,
and CVC eye-tracker!), we have a few points of consensus:

- The big technical challenge is likely to be the facial landmark detection
step, not the gaze estimation step. (Although, as Mike pointed out last
time, gaze estimation for most of us is categorical and the categories are
defined per study/lab, so there will at least be a UI challenge there in
allowing people to adapt an algorithm for their own end goals.)

- Robustness to movement, ethnicity, lighting, and partial occlusion are
all challenging for current approaches, but they're getting better.
(People's degree of optimism varies wildly... :) )

- The clear favorite for a starting point for this project is this work
<https://www.mpi-inf.mpg.de/departments/computer-vision-and-multimodal-computing/research/gaze-based-human-computer-interaction/its-written-all-over-your-face-full-face-appearance-based-gaze-estimation/>
from Andreas Bulling's group, an appearance-based approach that uses a CNN
and builds on OpenFace <https://github.com/TadasBaltrusaitis/OpenFace> for
landmark detection. (Although we also still need to try out & get in touch
with GazeCapture <http://gazecapture.csail.mit.edu/> here at MIT!) To get a
feel for it, I tried out OpenFace on some of our Lookit videos. Even out of
the box, it's working amazingly well at finding baby faces, even in the
situations where we'd previously encountered a lot of trouble. Here's a
great example
<https://drive.google.com/open?id=1pkmLJ_ICEcuShy9SNTWguGRVRhTITdJp>: The
kiddo's sucking a pacifier, not "perfectly" positioned, sometimes slightly
occluded by his mom's shoulder and hair. It loses track briefly but that
looks like it might be due to the jump in our video (since tracking
actually does take into account recent history). Here's another good example
<https://drive.google.com/open?id=1o5WKNCjYE58x6FcQ0NmK6B65TNZk7ro4> with a
kid who wiggles around a bit more.

*The data side:*

I've put together a rough draft of a standard for video metadata + coding
(just preferential looking as a start), along with an example of some of
our data in this format, which you can find at https://osf.io/xh54d/

This is just meant as a starting point: no one else spoke up about their
great enthusiasm for data formatting, so I started with something close to
our own data format - but I'm definitely missing difficulties that will
come up with varying setups & coding procedures!

*Funding opportunities:*

As a first step we're currently submitting a pre-proposal to MIT's Quest
for Intelligence "bridge" program (helping to apply AI in the service of
scientific research), proposing to collect a large preferential looking
dataset on Lookit & develop the tools needed to analyze it efficiently. We
have a few leads on potential collaborators on the technical side, but are
still figuring out details.

best,
Kim

---
Kim Scott
Research scientist | Early Childhood Cognition Lab | MIT
W: www.mit.edu/~kimscott | Participate: lookit.mit.edu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/baby-gaze-coding/attachments/20181024/ce8a0066/attachment.html