Gaze behavior in children is often measured as a hallmark of typical development. Notably
children diagnosed with an autism spectrum disorder have atypical gaze behavior from
looking preferences to metrics and timing seemingly from very early in life. The measures
in this literature, however, have been largely collected through remote eye tracking measured
while a participant is observing objects, faces, or scenes presented on a screen. To
demonstrate validity with real-world behavior we need to make these measurements in the
real world with dynamic social scenes. This presents challenges, however, since it is
much easier to describe interest areas prospectively on a screen than in the world with dynamic
moving objects and people. It is also currently much easier to obtain reliable gaze estimation
data using remote versus wearable trackers. We and others have contributed to this effort
previously, both in terms of offline and real-time object and face detection strategies. Still,
our field is a way from having a tool that can readily be used in clinical practice for
measuring gaze behavior in real world settings.
In 2018 we presented a method for quantitatively analyzing gaze behavior and fixated object
identification using a runlength filter approach on the gaze-in-bounding-box results (from
Chukoskie and colleagues in IEEE Cognitive Development). Our current work improves on
that in several ways.
First, we improved the quality of the data by improving the calibration approach. We
also smoothed the gaze data stream using a Kalman filter to remove blinking. Second,
we used a Faster-RCNN face detector. Third and most substantially, we developed a novel
approach based on unsupervised clustering to substantially improve the detection of
looks from the data stream. Fourth, we introduce a new set of metrics for evaluating performance
of our model compared with a human rater providing ground truth . Previous results focused only
on frame-level accuracy, which is useful for analyzing, for example, the proportion of
time spent by a subject in looking at different objects. Here we examine look-level results
as well. Look-level results can have value for assessing the timing and naturalness of
social gaze behavior. Finally, our current effort uses objects from the COCO dataset
[from Lin and colleagues in 2014], which demonstrates that the system works well with common real-life
objects and hopefully sets up our efforts for broader use in the future.
The details of the Kalman filtering and calibration can be found in the paper. From the COCO dataset
we chose 6 real-life objects that are shown in this figure showing our experimental set
up: keyboard, stop-sign, cup, bowl, bottle, monitor. The gaze tracking participant sits
across from another participant, now wearing an eye tracker. All objects and the second
participant s face are readily viewable by the eye tracking subject.
We estimated a tight axis-aligned bounding box around these objects detected in the world
video using a Faster-RCNN network [from Ren and colleagues in 2015] pretrained on the
COCO dataset using the tensorflow object detection api [from Huang and colleagues in 2016]. For
face detection, we use a Faster-RCNN model pretrained on the WIDERFACE dataset [from
Yang and colleagues in 2016]. The detections obtained from the two networks are combined.
Henceforth, we will refer to a human face as an object in the list of 7 objects of interest.
Our overall goal is for the system to detect the looks of the subject to these objects
of interest.
Here in Figure 1, I'm showing you a system diagram beginning with gaze and world view
video data obtained from the Pupil Pro headset and ending with identified looks to the 7
objects. The top of the diagram shows the overall data path while the bottom characterizes
what constitutes look detection for each object. In the look detection diagram, which is an
expansion of a middle step in the overall system diagram, the first step is recentering
the coordinates on the center of each object. Next the unsupervised clustering occurs. We
use the DBSCAN algorithm [from Ester and colleagues in 1996] since it does not require the number
of clusters to be pre-specified, and it is inherently density-based, which is suitable
for our purposes. Once the clusters are obtained, we only need to retain the clusters in a small
region centered at the origin since we perform a separate clustering to find the looks to
each object. For this, we examine the detections of an object i across all frames and find
the dimensions of the maximum bounding box around the object. Next, we only retain clusters
whose centroid falls inside a region 90% of the maximum bounding box for that object,
centered at the origin. We chose parameters for the DBSCAN algorithm based on the knee
in the precision-recall curve, shown here in Figure 2 which led to an epsilon of 25.
We compared our system s performance with a holistic human expert s ground truth evaluation
of looks to objects. We sought to characterize performance in terms of precision, recall,
and accuracy which are defined in terms of true positive, false positive and false negative
looks. The True Positive looks were defined by the ground truth, while the false positive
and false negative rates are assessed by instances where the system performance deviated from
ground truth as shown here a false positive where the algorithm detects a look where the
human expert does not and here a false negative where the algorithm fails to detect a human
expert-identified look. In this framework precision is the true positive look frames
divided by the sum of the true positive and false positive look frames. Recall is the
true positive look frames divided by the sum of the true positive and false negative look
frames. Finally, accuracy is the true positive look frames divided by the sum total of the
true positive, false positive, and false negative look frames.
Now for some results. Although the Frame-Level results showed somewhat better performance
of the clustering approach in terms of recall and accuracy, we were more interested in the
look-level performance comparison of the clustering versus runlength approach. The look-level
detection results, shown here in Table 4, reveal that the numbers of false negatives
are low, 1.8% for the clustering approach and 2.6% for the runlength approach, indicating
that most of the ground truth looks are being detected by both algorithms. For faces, somewhat
more looks are missed by the runlength algorithm, likely due to the subject looking around the
face as is typical during face scanning. The number of extra looks is much higher for the
runlength algorithm, which means it is splitting some ground truth looks into multiple looks
to the object. The number of false positives is also much higher for the runlength algorithm.
Overall, the clustering algorithm is considerably more accurate at detecting looks.
For next steps we plan to improve ground truth characterization of looks by using multiple
trained raters. This would provide an advantage of whether an algorithm s performance is within
the range of normal variation in trained raters. We are also seeking to speed up performance
with the goal of running these calculations on a laptop in real time. Currently the object
detection step is taking the bulk of the time and is ripe for improvement.