Hi everyone!
My name is Mihai and I will present our work on detecting pursuits through gaze estimation
and optical flow.
This is joint work with Vincent Becker, Chenyang Wang, and Andreas Bulling.
In recent years, Pursuits has emerged as a gaze interaction technique that enables spontaneous
and calibration-free interaction with dynamic user interfaces.
It is based on the smooth pursuit eye movement and occurs when following a moving target
at an appropriate speed.
By correlating the eye's motion with the trajectory of on-screen objects, we can robustly identify
which target the user is following.
The original paper introducing Pursuits laid the foundation for follow-up works that cover
applications in many interactive settings such as public displays, smartwatches, and
even virtual reality.
However, all of these works rely on dedicated eye tracking equipment, which hinders the
large-scale adoption of such an interaction technique.
To address this limitation, we propose a novel method to detect pursuits using a single off-the-shelf
RGB camera in unconstrained remote settings.
Our method combines appearance-based gaze estimation and optical flow into a joint pipeline.
It is therefore able to capture both the gaze direction and the eye movement dynamics during
a pursuit movement.
Let's have a brief look at our approach.
Given a sequence of images, our method first detects the user's face and the facial landmarks.
This input is used by both the gaze estimation and optical flow components.
On the gaze estimation side, images are first normalised.
Normalisation is a preprocessing step that, by rotating and scaling the image, cancels
out differences in user appearance caused by the distance between the user and the camera
or different hardware setups.
Our method uses a full-face appearance-based gaze estimation neural network that predicts
the 3D gaze direction, which is then projected into 2D.
Given multiple 2D gaze points over time, we use the Pearson correlation coefficient to
identify which moving target is most similar to the motion of the users' eye.
The goal of the optical flow component is to complement the gaze estimator.
Similarly to gaze estimation, the first step is to normalise the image.
A key difference from the other normalisation step is that we do not rotate the input image
since this will change the direction of the optical flow vectors.
We then use dense optical flow to estimate the pattern of apparent motion between two
image frames.
We only select flow vectors that are inside the polygon defined by six eye landmarks,
as shown in the figure.
To calculate the eye movement direction, we average these flow vectors in Cartesian space.
Lastly, we calculate the average cosine similarity between the movement direction of the eye,
as obtained using optical flow and the target motion vector.
The final step in our method is to combine the predictions from the two components.
Each one of them will output one detected target for each time window, which was 1s
or 30 image frames in our implementation.
We use a strategy that merges the detection results whenever the two components agree
or when only one of them can detect a target.
To further improve robustness, we took several such detections into account and implemented
a voting scheme.
Empirically, we achieved the best results when a target received three votes.
These votes do not have to be in direct succession.
We evaluated our method on a novel dataset collected from 13 participants: 10 male and
3 female.
Their ages ranged from 23 to 50.
Their task in the study was to follow a red dot moving along multiple predefined trajectories.
During each trial, we recorded participants with a webcam and also collected gaze data
from a Tobii 4C eye tracker.
The images on the right show our experimental setup.
Before starting a session with a new participant, we calibrated the eye tracker using the software
provided by the manufacturer.
The moving dot that participants had to follow was moving along one of four types of trajectories:
Linear, circular, orbital, and rectangular.
Linear trajectories moved back and forth along a straight line.
Circular trajectories followed a circle, with all targets being on the same circle.
Orbital trajectories also follow a circle, but other targets moved on circles with different
radii.
Rectangular trajectories followed a rectangle, with all moving elements on the same rectangle
at an equal distance apart from each other.
We fixed the size of the trajectory as the similarity metrics we used were invariant
to scaling.
We also tested different velocities.
For example, normal, which was about 45 degrees/second versus fast, about 63 degrees per second for
circular trajectories.
So, how well does our method work?
We evaluated the effect of having different trajectory types, velocities, and multiple
pursuit targets, which we called competitors.
We measured accuracy as the percentage of correctly identified targets.
And we also compared our method to two baselines.
First, EyeFlow is the current state of the art for detecting pursuits using optical flow.
While this method was designed for head-mounted settings, it is the most similar to our work.
Second, we compared our method to an implementation that uses a commercial, widely available eye
tracker, namely the Tobii 4C.
Results show that our method significantly outperforms the current state of the art.
Moreover, it is also competitive to a commercial eye tracker for a small number of targets.
We also compared the results at different target velocities.
While the overall average accuracy is very similar, we notice a lower standard deviation
at higher velocities.
In an ablation study, we wanted to investigate which component (gaze estimation or optical
flow) contributes to the overall performance.
When the number of competitors is low, gaze estimation is able to achieve similar results
to our joint method.
With an increase in the number of targets and competitors, the accuracy of the gaze
estimation component decreases and matches the one from optical flow.
In such cases, our joint method is able to maintain an improvement of about 8 to 10 percentage
points.
The analyses shown in this presentation only show some of the evaluations we conducted.
For additional results and insights into other aspects such as runtime and response time,
please have a look at our paper.
In summary, we proposed a novel method to detect pursuits by combining gaze estimation
and optical flow to jointly analyse the eye movement dynamics.
Our method only requires a single RGB camera, which is readily available on an ever-increasing
number of devices.
And, finally, we demonstrate state of the art results that are also competitive to a
dedicated eye tracker for a small number of moving targets.
Thank you!