My name is Ludwig Sidenmark and I'm a PhD student at Lancaster University. I'll present
our research project, BimodalGaze, which received an honourable mention award at last year’s ETRA.
This project was done in collaboration with Diako Mardanbegi, Argenis Ramirez Gomez,
Chris Clarke, and Hans Gellersen. BimodalGaze is a novel gaze-pointing technique
for seamless head-based refinement of a gaze cursor. The technique leverages insights into
eye-head coordination to classify and identify the difference between natural head movement
and explicit gestural head movement. We utilise this for seamless head-based refinement of
a gaze cursor - allowing users to quickly shift their gaze to targets with naturally
combined eye-head movement, and to refine the cursor position with gestural head movement.
I'll present the design of the technique, and results for how it supports this type of interaction.
Gaze is natural and fast for pointing, but poor for fine-grained cursor control and selection.
Alignment over a target is not precise and combined with eye tracking calibration and
measurement limitations induces uncertainty into gaze estimation.
A variety of work has tackled this issue by using a second modality to refine the gaze
position. The head is interesting for this purpose as it is stable and precise while
still allowing hands-free pointing. Head correction of gaze has been demonstrated on for example
desktop displays where gaze shifts are performed by the eyes alone, allowing head movement
to be treated as separate input for cursor refinement.
However, only small gaze shifts are performed solely with the eyes. Larger gaze shifts naturally
feature head movement to comfortably reach targets. It is therefore not straightforward
to use head movement for gaze correction, in particular when gaze is considered for
pointing across a larger field of view, such as in virtual or augmented reality.
To avoid usability issues caused by refinement being triggered unexpectedly, there is a need
to be able to distinguish between natural head movements performed during gaze shifts,
and gestural head movements that are independent from gaze. Previous work on Pinpointing has
made this distinction by requiring the user to manually toggle between pointing and refinement.
In our work, we aim to automate this process.
BimodalGaze uses gaze as the primary modality for quick and effortless pointing. When gaze
is not accurate enough for selection, the technique allows automatic switching to head-refinement
when users perform deliberate head movements. BimodalGaze leverages eye-head coordination
insights to decide if a head movement is gestural for refinement; or a natural movement during
a gaze shift that should be ignored. In BimodalGaze, all head movements above a
simple velocity threshold are processed and classified as either natural or gestural.
We have shown in previous VR work how natural head movements are used to further the range
of the eyes or to move the eyes into a comfortable position. Based on this insight, we define
two criteria to differentiate natural from gestural head movements:
(1) A natural head movement starts within a certain time period after an eye movement.
(2) A natural head movement will move in a similar direction to the prior eye movement.
The first criterion relates to the timing of head movements. Natural head movements
occur at the same time, or a shortly after the eye movement. As such, we assume that
a head movement is natural if the head moves within a certain time period after the eyes
move. The second criterion relates to the movements’
trajectories. As natural head movements are used to comfortably increase the eyes’ reach,
it is reasonable to believe that a natural head movement will move in a similar direction
as the eyes. A head movement is then defined as natural if the angular difference between
the trajectory of the eyes and head are within a certain range. If these criteria are not
true, we assume the head movement to be gestural and the technique switches to head refinement.
The technique will switch back to Gaze Mode during deliberate gaze movements, which we
define as any gaze movement over a velocity threshold. Alternatively, the system will
switch back to Gaze Mode if the distance between the gaze and pointer position, is greater
than a threshold. This is to prevent the cursor becoming too detached from the gaze due to
misclassification of a natural head movement detected as gestural, or due to eye movements
which are not detected as deliberate gaze movements.
We conducted a VR user study to evaluate how BimodalGaze performance is affected by target
distance and accuracy error. We also assessed the effectiveness of the automatic switching
between modes, by comparing it with the Eye+Head Pinpointing technique, where the switch is
manually triggered via a button hold-down. The selection of thresholds for BimodalGaze
has a large impact on its behaviour and requires careful consideration. The gaze and head velocity
thresholds were set to relatively high values to avoid unintentional switches caused by
minor movements. The timing and trajectory thresholds were
based on our prior work on eye-head coordination in VR. Head movements generally start moving
150ms after the eyes. We also found that for 90% of all gaze shifts with accompanying head
movement, the eye and head trajectory were within 20° of each other.
Finally, we set the failsafe threshold to a moderately low value so that the pointer
position can be freely adjusted while not being able to move the pointer too far out
in the periphery.
The headline results from the study are:
Users are able to rely on gaze when gaze only is enough for accurate selection. This makes
BimodalGaze different from Pinpointing as head-refinement is optional rather than enforced.
All participants successfully completed all selection tasks. When gaze was too inaccurate
or initial selections were erroneous, users had no problem correcting their input.
BimodalGaze enables selection of targets that cannot be acquired with gaze alone and participants
are able to successfully select all targets irrespective of scale of eye tracking error.
The results show that the switching between gaze and head modes can be automated using
the underlying knowledge of the way in which our head movement supports gaze.
The advantage is that our technique does not require explicit input for mode-switching,
resulting in a more seamless transition. This also allows BimodalGaze to be combined with
confirmation techniques such as dwelling to make the whole selection hands-free. Looking
even further ahead, the distinction of natural and gestural head movements makes it possible
to attach different behaviours to objects that take gestural head movement as input
while avoiding unwanted behaviours caused by natural head movements. For example, gestural
head movements could be used to manipulate gazed-on objects.
However, the study also pointed towards some performance limitations. In comparison to
Pinpointing, BimodalGaze had a higher error rate and was on average slightly slower for
selection by around 140ms. This was mainly due to two factors:
First, a common situation that led to errors in Gaze Mode was the natural overshooting
or undershooting of the primary saccade. This, combined with a high accuracy error, caused
the pointer to appear on the target when the participant’s gaze was not. The participant
would then perform a selection. However, when the button press was registered, the participant’s
gaze had moved onto the target, which in turn moved the pointer outside the target. Clearer
pointing feedback or target acquisition techniques could improve this aspect.
Second, the high head velocity threshold caused difficulties in identifying small head movements
needed for minor refinements. As such, users would either rely on Gaze Mode causing incorrect
selections, or perform exaggerated head movements that would overshoot the target. To improve
performance, we recommend using more sophisticated techniques and models (for example via machine
learning) to detect head movements. Please refer to the paper for full study details.
So in conclusion: Leveraging eye-head coordination insights
allows automatic refinement of gaze pointing for accurate selection which is significant
as it removes the need for any manual or other explicit input thus making the process more
seamless. The notion of classifying natural and gestural
head movements extends beyond refinement of a pointer, opening up new opportunities for
mapping gestural head movement without affecting the head’s natural ability to support gaze.
That is all for my talk, thank you for listening.