Hi everyone, I’m Ludwig Sidenmark from Lancaster University and with me for this presentation
we also have Chris Clarke from Lancaster and Xuesong Zhang from KU Leuven. And we are going
to present our work on Outline Pursuits: Gaze-assisted selection of occluded objects in virtual reality.
Occlusion presents a principal problem for selection in 3D environments as it reduces
the target area for pointing and increases selection ambiguity. Users can change their
position to improve the view they have of a target. However, this is a limited
option when movement is restricted, such as when sitting or if the tracking space is limited.
In this work, we propose to support selection of occluded objects with Outline Pursuits.
As shown in the figure, the concept is to display the outlines of objects that lie in
the pointing direction, and to generate a distinct motion along each outline. Users
can then disambiguate the selection by following the motion around their intended target with
their eyes.
Before we go into the details of Outline Pursuits I would like to take a step back and look
at some related work.
The most commonly used metaphor for selection in 3d environments is Raycasting where the
user controls a ray via a controller or body part. Raycasting allows selection of out of
reach targets, however targets at greater distances are harder to select due to limitations
of human motor control and motion tracking. To compensate for accuracy and occlusion issues,
raycasting can be combined with volumetric techniques for selection. These require disambiguation
from initially selected objects. This can be done via an additional manual step for
or by applying heuristics or contextual information for implicit disambiguation.
With gaze it is easy to be accurate from a user perspective but sensing is difficult
which limits usability. Pursuits offers an alternative selection technique where the
users follow a moving target with their gaze. A selection is then performed by correlating
the target's and the eyes' motion, without the need for eye tracking calibration. Selection
is robust provided candidate motions are sufficiently different.
Prior work has used smooth pursuit for selection and disambiguation in VR, however for selection
of objects presented in motion. A distinct novelty of our work is that we instead present
motion around static 3D objects to facilitate their selection by pursuit without changing
the object’s size or position.
Outlines has many utilities in virtual environments, it is used to show objects that are interactive
or to highlight a current selection and also as an accessibility tool to increase the visual
contrast between objects. We add to the utility of outlines by augmenting them with motions
for smooth pursuit interaction.
With that being said, Chris is now going to break down Outline Pursuits and the steps
of interaction.
Thanks Ludwig. From a user’s perspective, selecting an object using Outline Pursuits
involves two stages. The first stage is candidate selection, which is used to reveal potential
targets using coarse-grained pointing. The second stage is selecting one target from
the candidates using smooth pursuit-based selection.
We will first look at the candidate selection stage which consists of cone-casting, outline
extraction, and motion generation.
Candidate targets are selected using a cone-casting technique. This stage is designed to encourage
coarse-grained, lazy pointing which can be performed with any type of pointing modality.
By limiting the number of candidates for selection, we reduce visual clutter and increase the
robustness of selection using smooth pursuits.
As we are in a controlled virtual environment, we can accurately extract the outline of virtual
objects. This has two main purposes: in the first instance it reveals to the user which
items are available for selection, similar to the way in which outline highlighting is
traditionally used. Secondly, it provides a predictable path for which we can present
the smooth pursuit target movement without having to change the properties of the object
itself.
Once we have extracted the outlines we can augment them with target movements to be used
for smooth pursuit selection. The goal here is that each moving target moves in a unique
trajectory. Unlike previous smooth pursuit approaches, the shape of the trajectory is
dependent upon the candidate objects, and therefore we rely solely on the spatiotemporal
properties of the target movement.
In the first instance, we utilise the direction in which the targets move to create two groups:
clockwise and anti-clockwise.
For more than two candidate targets we set the number of rotations per second to be equal.
We then maximise the phase differences between target movements to ensure each trajectory
is as unique as possible. We ensure that the velocity of the target is within a comfortable
range for smooth pursuit eye movements.
We also have to carefully consider how to present outlines and target movements in the
presence of occlusion, for this we looked at four different variants.
Whole outline, where the entire outline is visible, even under occlusion.
Shared outline, where the target moves along the visible part of the object.
Cut outline, where the target moves along the shortest path to the next visible part.
And finally, Jump outline, where the target jumps to the next visible part.
A user study with sixteen participants revealed that performance and usability ratings were
consistent across outlines. This indicates that designers can freely choose an outline
that fits their application without risking major performance penalties. We chose to continue
with the whole outline, as this is the only variant which can be used for selection under
full occlusion.
Once the candidate targets have been revealed, the final stage is to select one of the targets.
This is achieved using smooth pursuit eye movements to disambiguate one target amongst
many, followed by confirming the selection.
Target disambiguation occurs when the user follows the relevant target using their gaze,
resulting in a smooth pursuit eye movement.
The system takes as input, the user’s eye positions and all of the candidate target
positions as spatiotemporal trajectories. We then calculate a similarity metric for
each target to determine which, if any, the user is following.
Traditionally, disambiguation and confirmation occur in a seamless fashion for smooth pursuit
interactions, when the similarity value of one of the targets surpasses a threshold.
In this work we also consider separating disambiguation and confirmation stages. For this, we need
only be concerned with the target that gives the best match to the eye movements at any
given moment, therefore removing the requirement for a threshold. This may be used for selection
based upon specific timing, or when a user wishes to highlight or learn more about an
object prior to making a selection. It also helps further reduce the risk of accidentally
selecting a target, as the user needs to explicitly confirm the selection.
I will now hand over to Xuesong who will discuss the techniques and applications we built based
on Outline Pursuits.
Thanks, Chris. We demonstrate Outline Pursuits with the implementation of two concrete techniques,
one for hands-free selection, and one where it extends controller-based raycasting. Each
leverages a different coarse-grained and selection confirmation mechanism.
The hands-free version uses the head as the pointing mechanism for candidate selection.
When the calculated correlation is above a given threshold we assume the motions are
matched and the corresponding target is selected. This technique allows for hands-free applications,
or for users to interact with out-of-reach objects without having to put down or stop
interacting with virtual objects already in-hand.
We developed a virtual cityscape application to illustrate this technique. The application
lets users explore the city and buildings using only their head and eyes, with selection
triggering additional information. Here, outlines not only show which buildings are interactive
but also are used in the selection process. Outlines adapt to the user’s perspective
and nearby occluded buildings, allowing selection from all viewpoints and accurate selection.
In contrast to other gaze-based techniques, such as gaze-dwell, Hands-free Outline Pursuits
enables a user to gaze upon an object for an indefinite period without making a selection.
The second technique is Controller-based outline Pursuits, which utilizes controller-based
pointing in the coarse-grained stage, and a button click to confirm the selection.
We developed a room planner application to showcase the technique. Users can select and
manipulate objects that are occluded or in close proximity to each other without forcing
the user to change their point of view or hand position. This means users are able to
select an object with little hand movement and thus plan their hand placement for the
following object manipulation. Separating disambiguation and confirmation stages allows
the user to check additional details prior to confirming selection.
We designed and conducted a user study to investigate the performance of the proposed
Outline Pursuits techniques, and gather user feedback.
In a VR environment, participants were tasked with selecting a target object among many
as fast as possible.
We varied the object densities, distance to the user, and levels of occlusion. We measured
selection time, error rate, head and controller movement, and user preferences.
For comparison against the hands-free Outline Pursuits approach, we used a gaze-dwell technique,
where a target is selected once the user has dwelled upon it for one second.
For comparison against the controller-based Outline Pursuits technique, we chose both
controller- and gaze-based ray casting approaches. In both cases, confirmation is made via button
click.
Users found gaze-dwell to be effective for selection at low occlusion because it required
minimal effort. However, selection was difficult at high occlusion because of the imprecision
of the eye tracker and large amount of head movement required. Hands-free outline pursuits
required minimal head movement, and showed no difference with gaze-dwell at high occlusion.
However, it suffered from higher selection times and error rates, despite feedback stating
the interaction technique was simple. This was in part due to some participants not reaching
the required threshold for selection, which resulted in larger variance across participants.
Controller-based ray-casting proved to be very fast, and the most popular selection
technique. However, users were less accurate with highly occluded targets, and required
more movement as the target distance increased.
Gaze-based ray-casting was similar, but deteriorated much more at high occlusion, due to eye tracking
imprecisions and the limited range of movements of the head relative to the hands.
In contrast, Controller-based Outline Pursuits achieved consistently low error rates across
all conditions. It also required less movement from the participant, independent of the level
of occlusion or distance to the target. Preference-wise, users felt there was little difference between
ray-casting and outline pursuit techniques, however many favoured ray-casting because
of the faster selection times at low occlusion.
Now back to Ludwig for the conclusion.
So to wrap up our talk. We presented Outline Pursuits, which is a novel technique for effective
selection of occluded objects in Virtual Reality. Outline Pursuits does not require line of
sight and requires minimal body movements for selection, making it ideal in situations
with restricted movements. By adding to the already existing utility of outlines, objects
can be selected without changing their properties. We demonstrate how combining smooth pursuits
for disambiguation with other modalities for confirmation reduces the parameter search
space and risk of accidental activations, whilst providing more opportunities for the
provision of feedback. Our results show the compelling advantages of Outline Pursuits
for highly occluded targets, which opens up further possibilities of combining the techniques
with alternative selection techniques to allow efficient selection at all levels of occlusion.
And that was all we had for this presentation. Thank you for listening.