- Bernie Peters.
My talk is gonna be on SmoothMoves.
It might feel a bit out of place because it
was planned for another session.
But because it focuses on a selection mechanism
that moves away from pointing hopefully you'll
find it's interesting and thanks again
for accommodating my talk at the end of your session.
My work includes myself of course, from Edinburgh Napier
David from Eindhoven and Ian and his students
from Ulsan in South Korea.
So I think we can all agree that augmented reality glasses
are rapidly maturing technology.
I would also say we can agree that it's quite
still very unclear what's the best way
to interact with AR content interfaces.
Current and viable input modalities
for head mounted displays includes on-set touch
and here user text entry, Google Glass here is
not exactly AR, but the example is still quite good.
You can also interact with mid-air gestures,
such as projected AR or using wearables such as gloves,
or your belt, quite interesting approaches.
Finally, you can also use clickers or proxy devices
such as your mouse.
This is a screen shot from the Meta 2 AR glass.
While these solutions are really, really interesting,
many smart glasses focus in augmenting
a primary physical task which can at times
fully engage your hands are the users hands.
In many of these examples you see here,
the head mounted display ends up serving
just as playback device not really as an interactive device.
As such, a more practical in appealing approach
to interaction with AR is by tracking
users eyes or heads movements.
Not only do the hands remain free
but all sensing can be integrated
into the head mounted display.
The problem with this approach is that
it requires explicit conformation mechanism
to trigger selection.
That is, you know where the users looking,
but not when the user actively wants to
interact with what's in front of them.
A common approach to this problem is to dwell
which enables you to select the target
by simply focusing your gaze on it
for a short amount of time Google calls it Fuse I believe.
The drawback of this approach is sort of
a lack of precision and a fixed time cost.
These quotes are from the developer websites
for both the Meta 2 VR headsets
and the Google Cardboard headsets.
Another approach to entering selections
is just using speech you can do the HoloLens,
but again the drawbacks of this approach
is it's a bit intrusive and especially
in shared work environments and can be hard
to capture in noisy environments such as in a factory floor.
So, this is not mine you can get the reference
there in the bottom so someone else is doing this,
this is from someone else's work.
To address these selection limitations
our paper follows a recent body of work
that looks at gaze input systems based
on Smooth Pursuits eye movements to trigger selections.
Smooth Pursuits are distinctive, continuous,
low latency adjustments of gaze that
are naturally produced only when visually tracking
a moving object.
The Smooth Pursuits systems operate
by showing your user set of moving targets
while tracking gaze and statistical matching
between gaze and the target trajectories
is used to infer which target the user is attending to.
There are two main advantages to this type
of input mechanism.
First, instead of relying on a set of predefined gestures
or speech commands that the user needs
to learn and memorize each control encodes
the necessary input information in their singular movements.
The second advantage is that because
they operate on movements not position,
the Smooth Pursuits interfaces are relatively immune
to target size and if the user can see the target
and track its they can actually interact with it.
You can imagine this the same interface here
if the red, green and purple were quite small
as long as their trajectory is quite distinct
the user could still interact with them.
So the sizes becomes less relevant.
We use this in a previous work on Orbits
that was presented here two years ago
and basically we use this advantage of target size
to create an enter interaction technique
that enables you to interact very tiny targets
on a Smart Watch interface.
But in AR interfaces it could also be used
to interact with faraway targets, like the sort
of lamp these units are.
Which could be hard to select with pointing
sort of just following this Fitz law tradition.
Because you only look at movements we can also
have controls that overlap into this space,
such as these two hanging lamps again
which could be hard selected pointing because
they were overlapping each other.
Finally, Smooth Pursuits input is robust
against tracking inaccuracies.
Capturing changes in gaze is much simpler
than accurately determining where someone is looking at
and also we don't need to go through this sort of
the tedious calibration process to map
the users gaze through to the display.
But there are also several limitations
to Smooth Pursuits input.
First, the wearable eye trackers are rather expensive.
You can see here two examples, one is the Pupil Pro
and these are already the cheapest versions of these.
This is already the Pupil Pro and you can add on
to the HoloLens and on the bottom you have EOG glasses
which compute eye movements and using pairs
of electrodes instead of cameras.
The second limitation is a form factor.
Eye trackers require cameras or electrodes
to be placed at very specific locations on the users face
with the former also requiring a clear line of sight
to the users eyes.
Of course optical systems are also susceptible
to changing light conditions such as those that occur
when using it outdoors.
At the same time when we're doing these user studies
for Orbits, we notice that at times
the system would stop working and this was because
the user was actually started following the targets
with their heads instead of their eyes
and this is because gaze is the inseparable product
of eye and head movements, of head and eye movement sorry.
The relationship between head and eye
is quite sophisticated.
At the most basic level the VOR are continuously
stabilizes gaze by adjusting.
Basically inverting eye position in response
to head position or changes in the response
to changes in head position.
In contrast during Smooth Pursuits,
tracking of moving objects the eye and head
moves together to keep the object optimally in view.
We thought if you can actually track the head
instead of the eyes during Smooth Pursuits type movements
we could address many of the limitations
of wearable eye trackers for AR.
Head movements can be tracked to the IMUs
which measure rotations along three axes
as you see here quite straightforward.
There are several advantages as you can imagine,
to these types of sensors in comparison to optical tracking.
First, is a form factor, the IMUs can be placed anywhere
on the head or smart glasses.
They're quite small and light you can embed
them into many head worn items such as jewelry
or headphones and furthermore these sensors
already embedded in most AR and VR headsets already.
In this talk we just I'm just gonna go over
our four contributions with our SmoothMoves
which is an input technique for AR
that relies on rotation data from head mounted
motion sensors that enables users to select targets
by Smooth Pursuits type head movements.
We also present three studies, comparison
of head movements to an eye tracking baseline,
across a range of target moving conditions.
Performance differences between handhelds
and head mounted base AR using SmoothMoves
and a qualitative user study of AR,
of an AR system for smartphone control
and more specifically as Smart Lights control.
Our work is heavily influenced by prior Smooth Pursuits
interaction techniques but it replaces
the use of eye coordinates on a plane or display
by yaw pitch data from the IMU, quite straightforward.
If the Pearson's correlation between a targets x and y
and the user's yaw and pitch are both higher than 0.8
the target is selected, very, again, very simple
and straightforward.
Visually SmoothMoves closely mimics
our early work on Orbits.
A graphical control is comprised of a trajectory
around the center points in case of AR
the center point could be a physical object
that you want to control or is a smart device
you want to control, and then you have one
or more targets that continuously traverse this trajectory.
As you can see here in the video target disambiguation
is achieved in two ways.
First targets move in different phases
as you see in the video and you can also
have them move clockwise and counterclockwise.
Let's go over first study quite quickly.
We captured both eye and head movement data
in three conditions that you see here.
This is basically what you told participants
to track things naturally, to track things with their eyes,
and track things with their heads.
Eye data was captured using the Pupil Pro
you see here the participant is using.
And as with others Smooth Pursuits input systems,
no calibration was was performed.
Head data was captured using a nine axis IMU.
You see position there on the the edge
of the pair of glasses and we use Mahoney's filters
to extract the yaw and pitch values.
Participants always started in natural condition
but then we counterbalance the other two conditions
and they would see that the moving targets
on an external display, positioned
around 60 centimeters from them.
We recruited 18 participants who completed 432 trials each.
Each possible track combination occurred once
for each of the three main experimental conditions.
The targets varied size, they varied trajectory visibility
so we would show targets with the line
or without line that shows a trajectory.
Speed type, direction types, speed type,
targets would speed up halfway through the trial,
a direction type targets would move in opposite direction
halfway through the trial.
We ran correlations between the all eye and head data
in the three experimental conditions
using different window sizes.
These results indicate participants follow
the target more naturally their eyes than their heads
which is not particularly useful,
it's quite expected I guess.
But the data from the head condition however,
shows that head based tracking can be readily achievable
by participants with even better performance.
And that the head correlation coefficients
were higher than for the eyes in any
of the target conditions as you see in the table here.
Again, you can see all this in the paper
in more detail of course.
The results were also uniform across eyes
and head conditions, basically small trajectories
were harder and targets that change directions
halfway through the trial.
They're basically the hardest one to select.
Finally, we also looked at the scale of head movements
which is quite different from the from eye movements.
Which can impact a range of factors
such as obtrusiveness, social acceptability,
and long-term comfort with a technique.
You see here some heat maps for the trials
that I've highlighted in black.
Participants you can see in line one,
exaggerated the head movements for smaller trajectories
and modestly reduced them for larger trajectories.
These are the means, if you look at medians
the head movement can be very subtle
for the small trajectories, median head rotations
were just 6.7 degrees and so we believe these movements
are sufficiently small to ensure the technique
is discreet and not very fatiguing to use.
On to study two quite quickly.
We calculated also error rates and acquisition times
in two conditions, the head-mounted display condition
which in which users were using the Epson MOVERIO AR glasses
and hand held the display which used a Nexus 6P
which is a more common AR system if you think
of Pokemon Go and all these new AR applications
you can get on your phone.
These were again counterbalanced course participants
and in both cases participants wore
the same head-mounted IMU that we showed in the first study.
We recruited 16 participants who completed 192 trials each.
We re-examined trajectory size as we've shown
to impact performance in the first study
and added target cardinality as a condition.
Shape was equally distributed among starting cardinality
and the remaining variables were constant.
We didn't change the speed type and direction type anymore.
Participants were simply asked to track the target
in red which was chosen at random.
Onto the results, error rates increase
the target cardinality which is not particularly surprising.
The more targets you have the harder they are to select.
If we exclude targets with very small trajectories
and interfaces with over six targets
we report a mean rate of 2.6 for the glasses
and 4.9 for the phone.
If you compare this with similar hands-free approaches
it's quite quite good results.
The acquisition time is around two seconds
and only negatively affected by smaller trajectories.
These two seconds also include the half a second
at the starts where no correlation is populated
to minimize false positives doing visual search.
And again, these results also quite positive
you compare two similar approaches.
In summary there is also the study confirm
the SmoothMoves targeting works well
in two different AR scenarios and in fact,
might be particularly suitable for head mounted displays.
This is useful as these systems already
have all the quite sensors to support the technique.
We conclude with the design and an evaluation
of a prototype they use AR for displaying moving controls
around the Philips Hue Smart Light
and SmoothMoves for inputs and captured using
a normal HoloLens with no modification.
We used Smart Lights we could have used
any other smart device that could benefit
from an AR interface.
The motivation for the the the the prototype
you can see on the right and the idea of the prototype
is really really simple.
You see to the moving controls displayed in space
in proximity to device they control
or in person me to the device they control.
These positions are set once using predefined images
or real-world objects, so you can tag your lights
and then these controls will show.
The controls enable the user to turn the lights
on or off as you've seen in the video playing
or they can access preset light schemes
like a cool energy lights or like a warm relaxing lights.
These controls also enable the user to change
the intensity of the lights and to scroll
through different hue colors using continuous
head movement, so the plus minus and the RGB logo,
if you just follow them you adjust the intensity
or the color.
We recruited 10 participants and we had this lab study
that took over at 30 minutes with each participant
and was based on a participatory design technique
to elicits in-depth user feedback.
In general, participants responded positively
to the technique, trying it is clever, minimalist.
There were some concerns over selection times
or the technique initially required some concentration.
But some of these participants then got better
at the task quite quickly as well.
Participants also appreciated simplicity
of the interface, they can see themselves using it
in home or work-related tasks that occupy both hands.
They also mention accessibility for as a use case
it could be quite useful as well.
Some participants also felt that it
could be easily integrated with other input modalities
such as using course gestures to bring up the movement.
You don't have if you don't have a smart room
that has all of these movement always on.
You could just use the course gestures
to start, to turn off the system or using speech
to provide more detailed instructions when needed.
This is definitely interesting future work
that we want to look into.
And again participants prefer discreet over continuous input
reflecting this general idea the SmoothMoves
is more suited for quick and direct interaction.
And finally the most interesting quotes are,
this relationship between eyes and head
is what makes gaze and these the quotes you see here
it's reinforced the idea that gaze is the combination
of eyes and head motion.
For several participants even with the instructions
to move their heads, these modalities
were hard to separate and distinguish.
As some of the feedbacks, such as doing it with my mind
it's something we find often when people
are describing eye tracking experiences for example.
I was wondering if the results from the first study
where the head data was so poor in the natural condition,
were simply due to the protocol we chose to use.
The fact that users know they were using an eye tracker,
even if we told them just to follow targets
without mentioning the eyes or the head,
the fact that we're using an eye tracker
could have conditioned their behavior somehow.
In conclusion that we can all have coffee.
SmoothMoves was well received by participants
and it was viewed as convenient, relaxing
and well-suited for quick interactions
in hands-free situations and was also deemed unobtrusive.
In contrast with Smooth Pursuits input systems
based on eye tracking, SmoothMoves
can be immediately implemented on a wide range of devices
that feature embedded motion sensing such as
most AR and VR headsets.
I'm going to open for questions, but I understand
if you guys want to rush for coffee.
I'm gonna say behind a little bit
if anyone wants to chat I will just hear
while the video play in the background.
Thank you.
(audience applauding)
(audience member speaking off-mic)
- [Audience member] How do you avoid false positives
because those motion attracts the eye
and you tend to follow them even without thinking
of selection.
- Yeah, that's a very good point.
This is something we were very careful
when we design a purely eye based system
in which the the the target information would be at the core
and then the controls would be orbiting
around the control so that there is this two-step
of actually reading the center piece that
has no actionable information and then opting to look
at the motion.
But that's one of the the challenges here
is movement attracts the eyes.
That's how we evolved and how to train people,
and I think that's reflected in one of the comments
where people say it would be really great
if I could use course gestures just to bring
all this movement up and then I can actually
bring it down because I really can't cope
with these things.
Even though some people find it calming at some point
it is one of the challenges, definitely.
I think these studies always reflect
a very narrow experience when people come into the lab
and they experience this for the first time over an hour.
I will be definitely interesting to see
if people can learn how to cope with movements
a bit better and to avoid triggering things
when they know that it could trigger something.
- Quick question.
How could you, you know how short this
you can reduce the selection.
You say that the selection time is around two seconds.
Is there a way to make it smaller?
- This is why when we did the first study we had,
we looked at different windows sizes
to see all little data you can use,
and again one second is quite standard across
other similar approaches and it seemed to work well,
so it's a compromise again.
- [Audience member] Okay, very good thank you.