- [Euan] So my name's Euan,
I'm from University of Glasgow
and like you just said
I'm gonna talk about "Do That, There".
So, when users want to use meteor gestures
to interact with something
one of the first things they have to do
is find where to perform their gestures.
If they don't gesture in a place where
the sensors can see them,
then their actions are going to
have no effect on the device.
So that makes there show as an example
of how that may happen.
The user wants to gesture towards their
phone on the coffee table
but their hand isn't within
the sensor space for that device.
Unlike other types of input like touchscreen input
or keyboard input the meteor gesture sensing
has a bit of a more vague and ambiguous boundary
between when users are interacting with the system
and not interacting with the system.
Also, uncertainty about where the sensors are located
and what they can see can add to this problem.
Now that more devices and technologies
are able to sense gestures and movement
there's also the problem of having many devices
within the same space sensing input at the same time.
In this situation, shown by the image there,
there's also the chance that the users gestures
will be sensed by more than just the device
that they want to interact with.
And also, there's the risk of ordinary movements
being treated as gestures unintentionally.
So for an example, if the person they imaged
there was to reach forward and pick up their phone
there's a chance that their TV for example
might think that that was a gesture.
So these are problems that occur when users
address meteor gesture systems.
Addressing a gesture system is when you
initiate interaction with it.
And it's the first step of interaction
which happens before users actually do
anything to control the system.
As mentioned in the previous slides,
three usability problems might occur at this stage
are user gesturing in the wrong place,
gestures effecting the wrong system,
and ordinary movements having unintended
effects on a device.
Those last two are more commonly known as
"The Midas Touch" problem.
So, one way to overcome these problems
is by giving users feedback.
A common way of doing this is to visualize
what the sensors can see on screen like shown there.
So the user might see something like a silhouette
of their hand or a silhouette of their body
which they can use to adapt how they gesture.
So for example if they weren't gesturing in a
very good place then they might move their position
in front of the sensor so that their movements
are more likely to be picked up.
But what about devices that are unable to give users
that same level of feedback?
Rich sensing capabilities are being added
to increasingly smaller devices.
Devices like mobile phones and small household objects.
And they have small screens or no screen at all
in which to give users feedback.
So in the case of mobile phones for example
we can interact with them using gestures
but we already struggle to fit good interfaces
on those small screens.
Information there is already quite cluttered
so providing more is just going to add to that problem.
Also the small screens might be difficult
to see from quite far away.
Household interfaces and devices might
also have small screens or
no screen on which to get feedback
so we need to think of another way to do that.
So we investigated interactions for addressing
the gestures systems focusing on devices like these
with limited display capabilities
but the interaction techniques could also be used
by pretty much any meteor gesture system
using TVs or large displays for example.
So we considered the use of interactive light feedback.
This uses simple light sources to display information.
These could be small lights embedded in devices
like the mobile phone shown on the left there.
Or they could be separate light sources
like the lamp shown on the right.
This approach allows devices to illuminate
the area surrounding them creating
ad-hoc displays for giving feedback.
In the case of mobile phones this
keeps the screen free for showing content.
Feedback can be shown over a much
larger are surrounding the device.
For simple objects with no display at all
this allows visual feedback to be given
when that would otherwise be impossible.
We also considered using tactile feedback.
So tactile feedback during meteor gestures
can be challenging because the user
doesn't actually contact the device
that they're gesturing at.
So we used vibra-tactile feedback from wearer devices.
Devices like smart watches or activity trackers
which have this capacity
which we could use to provide
information during other interactions.
Finally we also used audio feedback
because it's an easy output modality
to provide in terms of technical challenges.
So we developed an interaction
technique called, "Do That, There"
which can overcome some of the usability
problems which I spoke about earlier.
And it uses the three types of output
which I just mentioned.
This interaction has two parts.
"Do That" which tells users how
to direct their input towards the system.
And "There" which help them find
where to perform the gestures.
I'll now talk about each part separately
starting with "There" and bring them
both together later on.
So the data interaction helps users find
where to perform gestures so that there
input can be sensed more reliably.
It does this by giving users feedback
which tells them how well they can
be seen by the sensors.
So for camera based systems this might be better
when the user is closer to the sensor field of view
and also an appropriate distance from the sensor.
This type of feedback also lets the user know
that they can be sensed and that they system
is tracking them and paying attention to them.
So we use brightness of light to show users
how well they can be seen with the light
becoming brighter as they gesture in a better position.
Audio and tactile feedback used a Geiger counter metaphor
with repeated tones becoming more frequent
as they approach a better position for performing input.
These designs map just one dimension
of information to feedback so it doesn't give
explicit instruction or spatial cues
like move left or move closer.
So the idea is that the users have to explore
the space themselves and form their own
understanding of how to have more success with interactions.
The paper has more details about how
we arrived at these designs.
So our first study of value to this feedback
to see effective it was for guiding users.
We did using a targeting task
where participants had to find target points
positioned over a mobile phone.
LEDs around the edge of the phone gave light feedback
with tactile cues being delivered through
an actuator worn on their wrist.
Audio feedback came from a loudspeaker.
So we looked at each of those feedback types separately
as well as other combinations.
The target task shown is shown in the video there
was the users had to explore the space over
the phone using extended input finger
using the feedback to guide them towards the targets
and when they thought they found that
they used their other hand to
press a button to end the task.
So we found that users could locate points
with 51 millimeters accuracy.
We found that feedback modality had
an impact on the accuracy with better
performance when conditions included audio feedback.
At the same time these conditions were also
slower than those without audio.
When light feedback was given accuracy was
not as high as with audio but
the interactions were quicker
and performance was best when
these types of feedback were combined.
So some participants explained that
they would use the different types
of feedback for different things.
Using the light to initially position their hand
before honing in using the audio or tactile feedback.
So next the "Do That" part of the interaction.
This tells users how to direct their input
towards a particular system or device.
This helps to overcome "The Midas Touch" problem
because the user shows which system they
want to interact with and also
shows an intention to interact.
As an example of why that's necessary
the image there shows how multiple devices
might be sensing input from the same space,
potentially all of them acting upon
any movements that the user makes.
By directing input towards one system in particular
the others know that any actions that follow
are not intended for them.
So for this interactions users directed input
using something we call rhythmic gestures.
Rhythmic gestures are gestures are repeated
in time with a rhythm shown using an animation.
Those animations might be shown using
something like the LED display shown
on the previous slides
or they could also be shown on screen
if that was available.
By combining different hand movements with
different movement speeds we can create
quite a large design space just from a few simple gestures.
So we chose five hand movements for the
gestures shown on the screen there.
That was waving the hand from sided to side,
moving up and down,
forwards and backwards,
and moving in a circular path.
We also selected four speeds for those gestures
which gives a total of 20 gestures.
So by assigning different movements and speeds
to different devices users could direct
their input towards one by performing
the appropriate gesture.
For example, if they wanted to direct their input
towards the mobile phone there they'd move
their hand from side to side in time
with the animation as it moves across the table.
Effectively users are using this to make a selection.
These are examples of what the animations
might look like if we were using a simple circular
LED display around a dial.
Basically the movement of the light shows users
the movement that they need to make as well
as the speed at which they to replicate that movement.
So our second study evaluated how well
users could actually perform this type of gesture.
We wanted to see if they'd be able
to move in time with the animations
and at different speeds and we were also interested
in what role feedback had on their gesture performance.
So we get four types of feedback about gestures.
One just used the LED animations to show them how to move.
The other three gave additional audio
and tactile feedback which was basically
given a short tone or a sound at the end
of each part of that individual gesture.
The details about how we came to that's
in the paper as well.
So the task in this experiment was to match
a rhythmic gesture which means to perform
the correct gesture in time with the animation
for a minimum period of time.
That was approximately three full movements
in sync with the animation.
We gave users a limited amount of time
in which to complete each task
and we measured the success rate of that,
how long it took them,
and we also got them to give
difficulty ratings for that as well.
Unlike in the previous experiment,
we used a wall-mounted circular LED
with a connect for gesture sensing.
So, users performed almost all
rhythmic gestures successfully.
The success rate was 93%.
Success was variant across gesture movement and tempo.
There's a paper being presented tomorrow
by another group called path sync
which presents a similar interaction technique
and they got very similar success rates
as well which I thought was quite interesting.
In our study, the two circular gestures
performed poorly especially the fastest movement speed
which are highlighted in red.
In contrast the side to side and up and down gestures
were performed the best with almost 100% completion rate.
In the path sync paper they also found
circular movements were not performed as well
as more rectangular trajectories
so there might be an interesting reason for that.
Generally rhythmic gestures with the slower movement
speeds were more successful with the
success rate decreasing as the gestures got faster.
The results showed that feedback had no
effect on gesture success.
So participants didn't perform any differently
when given non-visual feedback about their movements.
So, users took just over two seconds to match each gesture
with times there normalized
in the 500 millisecond interval.
Feedback had no effect on time
but as the graph shows interval and movement did.
So the gestures with shorter intervals
or faster movement speeds took longer to complete
with the most noticeable difference with
the shorter interval there.
The circular ones, also highlighted in red,
took longer to complete than the other three movements.
So, finally we combined the two interactions
into one technique called "Do That, There".
Finding where to gesture and directing input
at the same, directing input towards
the system can happen at the same time
and interaction would be quite cumbersome
and time consuming if users had to do these separately.
More experienced users are also more likely
to know where to perform gestures for a system
so they can just skip this step and go straight
to directing their input instead.
So with the combined technique users gesture
to direct their input while also using
the feedback from the system to adapt
where they're gesturing so that they
can be more accurately sensed.
As the video shows there,
the gesture animation changes in brightness
based on where the user's moving their hands.
So when the animation's at it's brightest
it's because they're gesturing in a good position.
So each of the interaction is it's own,
has it's own feedback which can
be combined in different ways.
One option is to just give both types
of feedback and all modalities at the same time.
So with this the gesture animations would
be brighter when users gesture in a good space
and more difficult to see when
they're gesturing in a bad place.
The audio and tactile feedback could be delivered
at the same time as well.
An alternative to this is to use different
modalities for different types of information.
So, since light is needed to show
the rhythmic gesture animations for directing input
we could use audio or haptic cues instead
to help users find where to gesture
using the feedback which we used
in the first experiment.
So we looked at this design problem
in the final study which I'll go into in a second.
Another design question when designing these techniques
is how much feedback do users actually need
to tell them where to gesture.
So, once users have actually started performing
gestures and have started to interact
with the system they might not benefit
from having this extra feedback which
tells them where to gesture.
Instead, it might be sufficient to stop
giving that feedback once the gesture has been progressed.
So with these questions in mind we came up
with five feedback designs.
They're explained in more detail in the paper to save time.
These designs would allow us to investigate
two questions in our final study.
The first is, how should we present two types
of feedback using multiple output modalities?
So, should we combine that across them
or divide the information between the two?
And, do users benefit from feedback telling
them where to gesture all of the time
during interaction or would it be sufficient
to stop that once they start to perform them?
So, our first study investigated how well users
could use this interaction technique
as well as investigating the effect
of the different feedback designs mentioned before.
This was basically a combination of the past two studies
so users had to match a rhythmic gesture
while also performing it as close
to a target point as possible.
There are two factors in this study
which are feedback and gesture.
So we chose the two best gestures from the last study
which were moving the hand from side to side
and moving the hand up and down.
And we had those performed at the 700 millisecond
movement speed because that did quite well before.
So, in this experiment users matched 99.9% of gestures
which is similar to the completion rate
for those gestures in the previous study.
They also gestured within 80 millimeters
of the target point.
It took them just under four seconds
to find the target point and complete a gesture.
And that four seconds includes just over
two seconds spent matching the gesture animation.
Interaction times were the lowest when
the feedback was combined across modalities.
This might have been because the visual feedback
directly effected the visibility
of the gesture animations.
So that have might have had a more noticeable effect
when users where trying to follow that.
So for example, that means if your hand was
in a bad position the animation would be
more different with the scene so it
compels users to find that space
before they actually start to provide the input.
We also found that users did not necessarily benefit
from getting feedback which tells them
where to perform gestures all of the time.
So the highlighted results there
show the designs of where the feedback stopped
once the gesture started.
It wasn't that much different
to the other designs we discovered
throughout the entire interaction.
So in summary our "Do That, There" interaction
seemed to be successful.
Users completed almost all of the gestures
and did so within 80 millimeters of the target points
in that third study.
That's quite good considering that they were
gesturing with quite large arm movements
from the opposite side of the room several meters away.
Interaction time's good as well.
Users found where to gesture
and matched the correct gesture within about four seconds.
That is they've been able to address
the system within four seconds.
In terms of feedback design
we've found that it was better to combine feedback
across all of the modalities rather than give them
different types of information from different ones.
This seems to have come down to the more
noticeable effect of the light feedback
on the animations as mentioned before.
We also found that we could stop giving feedback
telling users where to perform gestures
once they had started providing input.
So, to conclude, users need to address
meteor gesture systems so that their
input can be sensed and so that there gestures only
effect the system they intend to interact with.
In this paper we present, "Do That, There"
which is a technique for addressing meteor gesture systems.
This can be used by a variety of system form factors
without needing the screen for feedback.
This means that it can be used by typical gesture
systems with large displays as well as
emerging gesture systems like mobile phones,
small objects or in-car interfaces.
Devices which are more limited in their ability
to provide feedback but which increasingly
have richer sensing capabilities.
Thank you. Any questions?
(clapping)
- [Voiceover] Hi. Thank you for the informative talk.
This is Kai from UMBC. I noticed between study one
and study two you went from a single finger gesture
to an arm or a whole hand.
Did you see any reduction in accuracy from
the single finger which required you to go
to a larger area in terms of the whole hand for gesturing?
- [Euan] No. So the reason we looked at those different
types of gesture was to cover a broader class of devices.
So not just focusing on mobile phones
but also small objects which you might interact with
from the other side of the room like
light switches, music players,
thermostats, that sort of thing.
So in the first study we found that users performed with,
I think it's 51 millimeters accuracy for those
small finger movements and the accuracy
of the much larger arm movements from farther
away was 80 millimeters so there wasn't that
great a difference between the two.
There wasn't any reason we made that switch.
That's just part of the design to cover those
but yeah, the accuracy was similar.
- [Voiceover] So there seems like really well
design focusing on when they're actually trying
to manipulate the device but if I walk into
a room and the thermostat's glowing at me
and the speakers are glowing at me.
Have you thought through that side of it as well?
- [Euan] So, do you mean in terms of their obtrusiveness?
- [Voiceover] Yeah, when they're distracting, yeah.
- [Euan] So the one thing, in this is was just interacting
with the one device.
So, you know something I'd like to do in the future
is actually have multiple devices there
showing those animations at the same time
and seeing if those competing animations
would be distracting and effect the input.
Research from psycho motor studies
on sensory motor synchronization which is moving
in times with basically LED animations
like the rest suggested,
distractors have a small but not much
of a noticeable effect on that.
I would guess that it wouldn't be a problem
but I think that research still needs to be done.
Thank you.
(clapping)