- Good afternoon, everyone.
I'm (mumbles), presenting our work,
Designing a Willing-To-Use-In-Public
Hand Gestural Interaction Technique
for Smart Glasses.
This is an cooperation between
Helsinki Institute for Information Technology,
Helsinki University,
and also (mumbles) University.
I would like to show a video first
to illustrate the problems when using
smart glasses in public space.
- Banana.
Banana.
(audience laughing)
Banana.
(audience laughing)
Yeah, so, as you can see in this video,
current interaction techniques
for smart glasses can be so obtrusive
that people don't want to use them in public.
So we surveyed the possible techniques
and also proposed design principles
in this paper.
We also proposed a solution that is
it studies a glove system
enabling independent tracking
of the hand
and also provide tactile feedback.
The system was evaluated in public
and with three distinct interaction scenarios,
and the results shows that people are able
to complete all the tasks in an unobtrusive manner,
and they are willing to use them in public.
So now let's go back to the problems
and the motivation.
Smart glasses have already emerged into market,
especially this two big companies
proposed their design
and also their envisioned future applications,
but actually there are so many different variations
of the glasses and also manufacturers
producing the smart glasses,
and is actually now a commercially available product.
But then we have this two question.
How are we going to interact with them,
and how acceptable are they?
For example, Google Glasses use this voice control
and also touch and swiping gesture on the frame.
(mumbles) interaction is very limited.
Another example is this HoloLens.
Their air tap just requires you to point
your hand up,
and then bend your finger
to make a click.
The camera is installed on the glasses,
so the gestures have to be performed
in front of your face,
which results in fatigue
and also getting too much attention.
So but this example can also
too much attention
either disturbing others
or is too embarrassing to use in front of others.
So we targeted this social acceptability issue
since if it's not socially acceptable,
people merely don't want to use them.
So there are several possible techniques
that can be categorized in this three types.
The first one is touch and handheld.
For example, Absen has this handheld design
with touch pad interface.
This can enable unobtrusive interactions
since you can even use the device with your hand
in your pocket,
and you can also make very subtle finger gestures,
but this design requires the users
to hold an additional device in the hand
while high mobility and also high degree of freedom
should be the major concern in this context.
Moreover, the direct interaction becomes difficult
because the visual contents are no longer touchable.
Another option is to use hand free techniques,
for example, gaze input,
facial recognition,
and also head gestures.
Only slight muscle action is involved
in this type of interaction,
but that requires excessive sensors
wear on the head,
and also require much calibration and is prone
to tracking errors.
Another type is on body gestures.
One body gestures requires users
to perform a gesture in physical contact
with part of the body.
Touching on the body can be of less concern
since we usually do that unconsciously.
However, the interaction might be limited to touch events,
and in this case, palm type,
that requires both hand.
We have already seen examples of mid-air hand gestures
that can get too much attention,
but still we want to go back to this
because it has the potential for enabling
natural and intuitive interaction,
just like we're interacting with our daily life objects,
and it is actually the technical limitation
that causes this social acceptance problem.
Most techniques are based on the computer vision,
and that has these two problems.
The first is occlusion,
that the system simply cannot truthfully
recognize your gestures,
and also viewing angle,
that you have to perform your gestures
within the viewing angle of the camera.
In the example of air tap in HoloLens,
rather than using the hand to interact
with the visual content,
it's more like just showing the gesture
to the camera,
and it is possible to wear the camera
on other parts of the body
so that you don't have
to perform the gesture up in the eye level.
But again, this example relies on computer vision,
and that does not solve occlusion issues.
So we propose to use sensor equipped
wearable devices around the hand.
This is our design.
(cheerful electronic music)
Our sensor based glove use
initial measurement unique for tracking
hand orientation,
and also on the fingers we have flex sensors
to track the finger pose.
At the same time, we also vibrotactile
actuators for providing tactile feedback
on the fingers,
and also Bluetooth module
for wireless communication
with the smart glasses.
The distributed sensors on the hand
enable tracking both coarse arm movement
and also fine finger gestures.
Additionally, the tactile feedback can be applied
directly on the action-performing body.
Both of these advantages are not possible
on other form factors such as rings or wristbands.
The basis of this interaction
is built on mapping a reference orientation
to the center of the display,
and active region can be divided accordingly.
This technique allows users
to set a reference orientation
in any pose freely,
which enables the gesture to be performed
in a comfortable and less obtrusive pose,
and it's easy to adapt to various conditions.
I will introduce three common scenarios
that we develop and is quite common
in mobile applications,
text entry,
selecting one from many,
and scrolling.
This is a video of showing how the interaction
is like when using a glove with smart glasses.
Please note that even though it looks like
the hand is in front of the face,
but the glove can be actually used in low pose.
Here we're just trying to show
both the hand gestures
and the visual interface at the same time.
So as you can see,
three keys are grouped together,
and they are highlighted as group
according to your hand movement,
and each finger is associated to one of key
in the group.
To type, you simply bend one of the finger,
and we have this design to fix
the key group composition,
so each key is always associated to a specific finger.
In long run, with muscle memory,
can improve your performance,
and also we have special design for the margin
that this is to filter out
either sensor noise
and also the hand jittering,
especially between the edge,
between the groups,
so it's a bit feel like speaking
when you are moving in between the key groups,
and the glove also provide tactile feedback
when you are typing.
So when you hit the key,
in the visual finger,
you can feel a vibration pulse,
and also, when you cross the edge
of the group, the key group,
you can also feel a vibration pulse on the finger.
The second one is selecting single from many.
The system is not activated until a pointing gesture
is recognized,
and one of the item will be highlighted
according to pointing orientation,
and by bending the index finger,
you can select an item.
Rest of design is grab and drag it gesture
to activate a swiping action.
Again, the same vibration is applied here,
so when you are selecting an icon,
you can feel vibration pulse on the finger,
and also when you are moving
between different items,
and the third scenario is scrolling.
Again, this system is not activated
until you make a grabbing gesture,
and the inclination of the hand
triggers scrolling up or down,
and in this example,
we can see a hyperlink showing
on the top right corner.
To select the hyperlink,
you make pointing gesture,
and move your hand toward that direction,
and our idea is that this system
will not require you to move
from this current orientation
and all the way to the far end.
When the system recognize that
you are making a pointing gesture
and you moving toward that direction,
we can already highlight the hyperlink
so that you don't have to move that far,
which will get much more attention
than just a slight action here,
and again, to bend the index finger,
you can make a selection.
In the evaluation,
there were there tasks
built around these three scenarios,
and the way to evaluate the social
acceptability is we adopt this questionnaire
proposed by Rico et al,
and in their design,
there's only this
interface showing a video
in which there is a man
performing the gestures,
and on the side, there are these questions
regarding in which locations
or in front of the different types of audience,
whether you are willing to use
these gestures in front of them in these conditions.
But in order to get more authentic responses
from the subjects,
we conducted this study
in public space.
And this is the result from the measurement
presented with acceptance rates,
and the red area shows how unwilling
the subjects were when using the technology in public,
and the blue code is for positive responses,
and the white part in between red and blue
is neutral responses,
and the three bars are grouped together,
each representing one of the tasks.
So from left to right is
scrolling, select one from many,
and typing.
Let's have a closer look at the responses.
So regarding the location,
the more private location is,
the more willingness there is
in using smart glasses with these gestures,
and we can see on the right side,
there is this work place,
which should be also considered a public space,
but in this context,
it's actually more like little bit seen
as using smart glasses for professional purposes,
so it's still higher acceptability.
And the audience is also a factor effect
in whether you want to use the techniques or not.
As you can see,
in front of strangers,
not so comfortable for users to use these techniques.
Especially in the middle bars,
we found the least acceptance rate
for the middle bar,
which is the selecting single from many
that involves much of pointing gesture.
So the next one is
we achieved this unobtrusive interaction.
In an earlier elicitation study
where the subjects,
they see an animation of visual interface
in front of the eyes,
and they invent or create
their own hand gestures
to correspond to this visual interface,
and in the study,
they found that 37% of hand gestures
were performed in front of the face.
But in our study when there's actual
and real interaction happening,
none of our subjects performed the gesture
in front of the face,
and especially in public space,
and the subjects were able
to complete the task in an unobtrusive manner
by keeping pose low.
For a text entry performance,
we can see like in the blue line
shows the text entry speed
in words per minute,
and the red line shows the error rate,
and within only 20 minutes,
our subjects can reach 5.4 words per minute.
And we can compare with other techniques.
For example, this palm type,
it is also a hand worn text entry technique
for smart glasses
that reach 4.6 words per minute
on their IR sensor-based prototype.
Another example is using swiping gestures
on Google Glasses.
They averaged 8.7 words per minute,
but the results comes from on the last block
of an 80 minute experiment,
and they didn't require subjects
to correct any errors.
So we do think our subjects could continue
to improve their performance over time,
but that also requires a long term study.
We also found that tactile feedback
is useful for subjects,
and help them feel the visual interface more
in a more tangible way,
and just to remind you,
we have this two places
that apply the tactile feedback.
The first one is when making a selection
in the visual finger,
and also when crossing the edges.
I think most of people have this question.
Would people be willing to wear
an additional garment?
Well, what we envision is an integrated
wearable system with more than one sensor on bodies,
so that offers more interaction possibilities.
For example, there are more people wearing smart watches
just like smart glasses
coming more and more acceptable.
To conclude, we found that obtrusive interaction
would affect willingness of using
new techniques, technologies,
and our hand-worn proposal,
our hand-worn solution,
uses independent tracking technique,
and also provides tactile feedback.
Our subjects were able to complete
all the tasks in an unobtrusive manner
by keeping the pose low,
and also they are willing to use
the system in public.
I would like to show the happy ending.
(cheerful electronic music)
- [Man] Banana.
(audience laughing)
- Thank you very much.
(audience applauding)
Any questions, comment?
- [Man] I just had a very specific question
about why you didn't put the glove
on the other two fingers even though,
I know you weren't using them as input,
but I'm just curious why you made that decision.
- So what do you mean?
The (mumbles) fingertips?
- [Man] Why the glove didn't enclose
the other two fingers?
Seems like an interesting aesthetic choice,
I was just curious.
- Just because I guess everyone has smart phone
with touch interface,
so to make it more versatile,
we decided to leave the finger out
so that you can also at the same time
use different devices.
- [David] Hi, I'm David Auberstein
from (mumbles) university.
So I was wondering,
you were using the sitting condition, I think,
most of the time.
Did you actually like try
how the gestures would work
and how unobtrusive they were
if people were standing or walking?
- We did the study on like when people are sitting,
and we didn't consider the mobile cases,
but in our questionnaire,
there's an entry that's asking people
whether you want to use this kind of techniques
when you are as a passenger,
and also on the sidewalk,
and the results also show like very low acceptance rate
because it's not practical, like here.
- [David] Okay, I have another question.
Did you actually like try to do it
at a public space?
Like, let's say not in front of lab,
but like in a park,
or in a public scenario,
and look at like how bystanders perceived the interaction?
- You mean in our study?
- [David] Yes, or like, did you think about
like extending your work or in future work doing that?
- Yeah, I think that's possible,
but that's just a way to,
because there are so many different conditions,
and we are not possible to cover all different conditions.
For example, it can be a cafe.
It can be a bar,
but what's more important for us
is to verify whether,
to give the users an opportunity
to use the technique in real environment,
and then just to see their responses.
- [David] Okay, thanks.
- Thank you.
(murmuring)
- [Man] It's sort of a follow up on that.
So you defined social acceptability
by how the user perceives
their willingness to use it in public.
How about bystander perception?
- Yeah, we're aware that the social acceptability
should be considered two parties,
but unfortunately we didn't include
that part in our study,
but we are (mumbles).
Thank you.