- Thank you for the introduction.
Good morning everyone.
My name is Hyunchul Lim
and I'm a research assistant
at Human Centered Computing Lab
at Seoul National University, South Korea.
Today I'm so pleased to present my work named Touch+Finger.
This work is in cooperation with my advisor
Professor Bongwon Suh and Professor Joonhwan Lee
and three PhD students,
Jungmin Chung, Changhoon Oh, and SoHyun Park.
At first, let me introduce Touch+Finger briefly.
Touch+Finger is a new interaction technique
that augments touch input
with a multi-finger in-air gestures
for reach and expressive interaction.
Let me show you a quick video
demonstrating the basics of Touch+Finger
for a better understanding.
As you see in the video,
the main idea of Touch+Finger
is to use idle fingers.
What I mean is that
when one primary finger is engaged in touch interaction
the rest of the secondary finger
are in idle states.
So we call them idle fingers.
Since they are able to move
a user can use these idle fingers to add modality
and expressiveness to the primary touch interaction.
For example, by making different hands pose before a touch
a user opens a file by tapping with a basic hands pose.
Or open a context menu of the file
by tapping with a spread-over hands pose.
Also, as we saw in the video before
a user can use the side of the finger to make
an expressive brush stroke in a painting application.
So, why I have introduced the amazing idea of Touch+Finger
here I go through the motivation and related works.
Reasons to re-invent the finger aware shortcut
includes the input space on the keyboard
by pressing same key with a different hands pose
for different shortcut commands.
I will work closely with this work in that
it also use the rest of the idle fingers.
But our work focuses on touch-based devices
rather than a keyboard.
Also many research has been explored in increased
input of vocabularies by combining... oh sorry
by combining touch and gestures.
For example, touch interaction can be enhanced
by regulating additional information
such as in-air gesture with a thumb
or with a touch screen on a smartphone.
Or the thumb based touch gestures on sensitive
touch sensor in a ring-like devices.
Or the movement of the wrist by using
ultra sensory microwaves.
However, based off our knowledge we are not aware
of any existing work that enhances the primary
touch interactions by making full use
of these idle fingers.
So, our goal with this research is to explore how
the rest of the other fingers can be used
as additional information,
for enhancing the primary touch interaction.
To do so, we created a design space, covering a variety
of interaction techniques with these idle fingers.
We aimed to augment existing two dimensional touch gestures,
by using idle fingers.
So, we have divided a combination of touch gestures
into two categories.
Single-touch, and multi-touch.
In this paper, for single touch we used an index finger
as a primary finger, and for multi-touch
we used a thumb and an index finger
as the primary fingers.
In this case, the secondary fingers are the thumb
and/or the middle plus fingers.
This is middle plus fingers.
The middle plus finger means that the motions
of the middle, the ring, and
the little fingers are together.
Also with touch interaction, we divide the touch
of interaction into two dimensions,
before touch and the during touch.
Before touch is the period that's just before
the primary finger touched the screen.
In this case, the secondary idle finger
can make a different hand pose
before a touch event.
During touch refers to the time when
the primary finger is touching the screen.
In this period, a secondary in-air gesture
with idle finger enhanced the primary touch interaction.
By combining these two... two factors,
we created the two-by-two design space
of before single-touch, before multi-touch,
during single-touch, and during multi-touch.
So based on the design space, we determined
all possible motions of the secondary finger
in each sub-section.
Now, let's look at before single touch gestures first.
This is a basic hands pose,
and this is a stick hands pose,
and bend hands pose.
This is a L-shaped hands pose,
this is bend all hands pose.
This is a spread all hands pose.
This is all different by using either fingers.
The six different hand poses can be made
with the secondary idle finger before touch
with the primary fingers.
In the case of the different multi-touch gestures,
you just can make a limited number of hand poses
due to their anatomical limitations.
This is a basic plus two hands pose,
this is a bend plus two hands pose.
This is a middle plus fingers bend.
This is spread plus two fingers, sorry,
spread plus two hand pose.
So there are three different hand poses
while performing multi-touch gestures
with the thumb and the index fingers.
Now we look at during single-touch gestures.
In this case, let me give you a video
demonstrating each in-air gestures.
This is tap.
This is a flick gesture.
It's a bending.
It's swipe up.
Swipe down.
Bending plus.
A flick plus.
Bending all.
Flick all.
A user can perform nine in-air gestures
while performing single-touch touch gestures
such as tap, and drag on the screen with the index finger.
Lastly, let's look at during the multi-touch gesture.
In this case, the only secondary fingers
are the middle plus fingers.
So there are only two in-air gestures available.
This is flick plus two.
This is a bending plus two gestures.
In this case, only standard multi-touch gesture
can be performed while performing secondary
in-air gesture.
This is because we found that,
it is difficult for middle plus fingers
to perform in the air gestures while simultaneously
performing a dynamic gesture with a primary finger.
So this figure shows all of the touch plus finger
gestures we explored in our design space.
All of these gestures are not user-friendly.
So we conducted a user study to investigate
the easier of the performing the gesture
in evaluation section.
Now, we talk about a touch plus finger prototype
to implement the touch plus finger gestures.
Our prototype consists of three parts,
a ring-like device with an IMU sensor attached
for detecting hands pose or in-air gestures.
A touch-based device, in this case, a tablet,
for getting touch input information
and the external PC for data processing and classification.
Our prototype was simple, robust, and reliable
enough for initial exploring
of touch plus finger gestures.
This shows our classification model.
The calculated IMU sensor based and touch information
are used to us for classification.
We tested several basic classifiers
such as decision tree and SPN
And then, the decision tree classifier
achieved the best results in our pilot test.
So decision tree was applied to our prototype.
With this prototype we evaluated
the touch plus finger gestures.
We measured two things.
One thing is the recognition accuracy and
the other one is the user rating
of each gesture with five point scale
and a semi-structured interview.
For the (mumbles) we recruited eight participants.
They are asked to wear our prototype and then
perform 20 touch plus finger, 10 times for 10 seconds
for steady and dynamic touch gestures
with primary finger.
We collected 2,080 gesture samples
in total for classification.
This procedure is a little bit complicated, I think.
Detailed explanation is given in the paper.
And we also asked the participants
to rate it properly,
how easy the gesture was to perform
on a scale of one to five.
The results.
For gesture classification, we conducted two validations.
Overall touch plus finger gestures
were classified very well,
with high accuracy of over 99%.
It means that the participants
performed touch gestures in an almost identical manner.
Next, participants gave both
positive and negative feedback to
touch plus finger gestures.
For before touch gestures,
they were able to make all of
the hand pose easily except for L shape and the bend.
This is because the participants
thought that this gesture are uncomfortable,
requiring them to put in more effort
to maintaining the poses.
For the ring touch gestures,
there are significant differences
between static and dynamic gestures with primary finger.
While performing the static gestures with primary finger,
in this case a tap on the screen,
participants were able to perform mostly
in-air gesture without any problem.
Except for bending, flick plus,
flick all or flick plus two.
Some participants thought that
these gestures were quite stressful.
On the other hand, while performing
dynamic touch gestures with primary finger,
for example drag on the screen,
participants report that most in-air gesture
were very difficult.
This is because secondary in-air gestures
draw much attention away from
the primary touching interaction.
So participants reported that only a few in-air gestures
such as tap and swipe up and down
are easiest to perform.
Now, let me show you a video demonstrating
some of the examples of
a prime touch plus finger technique.
This is a free-form line with a basic hand pose,
and a straight line with a stick hand pose.
Also a curved line with a spread all hand pose.
So we can fluidly switch between interaction modes
in a drawing application.
So here we talk about the discussion.
We think that our prototype is enough
for initial exploration of a touch plus finger.
But there is still room for improvement.
We believe that our ring-like device
can be more miniaturized so that it can be
a stand-alone wearable device.
Also, some participants pointed that
wearing multiple rings may be impractical.
We think that this could be solved
in the future by exploring alternatives
such as depth cameras or capacitive sensors.
And we found that there is a trade-off
between discomfort of performing
secondary in-air gestures and the
high classification accuracy.
Interestingly, we think that the
fifth primary finger on the screen
may result in discomfort of performing in-air gestures.
But it also restricts the possible
secondary fingers movement allowing users
to perform the gesture almost identically.
It could help to achieve the high accuracy.
Lastly, in the picture work,
it would be interesting to explore
potential combination of touch plus finger gestures
on other devices such as small screen devices,
virtual headsets and laptop interfaces.
So we conclude that by using idle fingers
as additional (mumbles) for primary touch interaction,
we can enhance the touch interaction in
the various ways we explored.
Thank you for the listening and
I'm happy to take your questions.
(audience claps)
- [Announcer] We have time for one or two questions.
- [Audience Member 1] - Hi, Dan Ashbrook,
University of Copenhagen.
With the two accelerometers, the two IMUS,
can you ignore the angle of the hand?
Or do you always have to be in one hand position?
- [Lim] Actually in our experiment,
we did ignore the angle of the (mumbles) variables.
This is because we just conducted
our usual studies in our controlled environment.
- [Audience Member 1] Right.
- [Lim] So.
- [Audience Member 1] So do you think that
this would work while walking?
- [Lim] Actually our work is focused
on exploring the gestures and so
I think that could be addressed in the future work, I think.
- [Audience Member 1] - Okay.
- [Lim] Yeah, thank you.
- [Audience Member 1] Thanks.
- [Announcer] Any other questions?
- [Audience Member 2] So I have one question.
Have you thought about instead of wearing
the rings on the finger that's touching the screen,
wear it on the other hand and
whether that could improve other types of techniques?
Cause I could basically imagine
touching the screen whilst scratching my head
and have that as a new interaction technique.
What do you think about that?
- [Lim] I think that's a good question and interesting.
Actually, I didn't think about that.
- [Audience Member 2] But you think it's possible?
- [Lim] Yeah, yeah, I think that it's possible
using multiple rings wearing in the other hands,
and for more expressive interactions, maybe.
I appreciate your comment and in the future
I will address it to you.
- [Announcer] Great.
Let's thank our speaker one more time.
(audience claps)