- Thank you, sorry for the technical difficulties.
Welcome to my presentation, my name is Mohammed Sami Uddin,
and I am here to present our work on HandMark Menus,
a technique for rapid command selection
and large command sets on multi-touch displays.
And my co-authers are Carl Gutwin and Benjamin Lafreniere.
In this presentation I am going to show how hands
can be used as powerful landmarks in large tabletops
for fast command selection.
Landmarks.
So landmarks are basically easy to remember, iconic places.
They help us to develop spatial memory
about their surroundings.
We can easily remember different objects
and places with reference to landmarks.
Different graphical user interfaces have leveraged
the concept of landmarks to develop spatial memory
of starting commands.
So they help us to remember those commands
and revisit those commands quickly.
Take this image as an example.
So when we want to close an open window,
we always what to look for that cross button.
So based on the operating system, it would either
on right side or left side on the corner.
So here, corners of the screen are landmarks
that help us to remember commands' location.
So there are some other techniques which also have
leveraged this spatial memory techniques of landmarks
for fast command selection.
FastTap menu technique for tablets is a good example of it.
FastTap uses a spatially stable grid to place commands.
So here, corners and grids are working as landmarks.
So they help users to remember commands' locations.
So once the locations are learned,
users can use landmarks and spatial memory
to access those commands quickly.
However, that technique is not feasible for large tabletops.
Large tabletops mean less number of landmarks.
So corners are still there, but what about the commands
placed at the middle and edges of the screen?
So fewer landmarks in tabletops make it really hard
for the user to remember the commands.
But for any kinds of tabletop interactions,
so one tool is always common, that is our hands.
So hands and finger are really powerful natural tools
so that they're always available
in all touch-based interfaces.
We carry them with us all the time, and we are very much
familiar with our own hands and fingers.
We know their shapes, sizes, and positions accurately.
We can even touch a particular finger of one hand
with another hand, without even closing our eyes.
We know them very accurately.
So why not to use the all-time present natural artifacts,
hands and fingers, as landmarks
for better tabletop interaction?
We use hands as landmarks on tabletops,
we have created a new technique called HandMark.
It helps the users to learn commands' location
by using the proprioceptive knowledge
of their own hands and fingers.
As commands are placed around and between
the places of fingers, so users can remember
commands' locations with reference to them.
We tried in different ways and came up with two
versions of HandMark techniques.
The first one is HandMark-Finger,
you can see on the left side.
It's for relatively small command sets.
And another one is HandMark-Multi, so using the right side,
so that supports the relatively large command sets.
The details are provided in the paper,
here I am going to give you a high-level overview
of our techniques.
So HandMark-Fingers.
HandMark-Finger is a basically bimanual technique.
Naturally the commands are hidden from the users,
so this technique requires one hand
to invoke the command sets and another hand
to select the commands.
So each provides modal access to two sets of commands,
each belongs to one hand.
As you can see in the video, we placed commands
around and between the fingers of both hands.
A single side has 21 commands, so by combining two hands,
so it can support 42 commands.
So this technique can be used in two ways,
at the beginning, when users are at the novice stages,
so they can complete one task
by following three simple steps.
First, they can invoke a set of command by placing
all the five fingers of one hand,
keeping some spaces between the fingers,
and then search visually for a desired command, and finally,
execute the command selection with another hand.
Here you can see, yeah.
And once the locations are learned, users become expert,
and as you can see in the video,
expert users can combine menu invocation
and command selection into single step,
to make the process faster.
Here you can see.
Now the second version, HandMark-Multi.
Similar to the previous HandMark-Finger,
HandMark-Multi also is a bimanual technique.
It provides modal access to different command sets.
The main difference between HandMark-Finger and -Multi
are in their commands' placement
and their menu invocation technique.
In HandMark-Multi, we placed a set of 20 commands
in a spatially stable grid between
the space of thumb and index.
There are eight sets of commands,
and four sets belong to each hand.
So in total, it supports 160 commands.
As you can see in the video, different command sets
are mapped to different number of fingers,
and for eight sets, there are eight finger combinations.
So the technique requires specific number of fingers
along with index and thumb from one hand in L-shape posture
to access a particular set.
Same as HandMark-Finger, in this technique,
novice users can complete one task
by following the similar three steps.
However, once the users become expert,
they can combine menu invocation with specific
number of fingers and command selection into a single step.
You can see there.
To evaluate the performance of our new techniques,
we conducted experiments, so in one part of our experiment,
we compared our HandMark-Finger
with standard tab-based menu system.
Tabs-2 had only two sets of command, two taps
to match the HandMark-Finger.
In another part, we compared our HandMark-Multi menu
with similar standard tab-based menu system.
We are calling it Tab-8.
But this time, there are eight tabs in the system.
So for both of the standard tab menus,
we placed command at the top edge of the screen,
and commands are grouped by colors and types
so that they can be easily recognized.
The participants could tap on their tab head
to access one particular command set.
Eight targets were used in the experiment,
and there were appeared in random order
before the participants, and you can see the location here
for both of the techniques.
We have used only eight targets in the experiment,
so that participants could easily familiarize with them.
We have analyzed the data from the experiments
and we found some interesting results.
So first I am going to show you the results
between HandMark-Finger and Tabs-2.
In case of selection time, between HandMark-Finger
and Tabs-2, so if we look at the graph,
so we can easily see that HandMark-Finger performed
better throughout the time, and it was highly significant.
At the beginning, participants had no idea
what the command's location, so they had to search
visually between the two sets to find a particular command,
and that's why they took more time to select command
in the earlier stages.
However, this graph also shows us that once
the participants learned commands' locations
with reference to hands and finger,
they started to perform fast.
We farther analyzed error and
wrong set selection rate in our experiment.
So, in our experiment, error means the incorrect command
selection, so consider the chart at the left.
So we can see that errors were low for both
HandMark-Finger and Tabs-2.
We have also analyzed the wrong set selection,
so it means the number of sets opened to search a command.
If you can see the chart at the right,
we found that wrong set selection was low for both versions
and they are not different significant.
So these results indicate that participants were using
hands and fingers successfully
to remember commands' location.
We collected participants' preference
after the experiment and found that they
preferred HandMark-Finger by a large margin.
And overall, 12 out of 14 participants preferred
our new HandMark-Finger technique
over the standard tab-based menu.
So now I'm going to discuss about the results
between HandMark-Multi and Tabs-8 system.
If we consider the graph of selection time between
HandMark-Multi and Tabs-8, we see that overall
Tabs-8 performed better between the two techniques,
and in the early stages as the commands' locations
were totally unknown to the participants,
both versions were slow.
However, from the graph it is also evident that
at later stages after learning the commands' location,
HandMark-Multi also started to perform at an equal rate
with the Tabs-8.
To analyze this thing farther, we looked at the errors
and wrong set selection.
In case of errors, as you can see in the chart at the left,
HandMark-Multi had slightly higher error rate than Tabs-8,
and if we look at the right side chart,
and we can see that wrong set selection was
significantly higher for HandMark-Multi.
So let me assure you that wrong set selection
was not the count of errors.
Actually, it was the number of sets that were
traversed by the participants in order
to find their target command.
The reason lies in their menu invocation
method of the two techniques.
So if you can see in the image at the top
of the standard tabs, so there are eight sets of commands,
and they are clearly labeled with their names.
The participants could easily use those names
to select the tabs.
However, in case of HandMark-Multi, eight command sets
were mapped with eight different finger combinations,
and they are not visually indicated to participants.
So here, in this image at the bottom,
you can see there are four different finger combinations
for the four sets that are mapped to the left hand.
So there are another four with right hands.
We believe this was the main reason
which made the difference in errors
and selection time between two techniques.
Though errors was likely high, most participants
preferred our new HandMark-Multi techniques
over the standard one, and overall
10 out of 14 participants preferred
the new HandMark-Multi technique.
In the experiment our main goal was to see whether
the participants were using their hands as landmark or not.
During the experiment, we observed that all the users
were moving their hand, moving their menu hand,
selection finger, sorry, let me rephrase it again.
All the users moved their selection finger toward
the correct region on the menu hand,
even before the menu hand was placed.
So these prepared reactions were a clear indication
that participants were using hands as landmarks
to remember the command's location.
So as a summary of the results, I can say that
hands are being used as landmarks
to remember commands' location,
and our new technique accommodated large number of commands.
HandMark-Finger performed faster than standard tabs
in all stages, but HandMark-Multi was slow in early stages.
However, they performed equally
once the locations were learned.
And finally, most participants preferred
our new HandMark Menus.
There are some limitations in our experiments,
so we will try to work on those limitations in future.
So let me give you some example here.
In case of HandMark-Finger, our participants
were required to use all the five fingers of one hand
to invoke the command set, so it was reasonably
easy for them to place all the five fingers
of one hand on the screen.
But in case of HandMark-Multi technique,
specific combinations of fingers were required
to access a particular command set.
And within a short amount of time,
it was really hard for the participants to make
specific posture with different fingers.
So it took us farther research
to make the process more ergonomic.
Currently, our prototype support only one user
from one fixed location and orientation,
however, after invoking a command set,
it was possible to move and rotate command sets
into any direction to work on any occulation.
Farther work is required to allow multiple users
and different orientations of hand.
We also found that HandMark-Multi did not provide
any visual mapping indications
between fingers and command sets.
Participants had to go through different sets
by pressing down different fingers combinations.
We also saw that HandMark-Multi performed slow
because of this reason.
Different color legends attached with different fingers
might be helpful to map fingers with command set.
We would like to see in the future how mapping
of commands effect in selection time
and wrong set selection for HandMark-Multi.
So finally what I want you to take away from
this presentation is that hands can be used
as powerful landmarks in large tabletops
for fast command selection.
Thank you, and I'm happy to take your questions.
(applause)
- [Man] And I will get someone to bring a microphone to you.
- [Man] Hi, Richard Miller, Oracle.
I'm curious how much you thought about the type
of gestures that maybe you could have done to improve
the time on tasks, so you did this kind of two-handed
thing, which to me looked, I kind of cringed,
it seemed pretty awkward, and I'm wondering whether you
talked, thought about like a one-handed gesture,
and also maybe you could comment on sort of your
relationship between this and basic information theory,
and that's a lot to remember, you know the whole seven
plus minus two now is referred to as four plus or minus one,
so it seemed like a lot for me to do,
versus the other task, which obviously you could
just, you know, recognition versus recall.
Thanks.
- In case of HandMark-Finger, actually, we used two hands,
and we used the spaces between the hands, so it was
really hard for the people to remember the commands
at first, we thought that, but from the experiment
we found out that people are easily remembered the positions
and they preferred this technique,
so I think it's possible to remember the commands.
And that's what we are trying to see in this experiment,
and later on, we extended our experiment to see whether
people can remember multiple commands or not,
that's why we tried the HandMark-Multi system.
We multiplexed the commands, and we have used different
finger combinations there, and from that experiment
we also found out that people
are able to remember those commands.
It is also another point to be noted that
in HandMark-Multi there are 160 commands,
and people actually don't use all the commands at a time,
so they use specific sets of commands, so like save,
cut, copy, paste, these kind of commands,
so we can easily used these icon as a shortcut icon
or a hotkey icon, like you can see in the HandMark-Finger,
we placed some icons on top of the fingers,
so those can be used as shortcut keys.
And for HandMark-Multi, so from our experiment
we found out that the commands placed near to the fingers
like index and the thumb, so those are easily accessible
and people accurately remembered those positions,
so if we can place the commonly used commands
on those positions, we believe that it will really,
it's possible for them to remember.
- [Woman] Yeah, is this on?
Hi, I'm Joanna Bergstrom-Lehtovirta
Like this?
Hi, I'm Joanna Bergstrom-Lehtovirta from
University of Copenhagen.
I'm interested on how you mapped the targets,
especially in the finger case,
so how do you design the positions of the targets
so now there is about two between the fingers
and then one in top of the finger for each fingers,
versus to the other condition,
and how do you think if there would be, for instance,
more targets in top of the fingers or less or more
between the fingers, how would that affect
the target selection time?
- That's a really good question,
and we talked it during the course of this work.
For placing the commands, actually we have considered
adult finger size, so from previous experiments,
it is found that normally the human finger size
is around 16 to 19 millimeters and the targets
that can be easily accessible by the users
is around seven point nine millimeters.
So we considered those parameters to place
the commands, actually, and--