- Okay, so thank you for introducing me.
So as Sigmund said, I'm Bruno Fruchard
from Telecom ParisTech, and today,
I'm gonna talk about this paper called,
How Memorizing Positions
or Directions Affects Gesture Learning?
So, for this project, I've been working
on with Eric Lecolinet and Olivier Chapuis,
my two supervisors, and PH student at Telecom ParisTech.
So, first, let's ask a simple question,
that is how to activate a command?
And for that, I'm gonna answer it with a simple example
with the Adobe Photoshop software.
So, on this software the user
can select commands using different ways.
He can use, for example, the toolboxes
on the sides of this application
or he can use a more conventional way,
that is linear menus, as you can see here,
and as you probably all know.
Just you can click on the menu and roll it
and select the command inside the menu.
The problem is that, for novice users it's very,
it's an inefficient way,
because they can look for the command
and select it, but once they know the position
of the command, it's still very slow
for an expert user to find the command again.
So, there are other ways to trigger the commands faster.
You can use, for example, keyboard shortcuts.
But it's not using the same modality as pointing,
it's using another modality.
So, this thing is not very adapted for expert users.
So, an approach that is used in the literature
that we evaluated is using spatially consistent interfaces.
So, with this kind of interfaces here,
I'm showing the project called FastTap.
The idea is that the commands are displayed in a grid
and so the command is positioned, yes,
is placed at a position on the interface
and won't change through time.
So, the idea is that the user can select the commands
and will implicitly learn the position of the command
through time by just rehearsing the selection.
And so, at some point the user here
can know the command implicitly,
using his spatial memory skills.
So, this has been shown to be efficient to select commands,
but there are also other approaches
that are useful for memorizing commands.
And another one that is very known
is using directional gestures.
Here, I'm showing the example
of the marking menus technique.
And the idea is that the user can select menus
and commands using directions.
Here, on the left, you can see the Novice Mode
where the user can see the commands
and he can use the direction up
and right to select the menu and then the commands.
And the idea is that, then the user can know the gesture
and perform it faster without the need to see the command.
So, we can just perform the gesture,
the sketch on the right, in the Expert mode
and select the command faster.
So, again, it's using an implicit memory process
and it has been shown to be efficient in the literature.
Now, these two techniques, these two approaches
should be efficient for memorizing,
but it has not been really evaluated on these aspects.
It's more for command selection.
So, our question is that we have two ways
of selecting commands and memorizing them here.
It's using positions or directions.
So the question we have is how memorizing positions
or directions affects gesture learning?
So for that, we built two interaction techniques.
Here I'm gonna show the techniques using pointing,
so using positional gestures.
So the user can just select, first display the command
trigger the novice mode and display the commands
and then select the menu by just raising his finger
and then select a command inside the menu
by just pointing on it.
So, simple pointing on the mobile device.
The other technique is using directional gesture.
Here, the idea is that the user can just perform
a directional gesture to select the menu
and then another one to select the command inside the menu.
So, the techniques are quite simple and it's using
two dimensions, positions or directions, to memorize.
So, we compared them using a controlled user study.
We had for that 16 participants,
we had two conditions, memorizing using positions
or directions and for each condition here,
the participants had to memorize 16 commands,
so overall, 32 commands and we used also the whole set
of commands that was 64.
So, we had-- the participants had to memorize 32 among 64
because we used also distractors.
Okay, and the design was within-subject design
and we used 3 sessions for that to assess and evaluate
also the long term memory.
So, the task here is that we just showed a command
to select and the participant had to select
the right command.
So, we used two types of phases.
First, learning phases, in which the participants
could see the command and so, they had to select
the right one and memorize them,
and also, recall phases.
In these phases, the participants did not see
the command anymore, so they had to recall
what they just learned before.
Here is the first session using different
learning and recall phases.
After one day, we did a second session with less phases
and after approximately two weeks, we did a last one
to assess how the results could retain information.
So, we had here, two retention phases
that are really important because it's recall phases
after a period of inactivity, so the participants
did not select the command and memorize them
after one day or two weeks, and we wanted to see
if they could retain the information
and how much they could retain.
Now, the thing about the experimental design is that
we used some command patterns because for each menu,
the participants had to memorize half of it, okay?
So, we had to place the commands to memorize.
I talked about distractors and command to memorize,
so here are the commands, in bold are the ones to memorize
and the others are the distractors.
So, we schematized using these kind of drawings,
so here, the big circles are for the commands to memorize
and the small ones are the distractors
and we used different patterns.
The point here was to see if using different patterns,
different positions or directions, because here
I'm showing in the case of the positions
but it's the same for directions.
Okay, so if you take on the left, it will be
like the diagonals and the main axis
where the command will be placed,
and so, we use different patterns to evaluate
if it will change user accuracy and user recall rates.
Okay, now, what about the results?
So, here I'm showing the recall rates.
The recall rate is the percentage of commands
that were rightly selected during the recall phases.
So, how much the participants could recall.
So, we can see that for the first session,
the participants recall better and better to reach 77%
so, a very high score.
Again, here, the participants had to recall 32 commands
out of 64.
For the second session, so one day after,
we can see that they still recall more than half
of the commands after one day and they can reach 81%,
so again, a very high recall rate
and finally, approximately after 2 weeks,
they can still recall half of the commands,
so this shows that using this technique,
using directions to memorize is very efficient.
Okay, now, if we can compare it to the other condition,
so using positions to memorize, we can see that
the recall rate here is quite similar,
with a small advantage for the positions.
Yet, we did not have a significant difference here.
We explained that because we have a great variability
between the participants during these experiments
so that might explain why we didn't have
a significant difference here,
and also, that some participants explicitly said,
one third of them explicitly said
that they were using the same strategies
for the two conditions.
So, no significant difference but we still have
some differences and we can show some benefits
of using positions from rising.
First, we can see that they allow
for shorter learning times.
If we look at this plot here, this is the graph,
we can see that, okay, so this is the display time
of the commands during the learning phases,
so how much time the participant saw the commands
during the learning phases and we can see that
using the positions, they go quite faster,
so they need to see the command less to reach
the same recall rates.
So, that's a really interesting point
and a benefit of using positions.
The second one is that they allow
for faster learning curves.
If we look at this one, so something I didn't say is that
during the learning phases, the difficult mode
was the expert mode, meaning that the participant
did not see the command but they could still display,
they could trigger the novice mode and see the commands.
So, the point here was that if they could remember
the command, they could just select it without seeing,
without the need to see the interface.
With that, with this design, we can see that
the participants did not need to trigger the novice mode
as much as using directions than using positions,
meaning that, for the positions they relied less
on the novice mode so they didn't need to see
the command as much as with directions, again.
Another benefit is that they allow for less efforts.
Here, we made the participants fill some questionnaires
and we saw that for the cognitive load
and the physical load, the participants ranked
the positions higher.
And finally, the last benefit of positions is that
they allow the participants to use spatial
or visual landmarks.
So, I told you about the command patterns
and we can see that for one command pattern,
when the commands were placed at specific landmarks,
distinct landmarks, here in the extreme case,
we can see that using positions was more efficient
than using directions.
This is something very interesting
because it has been shown in literature that using
landmarks is helpful in the case of positions,
so we reproduce these results with this result of our study.
Okay, now, how can we explain these differences
between the two conditions?
Why do we have these differences?
So first, between the two conditions,
we use different visual representations, okay?
In the case of the positions, we used a rectangular
representation, in the case of the directions,
we used a circular representation.
So here, maybe using the rectangular representation
might help people, also might help the participants,
to create these landmarks and to rely on the landmarks.
So, maybe using a different representation
might change these results, so that would be something
to investigate a bit more.
A second explanation is that here, we compared two
interaction techniques, okay?
We didn't exactly learn positions and directions.
The participants didn't do that exactly,
but they were using techniques
and this could be explained, these differences
could be explained with an artifact of the techniques
because here, for example, the gesture to perform
the selection could have an impact on some memorization,
so that might be an explanation of these differences, also.
And finally, something that would be very interesting
and that is the last explanation we have, is that
we encode different...
Yes, using positions or directions can be encoded
differently in memory, so for example, positions
and that was the first hypothesis we had,
is that it's relying more on the spatial memory skills
than directions, based on the literature,
so that might be also an explanation here.
And so, the overall results of this experiment
is that in the literature, the gestures
and directional gestures has been used a lot
and it has been said that it's very efficient
for memorization but we should also using positions
and using pointing is also a very efficient technique
and has some benefits in our study for memorizing commands.
Okay, now, something else that is very interesting
is that participants spontaneously use strategies
for memorizing because here in the experiment,
our only instruction was to memorize as best as they could
but still, they spontaneously adopted some strategies
and the main one was using command grouping.
66% of the participants used this strategy
and so, let me give you an example of that.
If we use the directions condition, if two commands
were on opposite directions to memorize,
they could couple the two commands
and that would help them to memorize.
Something interesting here is that this kind of technique,
using chunks of information, chunking information,
is something known to be efficient in psychology,
but here, the participants spontaneously
used this approach.
Another strategy was leveraging semantic memory,
so 60% of the participants used that and again,
let me give you an example.
In the case of the positions here,
if the command to memorize was eagle on top of the menu,
the participants could say, eagle is up because it flies.
So, the idea here is that the participants could
use the concept of the command to map it,
to associate it to a position.
We have also some other examples, so what is interesting
again is that they can use the concept of the command
and that's again, something used in literature
that's leveraging memory, leveraging semantic memory,
it's very efficient.
Okay, so, we performed a kind of fundamental study here,
so how can we apply the results we had to a current phase
and to... how can we use that,
how can we use our experiments?
So, for example, for the strategy, command grouping,
we could induce that more in our interfaces.
So, in our interfaces, we used some kind of command
grouping using hierarchies of menu
but it could maybe be improved a bit more
using more explicit groups of command,
maybe smaller ones, using for example, four commands
as you can see here, and maybe even something using
like an acronym with the name of the commands,
because that's something also that was used
by some participants in the experiment,
but few of them, so I didn't talk about that,
but they were using also that to remember
a group of command, using the acronym so that
they could rehearse it and it was helping them to memorize.
So, that could be something that might be improved
in the interfaces we have now, probably.
Now, if we look at the other strategy used,
leveraging semantic memory, how could we leverage
such memory in our interfaces?
The thing is, in our experiment, we used
names of command that could be used, I mean,
it referred to concrete concepts but in the interfaces
it's not always the case.
We can have commands that are really abstract,
so one idea that is based on literature and psychology
could be to use peg words, so associating
the concrete word to the name of a command,
for example, nose for new or oven for open,
that might give more materials to the participants
to leverage their semantic memory.
Again, so that's just an idea that could be evaluated,
that would be interesting to evaluate these kind of ideas
and try to evaluate some new designs in our interfaces
to leverage more of the memory and to help
participants memorize better.
Okay, so just to sum up this talk,
we performed a study comparing two conditions
for memorizing, using positional gestures
or directional gestures and we showed that using
position or directional gestures enable efficient
command learning.
We showed also that using positions,
although for easier and faster command learning
and finally, that the participants spontaneously
elaborated strategies for memorizing, even though
it was not in the instruction of this experiment.
Thank you for listening, and I can now
answer the questions, if you have some.
(applause)
- Thank you very much, Bruno.
We have time for questions.
- Hi, thank you for your talk.
Juliet Rumburg.
I was wondering, maybe you said it, if you did
I'm sorry to ask again the question.
Did you look at the difference...
How many items there is in the menu
and I guess, of course, it influences memorization,
if there's too many items, it's harder to memorize.
Did you find a number that's the most appropriate,
what's the maximum number of things
with a satisfying memory rate?
Just, you know, if you have to apply it
in an interface, how many items we should put maximum?
- Yes, so here, we used only two levels for the menus
and it was also practical because the experiment
was too long if we used more, but in the literature,
there has been some studies using, for example,
48 items that could be memorized quite easily.
The thing is, it depends on the techniques used
to memorize and also on the expertise of the users.
So here, we can see that for 32 commands among 64,
so we have a whole set of 64 but we used only 32
to memorize, we can see that the participants
can be quite efficient with that already.
The thing is, there's no rationale
for the good number of commands, it's just that
there are some studies using some commands
sometimes it's less than 10, sometimes it's a bit more,
sometimes it can be way more, so for example, 48.
In psychology, they can use way more than that,
but again, it depends on the techniques, so,
for that, we chose that and it could be interesting
to maybe variate this number and to see
what it would impact, yes.
- Okay, thank you.
- Thanks.
- Okay, we have time for one or two more questions.
- Thank you.
Michael Kipp, Augsburg University of Applied Sciences.
I'm quite surprised.
So basically, you're saying positioning
is better than directional in certain aspects.
Didn't the authors of marking menus argue
that you have the motor memory involved?
So, with positional you have tapping, I guess?
And with directional, you have dragging?
So, what's your take on that this doesn't apply
or that this doesn't increase recall, the fact
that you can actually have used gestures
and as you said, you can build mnemonic bridges
using gestures?
Does it work?
- Yes.
So here, the gestures were quite similar at the end
because pointing and direction, so that was the main,
the whole focus of the study,
seeing if there was a difference.
The thing is, with this kind of memory,
so using the process memory, using the muscle,
it takes a lot of time to proceed,
and here with just these experiments,
the participants, we think that they did not
rehearse enough to use this kind of memory
because like for driving, you need a lot of training
and at some points, yes, you can know the commands
and it can be very efficient.
The thing is, it's not so clear what kind of time,
what is the span of time we need to use
this kind of memory and to make it efficient.
So probably, with this kind of technique,
if we take it longer, maybe we'll see more differences
in one way or the other, that could be very interesting
to do, the problem is that it takes a long time
to do these kinds of studies, but that could
be very interesting to know more
because it has been used as an argument,
to use some techniques in HTI, using memory,
but the thing is, we don't really have a rationale
on how it works exactly.
It's not even so clear in psychology,
so we would need more information, more knowledge
and that's to really to use it as a rationale
and saying this is used in this case, for sure,
after some time, for example.
- Okay, thank you again, Bruno.