- [Lecturer] So, good morning everyone.
Today I will present my work that I did with
Carla Griggio, Xiagojun Bi, and Wendy Mackay.
So, first may I know how many of you use just
your keyboards, or swipe, or swift key
for daily life?
Okay, so I see a lot of hands raised.
So, thank you, and for those of you who don't know,
so a gesture keyboard, is a soft keyboard
that let's you write a word by drawing a line
that connects all the letters in the word.
So now, imagine if you could, execute commands
with gestures as you type.
So here, I am writing a sentence,
"The party was cool", and then...
Sorry, Okay, I will repeat.
So, here I am writing the sentence,
"The party was cool", and I draw a simple gestures
to turn the word "cool" into red.
And what if I forget the gesture to turn
the text bold for example.
So I can just pause and a dynamic guide
will appear and show me the way.
So in today's mobiles phones we usually
invoke commands from a menu.
For example, if I want to change the font to "impact"
I need to display the menu first,
and then search for where the font is located in the menu.
I tap to invoke it and I finally tap on the text field
to redisplay the keyboard again.
So this process is cumbersome and slow for experts.
We introduce "Command Board" which offers the power of
a command-line interface on any mobile devices.
CommandBoard transforms a gesture keyboard into
and efficient and learnable command-entry tool.
Here I choose a gesture that turns the word
"music" into bold.
With gesture keyboards we can draw a gesture
on top of the soft keyboard, to enter text.
And the text output is displayed on the
output space above the keyboard.
CommandBoard creates and additional space above
the keyboard for interpreting a free form gestures.
Together with the keyboard, they create an
integrated command gesture space for
invoking commands.
We can think of this as extending a transparent
interaction layer above the keyboard,
that only appears if the users light up
when they are gesture typing.
So users can still see the output display as usual,
but they can also invoke gesture commands from the keyboard.
CommandBoard supports multiple command entry techniques.
I present two useful techniques.
The ex-type and execute, and inline gesture shortcuts.
So let's say here, I am writing,
"I would like to see the," and then from that point
I want to change the font to "impact".
With the type-and-execute technique, I first type "impact"
and then I slide up, draw the "execute" gesture.
And there, the font has been changed to "impact".
Now let's go step-by-step.
When I gesture type, CommandBoard reminds the first words
that is suggested by the keyboard, to see if any is
a keyword in the command space.
So, basically we check if the keyboard already appears
in the suggestion bar or not.
The suggested words here, sometimes
auto-complete your gesture input.
So you only need to write the first few unique
letters in a long command name.
And then CommandBoard will suggest the full command for you.
When I gesture type "impact" which is a
command name in this case,
the CommandBoard will display a preview,
and store the command name in the command buffer.
At this point if I slide up to the upper space.
And then I draw the execute gestures,
which is an upside-down "V",
and then release my finger, the command is executed.
And now the font has been changed to "impact".
If when I slide up, I draw another gesture
aside from the execute gesture.
For example here I draw a straight line instead.
CommandBoard will cancel both the word output
and the command execution.
CommandBoard let's users switch seamlessly
between typing and invoking commands.
For example, in the case of an unwanted
command preview appears, I can just ignore it
and continue gesture typing as usual.
And when I lift my finger from the keyboard area,
the word output is displayed as text on the output space.
And this is the second technique,
with the inline gesture shortcut technique,
you can type a word, and then slide up
to the display area, and then draw
a gesture shortcut, for example
that's the gesture shortcut for making a text bold.
And then, you apply the command to that word,
when you release your finger from the keyboard.
CommandBoard supports the transition from
novices to experts.
So when we invoke a command from menu
it is based on recognition, so the process
often involves, searching with multiple steps,
so it is slow for expert users.
On the other hand, CommandBoard is based on recall,
that allows users to invoke command more efficiently.
Here the challenge is how we support the
learnability aspect of it.
To help users learn and memorize the gesture shortcut
CommandBoard can be combined with dynamic guides
such as OctoPocus, so users can dwell above
the keyboard to bring up the gesture- the dynamic guide,
which shows all available commands,
and the corresponding gesture shortcuts.
So I can just follow the green stripe
through gestures to apply the command to "today".
CommandBoard can also be combined with other
dynamic guide techniques, for example
with marking menu that let's users to
select a command by drawing a directional line.
With marking menu users can also select
a menu from a hierarchical menu.
We conducted an experiment to investigate
the efficiency of CommandBoard
by comparing CommandBoard to other
recall-based, command invocation technique.
For example, we compare it to markdown symbols.
Markdown symbols is available in many
chat applications, such as, what's up, or slag,
to apply text formatting.
Users type the markdown symbol
before and after the word from the keyboard.
So for example here, I type, asterisks "cool"
and then asterisks again, and then it
turns the word cool into bold.
Okay.
The experiment is a within-participant design,
with one factor, the command invocation technique.
We measure the input time, that's the time
users spend to write a word.
And also the command selection time,
the time users spend to select the command.
We also count how many times the
participants made styling errors.
And additionally we also want to know
if users prefer CommandBoard.
We asked the participants to type three words,
for example, "ten below zero", and apply
text formatting to the second word.
So we want to know if the technique
is actually worth learning?
Is it actually faster for users?
And to do this, we simulate expert behavior
by showing the correct action to make sure the efficiency.
For example, for the inline gestures shortcut condition,
we display a dynamic guide, that shows the
gesture shortcut that the users have to perform.
And then for the markdown symbol condition,
we show the symbols the participant need to write.
This graph represents the input time
for each technique and each word.
The X-axis represents the technique,
while the Y-axis represents the input
time in millisecond.
We found that inline gesture shortcut technique,
is significantly faster than markdown symbols.
With the inline gesture shortcut technique,
users can select a command almost
twice as fast as markdown symbols.
Users made significantly less styling error
with the inline gesture shortcut technique.
The average styling error per trial
is 0.09 for inline gesture shortcut,
and 0.36 for markdown symbols.
All the participants significantly
preferred CommandBoard.
They can fluently select commands
as they type and see all the possible gesture shortcuts.
So what else can we do with CommandBoard?
With CommandBoard, users can also type
the menu names and CommandBoard displays
their item in the command bar,
and to choose the option, users cross the option
as the slide up, and finally perform the execute gesture.
CommandBoard also let's you search
adjust command parameters.
For example changing the font size with a slide bar.
So here, I type the command size and then I go up,
perform the execute gesture and then the
slide bar appears and I adjust the font size.
CommandBoard can also be used for easy mode switching,
for example, from texting to doodle drawing.
And then when you switch mode, the command space
is also switched based on the mode- the active mode.
If you would like to try the system, or you
would like to see more of it, you can
visit our demo booth tonight.
And finally to summarize, CommandBoard
links the keyboard and the display
by creating integrated input space
for invoking commands.
And we found that CommandBoard's inline gesture
shortcut technique is significantly faster
and less error prone than markdown symbols.
CommandBoard techniques exist in harmony with menus
and buttons, and transitions novices into power users.
So this is for my talk, and I would be happy
to take any questions.
Thank you.
(audience applause)
- All right, thank you.
Questions anyone? ... No?
What's wrong with the audience?