- Thanks for introduction.
Hello, I'm Yiqin Lu from HCI lab in Tsinghua University.
Today I'll introduce our work
"Typing on Split Keyboards with Peripheral Vision."
So, split keyboard, widely used,
handheld touchscreens, especially tablets.
So, as iPad, as Microsoft Surface,
or Samsung's Galaxy Tabs.
It split the conventional keyboard
into two halves and located them on two corners
of the touchscreen.
So the user can grip the device
and use his thumbs to tap on each part of the screen.
He can use his left thumb to tap
on left part of the keyboard,
and the right thumb to tap
on the right part of keyboard.
So in these settings, the split keyboard have some
benefits compared to the conventional keyboard.
First, this allows user to have a steady grip posture
of the devices so he can use it while walking
or lying down.
Second, it occupies less area of the touchscreen,
so it leaves more spaces for other component.
Third, two thumb typing is more efficient
than one thumb or one finger.
And the users receive a lot of training
during his daily smartphone uses.
However, a split keyboard suffers from the weakness.
That is, the user have to switch his attention
back and forth between the text feedback
and two keyboard halves.
This not only cause more eye and head movement,
but also limited the typing speed.
So, our motivation is to solve this problem
by attention switching
and to improve experience of using split keyboard.
This is video clearly showed
how users type on a split keyboard.
We can see that his eyeballs and head have obvious movement,
because he has to look at the left to right,
and the right to left,
which cost much time and effort.
Our idea is to type with peripheral vision,
what we call peripheral typing.
That is the user focuses his visual attention
at the text feedback and keep two split keyboard
in his peripheral vision,
like this picture.
This typing mode tend to require user to switch attention
frequently between these parts.
In this work, our contribution is to explore
and understand this typing mode.
Okay, with no knowledge of typing with peripheral vision,
we raised three questions.
First, can users type with peripheral vision?
Previous works have investigate that
the feasibility of eyes-free typing
and their answer is yes.
The user can actually type without any visual attention.
So we guessed the answer is yes.
So, the second question,
what's the performance of peripheral typing?
If the performance is promising,
then the third question is,
how to make peripheral type practical?
That means how can users peripheral type
on a real text input task.
To answer the first question,
we conduct the first study.
The goal of this study is to collect data for analysis.
We need data to prove that user can actually
type with peripheral vision.
We used the Samsung tablet
and develop an Android application on it, like this picture.
And we recruit 18 QWERTY users
who use their smartphone's QWERTY keyboard everyday.
And the task for them is to transcribe 40 random
selected phrases from magazines, for example.
And because user, we have no algorithm now
so we use a Wizard of Oz experiment,
that we fake the real typing experiments.
We always show the correct letter,
whatever user touch on touchscreen.
It feels so intelligent,
so user types in a very natural way.
We collect typing data in three conditions,
eyes-on, peripheral, and no-keyboard.
Eyes-on, the conventional typing mode,
we know that user allowed to switch his attention
between these parts.
And the name of eyes-on is derived from eyes-free.
Peripheral, the typing with peripheral vision,
and the user is only allowed to look at the text feedback.
And no-keyboard mode is that we hide the keyboard.
So it's the baseline condition of the peripheral condition,
that we want to know the effect of peripheral vision.
So the results collected,
our results is show in the picture.
And this is intuitive.
We can see that in eyes-on condition
that data distribution are very compact,
that you can see each distributions within each key.
And in peripheral vision,
the distribution are much larger
and there are some overlaps between neighbors.
In no-keyboard condition, there are super large overlaps
between neighbors and we cannot even know which the key is.
So, if there's no keyboard,
then due to no reference,
users are very free and fly their thumbs on the keyboard.
So we can see the touch distribution
is far away from the central.
But if we have peripheral vision,
then we can constrain our typing within the keyboard.
Okay, we still need more powerful evidence
for the feasibility so we used collect data
to run simulation.
For more details, please see our paper.
And the results is, we compute the top K accuracy,
that is, we predict K candidates with a higher possibilities
and if the correct word within this top K,
then we compute the, we say, is correct.
And we see in eyes-on mode
that obviously there are nearly 100% accuracy.
In peripheral mode, we can see that it has over 90% accuracy
on Top-1.
That means, over 90% of words we can predict once.
And surprisingly with so naive algorithm,
peripheral typing have about 99% accuracy on Top-5.
That means, if we offer five candidates
that user can almost select his word
inside the list all the time.
So, now we want to know the second question.
What's the performance of peripheral typing?
So, we conduct a second study
that to evaluate the performance.
We recruit 18 QWERTY users
and to transcribe 40 random text phrases.
And at this time there are difference,
that we have algorithm now,
so we provide a selection method for user to select
words from a candidate list.
We consider two conditions, eyes-on and peripheral.
In eyes-on mode we place the candidate lists
right above the two keyboard halves.
This is default setting on many commercial tablets,
and it makes sense because user can see the keyboard.
So the candidate is right here.
In peripheral mode, it's totally different.
We place the candidate list near the text feedback,
and we provide five, top five words.
Because it's enough.
The problem is user cannot reach the candidate list,
so we design an indirect selection method
that the user drag on the touchscreen.
The process is user first touch on the touchscreen,
then move to the word, then release to confirm .
The average speed, you can see in the picture,
that the peripheral mode significantly
outperformed eyes-on mode with 28% increase.
In other words, because user type more fluently
and casually in peripheral mode.
In this video we use a eye tracker
to track a user's gaze in typing.
We can clearly find that in this video
that peripheral typing user's gaze is fixed
at the text feedback.
But in normal eyes-on typing, user's gaze is switching
between different parts.
This proves that the typing with peripheral vision
actually reduce the attention switching.
Also, we show the gaze heat map.
So, all right, peripheral typing is fast, and user friendly.
So, can we throw out the eyes-on one?
Unfortunately not,
because peripheral typing has some defects.
First, user has to learn about the QWERTY layout,
but it's not really problem
because sometimes learning is necessary.
But the most important problem is the peripheral typing
strongly rely on the language model.
Which means user can only input word in the dictionary,
but for out of vocabulary words, such as names or address,
user can not type with peripheral vision.
And I think this is a common problem
for algorithms use language model.
But, in eyes-on mode you can type anything.
So the question is
how can we combine these advantages together?
Can we detect peripheral typing or eyes-on typing?
So our solution is to put them together at the same time.
So we design GlanceType.
GlanceType support both eyes-on and peripheral mode
at the same time.
There are two kinds of lists.
One is just near the text feedback,
which is the candidate list for peripheral typing.
The other is located on the region above the keyboard,
like conventional split keyboard.
So, what the benefit of this design?
This is because GlanceType allows user to
freely choose eyes-on or peripheral typing mode
without mode switching.
Two candidate lists are visible simultaneously.
So the candidate user want to select only depends
on user's intention.
When the user want to type in peripheral mode,
he can just drag and release to select word from the list.
In this case, he doesn't require to switch his attention.
When the user want to input an out of vocabulary word,
such as his name, he just tap on the,
with eyes-on typing, and the literal character screen
is shown above the keyboard.
And the user can easily tap on it.
This means, you can type anything.
And the punctuations, symbols, and digits are also be
support by eyes-on typing mode.
We design a swipe gesture for user to switch
from the alphabetic keyboard to punctuation keyboard.
So, to answer the third question,
how to make peripheral typing practical?
The goal of this, we conduct the third study,
and the goal of this study is to figure out what
people will do when free to type.
The corpus we use is change now.
We use the e-mail, dialogue, and Twitter datasets
as our corpus, and because these are all tablet scenarios.
We evaluated GlanceType,
GlaceType with no peripheral vision,
and commercial Samsung keyboard, at this time.
So the results show that GlanceType has higher speed
in entering overall performance,
and also GlanceType has 7% higher speed
than the commercial keyboard.
This show the potential of use GlanceType
in our real life.
And we also provide other results in our paper.
So, we answer all three question, and new question comes.
First, how much feedback is most efficient?
We explore the peripheral vision.
It's a intermediate mode between no vision and full vision.
So, how much visual feedback is enough,
and which visual feedback is best?
Second, how's the peripheral typing in other conditions?
We only evaluated in the landscape mode of the tablet.
And what about portrait mode
or typing on big table tops or larger displays?
That's our limitations and our future work.
Okay, that's all my talks.
Thanks for listening,
and I welcome questions.
(clapping)
- [Host] Thank you.
If there's any questions in the audience,
just raise your hand.
There's SVs with a microphone.
Okay, I can ask a question first, while you think of some.
Hold on, I had some written down here.
Okay, first one.
I may, okay, so, did any of the,
was there auto-correction during any of the studies?
And if not, how were participants dealing
with any mistakes they made?
- Pardon?
- [Host] Like, auto-correction for typing,
was that enabled during the studies?
- Yeah, enabled, enabled.
- [Host] Okay, so was there a sense that participants
were making mistakes often or?
- It depends.
We told them to type as natural as possible.
So sometimes the mistakes happen.
- [Host] Cool.
Do we, yep, okay.
- [Audience Member] Oh, it works now.
Yeah, so I'm Martin Lock from Strathclyde Uni Law School.
In your studies, the keyboard was always visible.
Yeah?
- Mm-hmm.
- [Audience Member] In the peripheral vision.
- Yeah, yeah.
- [Audience Member] Did you think about,
and you instructed your users to look
at the text input area, yeah?
- Hmm.
- [Audience Member] Is there, did you think about ways of,
so when I'm using a tablet, I'll be looking at the keyboard,
and I will naturally, I think,
start looking at the keyboard.
Did you think of ways that the interface could
encourage me to stop looking down there?
- Yeah, yeah.
I think, this depends on the,
I think this depends on user case,
because in some scenarios we can drag the split keyboard up,
and see that split keyboard
and the text feedback can inline.
So, in a line, so that's easy for us to type.
But in some cases that's using
maybe search engine that you type on the text feedback
and there will be a long list,
and various pictures around,
so the split keyboard can be,
is better to locate it in the corner.
- [Host] Okay, so I might have missed this,
how many of your participants were touch typers?
- Touch type for the physical keyboard,
but they don't have experience touch type on touchscreen.
- [Host] Okay, did you actually ask or assess that?
I just, I know that I've actually started getting
quite good the more I use-- - Yeah, yeah, yeah.
People, sometimes, just type on the keyboard
without looking at the keyboard.
That's part of the peripheral vision
that we investigate the full peripheral vision.
That's different.
- [Host] Okay, cool.
Is anyone else near the back of the room
that want to ask a question?
We can run out and give you a microphone.
No, okay.
Well, I think that we should thank our speaker.
(clapping)
So it's lunch time just now.
There's also, in the town hall,
there's a town hall lunch in room Lomond.
And that concludes this session.
So just thank you to all the speakers again.
(clapping)