- Thank you.
Yeah, my name is Anna Maria Feit,
I'm very proud to be the recipient
of this Outstanding Dissertation Award,
and to stand here today to talk about
something that I'm very passionate about,
and that I've worked for in the last couple of years.
Maybe here a little bit about my background.
I did my Bachelor's and Master's in Computer Science
at Saarland University
and Max Planck University for Informatics.
Then I moved to Helsinki
to do a PhD with Antti Oulasvirta at Aalto University
and now since last year I'm at ETH Zurich
as a Postdoctoral Researcher with Otmar Hilliges.
Probably most of you know this device
and use it probably every day in form or another,
and it's kind of the center of my research.
Text input is still the major activity
how we interact with computing devices.
But we also know that the keyboard is not very ideal
for entering English language, let alone any other language.
Let it be on the computer, and it's even worse
for mobile devices, or smart phones,
or virtual environments.
Text input on the QWERTY layout is
comparatively slow to, for example,
how fast we can speak or read.
It's quite difficult to learn and can be very unergonomic.
So one of the biggest problems I tried to solve
is that the QWERTY keboard is not very ideal
for inputting text.
And where it comes from
actually goes back almost 150 years ago
when really the very first typewriter came on the market,
produced by Remington.
We cannot be sure how the letters actually came together
on the QWERTY keyboard.
The fact is that it somehow emerged from
an alphabetical arrangement of the letters.
We still see that on the middleware
which is very much alphabetically sorted.
There are some theories if it's supposed to
support Morse code, telegraphers,
or that the letters are arranged in a way
for the mechanics of the typewriter to work better.
We are not very sure about this.
A little bit later came the touch typing system,
a system where you put your fingers
in the middle row of the keyboard
and use one finger for each of the columns.
Which is supposed to help you to type fast
and to keep your gaze on the screen and type by touch.
This was invented by a guy called Frank Edward McGurrin,
in 1978, who just tried to be as fast as possible
on the typewriter.
So you see that the way you enter text today
very much was developed for mechanical machine
150 years ago, and still we use this touch typing system.
It's the only way we know how to teach people to type.
And we still use the same QWERTY layout
for all kinds of devices,
which is kind of weird, right?
Some people of course recognized early
that this QWERTY layout may not be ideal for typing English
and tried to change it.
One of the most recognized efforts to do that was
led by August Dvorak, 1936.
He studied people's typing behavior
and the statistical distributions of English
and tried to manually rearrange the letters on the keyboard.
For example, to equalize the load of the hands and fingers
to make it, he recognized that
the speed to type a certain letter
very much depends on the letter that you type before it.
So he tried to minimize the awkward movements
that you had to do with your hands and fingers
to maximize hand alternation,
where different fingers of different hands could
type different letters,
and therefore to develop a better keyboard for people.
And he did many studies to proved the superiority of this.
And it's still available on most operating systems today
but not that many people use it.
The problem that Dvorak tried to solve formally,
the design problem, is to solve what we call today
the letter assignment problem.
Question of what is the best assignment of letters
to the keys of the keyboard that allows you to type fastest.
And this has already been recognized in the 1970s
by operations researchers as an instant
of the general assignment problem.
Which is a common problem to model many real world problems
like the scheduling of the shift of workers
of factory plants in a certain country, things like that.
It's a very complex problem.
For example, just the 26 letters of the alphabet
we already have 10 to the power of 26 different
possible arrangements, possible keyboard layouts,
which all have different properties in terms of
how fast they are, how easy they are to learn,
how ergonomic they are.
So it's really impossible to
try more traditional design methods,
try to construct all these different keyboard layouts,
test them with users.
So approach that we took to assess
to try to deal with this problem
is to use optimization methods.
Algorithms that try to find the minimum or maximum
of a function.
So very basic math.
But the challenge here is that
we need to develop a model of what we want to design for.
So we need to quantify for example the performance
what Dvorak did in his work.
And then if we do that, if we formulate it
as a mathematical function we can use
very fast methods that are out there
that are freely available,
that many researchers have spent a lot of time to develop,
very fast methods to find the minimum and maximum
of these functions.
And these give us then quantitative guarantees
on the goodness and the optimality of the design
that we get.
And allows us to very efficiently and very quickly
search through many, many different alternative designs,
automatically evaluate them with the functions
that we have developed, and pick a good solution.
Let me quickly show you how this letter assignment problem
can be formulated mathematically.
So imagine that you have n different letters
and different keys.
Now we use so-called decision variables.
X to denote if a certain letter
is mapped to a certain key or not.
So x is one if that letter is assigned to the key.
Now imagine that we have two letters assigned
with these two xik and xjl.
Now we want to know what is the cost of
entering the letter j after the letter i.
And this is represented by the c term here.
If we now sum up over all possible combinations
of letter pairs mapped to different key pairs,
then we get our objective function,
the function that we want to minimize.
We want to minimize this cost
of entering one letter after another.
That's basically all that there is.
We just need to add some constraints
that makes sure that other optimization process
assigns all the letters that there are
and doesn't assign letters,
two letters to the same key, for example.
So as I said, if once we have developed
this mathematical formulation,
this is an integer program,
we can build on methods from operations researchers,
just throw that in and get out a very good solution.
Some people have done that and they got
different keyboard layouts
which are seven to eight percent faster
than what we can do with the QWERTY keyboard.
The problem with these methods is that, of course,
or the questions there is, is how do you
quantify this cost term?
What is the cost of entering text?
How researchers have done this so far
was to rely on studies from over 30 years ago,
50, 30 years ago,
where people have studied a lot about
how people type on typewriters.
As shown here on the left it's very different
from how we type on our modern computer keyboards today.
So one of my research goal was to
advance our understanding of modern typing behavior
so that we could build keyboards
that are actually optimized for what we do today
with our keyboards, and not what we have done
30 years ago on typewriters.
I did two big studies to understand modern typing behavior
that are both published, and the data sets are published
and freely available.
One was that we first went to the motion capture lab
where we put little markers, reflective markers,
on the fingers of people typing,
and therefore could record, at every keystroke,
what they did with their fingers,
which key pressed, which keystroke.
And for the first time we could actually see
what kind of strategies people use.
If people still use this touch typing system,
or what strategies they developed for their own
if they were never taught this touch typing system.
Then we did a little bit larger scale study
with 160,000 participants doing an online typing test
where we could a lot, a lot of typing data
and for the first time really see at large scale
how people type based on the keystrokes that they do
in this online typing test.
The main findings that came out of this
is that there is actually a very large variety
of typing styles that adapt over time.
And the number of fingers that people use
in contrast to what many people think
is that it's not necessarily an indicator for typing speed.
So you don't need to have taken this touch typing course
to be similarly fast as people that have done that.
We found other factors, in contrast,
that make you a faster typist,
for example the visual attention,
that you can keep it on the screen,
so-called roll-over typing
where you prepare upcoming key presses
and already press the next key
before the previous one is released,
minimizing your global hand movement
and just moving your fingers individually.
These findings have many practical implications, of course.
How we can teach typing, but most importantly for this work,
they directly inform the optimization of the keyboard.
While Dvorak and others after him optimized
for the touch typing system,
a keyboard optimized for that might actually
not be ideal for modern typists,
that type very differently, as we've seen here.
And this gets even more obvious of course
as we look at other text input methods
that use the QWERTY keyboard
and it's really all around us.
You can look at virtual environments, ticket terminals,
they all use the QWERTY keyboard,
maybe sometimes an alphabetical keyboard.
But they may use very different forms of entering text,
not just using less fingers but for example
using rotary control,
pointing in the air with a remote controller.
So they are optimizing for the 10 finger touch typing system
doesn't really make sense of course.
But what is more, these are different scenarios
for input of text
and it may be that performance is not
the major goal that you might want to optimize for.
But there are other things,
like user experience, ergonomics, ease of use
accessibility, or learnability
that may be important in different scenarios.
And these are all very basic criteria to take into account
when you design any user interface, really,
but they are often ignored
when it comes to text input methods
and performance often seems to be the only goal
if you look at related work in text input.
So what other people have done so far
is mostly concerned with the physical keyboard,
as I said, touch typing.
People throughout the 2000s, people have stared to look at
soft keyboards, which are operated with one or two fingers.
And the main objective has been performance, really.
Then people have started to realize, okay,
maybe if I keep the keyboard similar to the QWERTY
then people are more likely to pick it up and use it
so there has been this objective function
of QWERTY similarity.
And then input recognition, how easy it is to, for example,
for an auto-correction system to recognize
what you want to type.
But I thought that we really need to expand
the space of optimizable text input methods,
and this was one of the goals of my thesis.
Because there are many more ways that you can enter text
using the hands, besides just pressing buttons
and selecting a character from this grid of buttons.
The hand's dexterity is really amazing
and there are many ways we can think about
entering text, for example, pressing multiple buttons
or doing this kind of drawing gestures,
or cording, or hand gestures.
Maybe we really use the hand's dexterity.
In all of these cases
the question is what is the best assignment of symbols
to these input actions.
I have shown that you can use this very same
integer program that I have shown you before
but the question is what is this cost term
that you want to optimize for.
In my thesis I present a big set
of these objective functions, of these cost terms
and partly taken from prior work
of what people have optimized for
but adding on top on a lot of new models
that quantify not only the performance
but also the learnability, the ergonomics,
and this input recognition of all kinds of
different ways to enter text.
So there in the thesis you can pick it up
and take the formula and then run your optimization on it.
I want to quickly show you three cases
where I have applied this.
The first is the question of how can we
transfer the skill from one domain to another.
And here we looked at how great pianists
can operate their piano
and wondered if we could translate this to text input.
So what we did is that we actually
optimized a mapping assignment of letters to piano keys
in a way that people that know how to play piano
can use the same motor movements to enter text with that.
And we trained a hobby pianist there
in a long term study of 140 hours
to memorize the assignment and enter text with the piano
and she got over 80 words per minute,
which I think is no other text input method out there
in research that gets that high
and that compares to professional typists
on the QWERTY keyboard.
And she was still improving.
And that was 140 hours.
To get to that in QWERTY, you need three,
four hundred hours of training so that's incredible.
Another thing is that we looked at mid-air gestures
and were wondering,
these kind of gestures that you see here,
what is good way to enter text with them?
Again the question, what letter should we enter
with what gesture, and there's really
no intuitive, no natural mapping
from hand gestures to letters, right?
So we developed models of performance and ergonomics
that quantified for each of these finger combinations
that you can do.
How fast and comfortable it is to do that
and use that then to find the optimal assignment
of the letters to the keyboard.
But very much resembles finger spelling
that we have in sign language.
It was actually predicted that the mapping that we found was
faster than the finger spelling, for example,
that you have in sign language.
Lastly I want to point out my crown project,
which I'm very proud of.
We just last month, we released a standard
of the new French keyboard,
where we got to apply our optimization methods
together with a committee of experts
to rearrange the keys on the French keyboard.
The problem that they had was
that some characters that you need to type French
are not actually available on the French keyboard.
These are special characters,
with accents, or ligatures, and these kinds of kinds,
which is very harmful to the French language,
which kind of deteriorates
because people do not properly spell it
because they just can't.
So the French government wanted to have a new keyboard
that has many more characters on them
and that allows people to easily type correct French.
And we posted, as this optimization problem,
where you have to assign many different,
over a hundred different characters to the
keyslots of the keyboard,
whereas the regular letters stay where they are
so that people wouldn't be too confused
and would want to take this keyboard.
And the keyboard layout that came out
was really a collaboration between
the experts of the committee and our algorithm.
So we went there and formulated
the proper optimization process based on their input.
Then we could optimize different keyboard layouts
and present to them.
They could use our tool to explore
what happens if I change the character set,
what happens if I remove these characters,
what happens if I vary the different objectives differently?
And this is something that I realized
at the end of this very long project
is that optimization methods are very powerful tools
for designing user interfaces
but even more so if they allowed
collaboration with people and input.
So have a look at my thesis if you want to know more
about optimization of text input methods.
Pick up these objective functions
that I present to you ready to use,
how people type on modern keyboards
and the projects I have show you.
And I really thank my great supervisor, Antti,
and I had Shumin Zhai as my opponent in my defense,
which was a great pleasure to discuss my work with him,
and all the collaborators and people I've worked with.
So thanks a lot.
- Thank you.
(applause)