we have been using touch typing for years and now it's time to reconsider the way we type
hi everyone i'm Mingrui Zhang from the University of Washington and today i'm presenting the work
in collaboration with Shumin Zhai from google and my advisor Jacob Wobbrock
Keyboard became the standard input device since 1874 and QWERTY layout is a standard layout for keyboards
however now we have devices in different shapes with different capabilities and
it's nolonger realistic to carry a keyboard everywhere so how can we communicate with those devices
previously the research has already investigated in various ways and the first way is
interface-specific solutions such as smart watch text entry and text entry in vr environments or
gesture style with entry for smart glasses those interactions are usually limited to
a certain device or interface and it's hard to be generalized
the other category is having anexternal device for text entry such as a wearable sensor or a new input mechanical mechanism
suchas coordinate system those solutions can deal with any interfaces but they have two problems
the first one is the interaction is relatively new and it takes a lot of time to learn it and
the second one is the performance is not really high most of them is less than 30 word per minute
so today i'm going to introduce TypeAnywhere it is a ubiquitous text entry interaction that
enables the user to perform quarter style typing on any surface with a pair of finger worn device
that is able to detect the finger tap actions the device we are using is a commercial product called
TAP it uses accelerometer to detect to detect finger taps and the device is originally designed
also as a keyboard however it is a chorded keyboard for single hand use where you need to
type different fingers together to input a certain character and that can be very hard
to learn instead we tap anywhere the user could perform typing as if there is a keyboard here is a
demo and i can type on a table surface or on a wall and i can even tab it on my laps
the ui interface for type anywhere contains four elements in the evaluation study there is a test
area displaying the phrase for the user to type and there is also a commit area and compose area
when the user performs typing the text will first appear in the compose area and the text in the
compose area can be changed anytime because the keyboard is decoding them however the user could
double tap space to commit the text and the text in the commit area will be fixed and no longer be
decoded and we designed several gestures for special commands you can tap different fingers
together for backspace and enter and you have this pinch style tapping to select word candidates
type anywhere use bird language model it's a neural network model to decode the finger tabs
into corresponding text we use the text before the typing as a pre-context and the input and
input it with the tabbed finger sequences the model will output text in the character level
a traditional keyboard code decoders will contain two parts the spatial model and the language model
and the spatial model's goal is to calculate the probability of each key given the tab position x
and y and the goal of the language model is to calculate the probability of the text or
the character for example if the previous text is g-o-o and then the next character might be
more likely to be d rather than a and combining those two models score the keyboard can decode
the tabs into text however the spatial model can be very user specific because everyone
taps in different style and in the typical way to train spatial models is to conduct heavy
user studies and get their taps distribution the more data you have the better the models will be
and that's a huge amount of effort so on the other hand type anywhere only use a finger id sequence
without their moving coordinates in this way there's no need to conduct heavy user
studies to conduct to collect their tapping data because the model will only transcribe
your tapping finger sequence to the text so we use existing large text corporates
including yelp and amazon reviews and translating the text into their corresponding finger in this
and using those finger id sequence to train the BERT decoder so if you use
a standard key to finger mapping then we can assign a finger with a unique id
and for example good can be converted to 5884 corresponding to left index right rings right
rings and left middle so to train a decoder with different key to finger mapping you
can also change the finger id correspondingly and there is no cursor in type anywhere so to correct
the text you already typed we need to design a convenient way to correct errors this is a demo
so basically to correct an error uh you you will perform this interaction called type thencorrect
um after you spotting an error you can just type the correction first and then tapping
three fingers together to trigger the correction function and possible errors will be highlighted
the highlight error will be used to guide you which error is able to correct here we
use the edit distance to locate possible errors and you can perform pinch gestures
to navigate between different errors and then tap the space gesture to commit the correction
we recruited 10 participants to evaluate type anywhere we trained three decoders as they had
three different styles of key-to-finger mappings the longitudinal study was conducted throughout
five consecutive days where each participant typed 30 minutes for practicing TypeAnywhere
and in the first day we also let them type on a physical keyboard
to get their baseline performance and at the end of the study we also asked them to
perform a TypeAnywhere typing on their laps and each day in the practice they type on a tabletop
this is the average performance of their typing speed the blue line represents the table
surface typing with type anywhere the orange line represents their physical keyboard typing and the
green line is their laps typing performance and we can see participants got better each day and at
the end of the study they reached over 80% of the physical keyboard typing speed which is the
fastest typing speed of published text entry studies on a non-physical keyboard to date
here is a uncorrected character level error rate participants achieved pretty high accuracy
with TypeAnywhere participants reported having a bit of mental workload during typing because they
need to recall their key to finger mappings however the physical load was low and the
performance was rated really high indicating that they were satisfied with their performance
we also observed a trend of their behavior change during the study in the first when they started
using the device they typed as there is an invisible keyboard on the table and they try
to move their fingers to each key's location however after they practiced with the device
for several days their finger movement was less and they realized there is no actual keyboard and
they could just tap their fingers and get the text and some participants view this as an advantage
because they did not know they did not have to move fingers precisely which saves time and space
although TypeAnywhere achieved a relatively high performance there is a lot of work to make it
practical some future directions include the input of special characters such as emoji punctuations
adding personalization to language decoders and another direction is to minimize the form factor
of wearable device for example we can have a wrist worn smart watch instead of finger worn rings todetect taps
for more technical and evaluation details please refer to our paper we open sourced
the TypeAnywhere system and welcome everyone to extend the idea to other applications thank you