I'm Mark Richardson, a research engineer at Facebook Reality Labs, and I'll be discussing
work from myself, Matt Durasoff, and Robert Wang on decoding surface touch typing from
hand tracking.
In virtual and augmented reality text input is important for communication and productivity.
Here we investigate the use of hand tracking to enable touch typing on a flat surface.
There already exists a wide audience of effective typists and this could allow them to transfer
their skills into AR and VR.
We've seen from recent consumer devices that hand tracking can be achieved through on-headset
sensors without extra peripherals, potentially making this suitable for on-the-go use cases.
Our investigation caters particularly to the fastest typists, touch typists.
Without the feel of physical keys and without using sight to find keys, fingers will drift
during typing.
Instead of requiring users to make precise contacts on a fixed keyboard layout, we investigate
using a motion model to recognize finger trajectories.
Here we side-step the current limitations of consumer head-mounted hand trackers by
using a high quality marker based hand tracking system.
While this introduces a gap in tracking quality compared to what's achievable on AR and VR
headsets, we can study the "potential" of hand tracking applied to this problem.
This problem is analogous to speech recognition, so our system design follows the design of
modern end-to-end speech systems.
For every frame of hand data, our motion model predicts a distribution over the set of keys
pressed at that frame.
We use a neural network to represent our language model prior, which provides a probability
distribution over character sequences.
A beamsearch decoder combines the character heatmap sequence with our language model prior
to predict the text that was typed.
To train our motion model we need a ground truth dataset.
We recruited 20 self-identified touch typists who we prompted with specific short phrases
and recorded their hand motion as they typed on a paper keyboard.
Participants also typed a set of phrases on a physical keyboard to establish a performance
baseline.
With this data we compared throughput across modalities and found no significant difference
in typing speed between physical keyboards and flat surfaces, so it seems that touch
typists are able to transfer their skills to typing on surfaces.
Compared to typing on a physical keyboard our surface touch typing system was nearly
as accurate with a mean uncorrected error rate of 2.7%.
While it looks like people might still type on keyboards more accurately than on surfaces,
the difference isn't statistically significant.
Typing on a flat surface isn't a new idea.
Soft keyboards are pervasive on our phones and tablets.
One question we had was if hand motion was fundamentally more informative than contact.
To better understand the performance of our approach we decided to compare against methods
that decode ground truth contact events from a touchpad.
To do this we placed a pressure sensitive touchpad underneath the paper keyboard so
we could evaluate our motion model against contact decoding baselines.
For decoding 2D contacts to text a method proposed by Goodman et al in 2002.
Given a set of 2D contacts per key we can fit a bivariate Gaussian per key.
Then a novel 2D contact can be evaluated according each Gaussian to get a distribution over the
set of keys.
This simple approach can achieve a mean UER of 5.7% but we found that our approach outperformed
it at 2.7%.
We posited three key factors that account for the delta.
The first factor is that our motion model can leverage finger identity information while
the contact method cannot.
Here for three different participants we plotted their fitted Gaussian models and we colored
each Gaussian by which fingertip typically hits that key.
In general we found most typists have reliable correspondences between specific fingers and
specific keys.
To somewhat quantify the value that finger identity adds we tried a modification of the
Gaussian approach where for every contact we used hand tracking to find the finger responsible
and then we fit one Gaussian per key per finger for each user.
With finger identity the mean UER of the contact approach improves from 5.7 to 4.8%, better
but still short of our motion model.
The second factor is that our motion model makes one prediction every 60th of a second
while contact methods make predictions only when a contact occurs.
The contact method predicts one key for every contact, so missing or spurious contacts automatically
mean incorrect decodings.
On the other hand, out of necessity our method is able to predict the possibility that no
key was pressed allowing for event insertion or removal according to the language model
prior.
Here for example it can predict an H that has no contact event to spell the word "WITH."
The contact method can't recover from this.
To quantify this effect we evaluated both approaches after filtering out all samples
with these artifacts.
This is an easier dataset so both contact and hand motion decoding perform better.
The gap between contact and hand decoding narrows from 2.1% to 1.6% though, suggesting
that continuous motion decoding is responsible for some of the accuracy gap.
But still, there's this 1.6% gap that's left, where does this come from?
We think the third factor is that our approach is able to leverage 3D trajectories instead
of only 2D contact points.
Here you can see two clusters of trajectories of all the times one user hit the "I" key
and the "O" key.
In green you can see a novel trajectory which lands on the "I" key among all the blue trajectories,
but the shape of the arc is undeniably similar to some of the red trajectories.
While the contact methods can only see the 2D contact, our motion model can see the shape
as a whole and can correctly classify this case.
Finally we enabled this end-to-end system including motion model and beamsearch decoder
with language model prior to run in realtime at 120Hz on a desktop GPU.
We created a toy text entry application with an Oculus Rift and the Unity game engine to
demonstrate real-time touch typing on a flat surface in VR.