The title and aim of this paper is Co-activation detection in ten-finger typing on a virtual keyboard.
One factor that has contributed to the slow adaption and use of virtual and augmented
reality, is the lack of a reliable, high throughput input method. Adapting the concept of the
common keyboard to create a virtual keyboard is one way of tackling this problem, but this
exposes a challenging input interpretation problem with many sources of noise. One particularly
challenging source arises from the physiology of the hand, combined with the lack of a physical
surface on which to type. Without the physical resistance of keys, intentional finger movements
can cause other fingers to move unintentionally, producing unwanted key activations.
These events are known as co-activations and eliminating them improves the efficiency of information
transfer and improves user experience. Determining an effective method for identifying them and
implementing this method into an existing statistical decoder is the aim of this work.
The first step was to review a dataset of 24 people, typing on a mid-air virtual keyboard
with all 10 fingers in virtual reality, for approximately 45 to 60 minutes. This was done to identify the
circumstances in which they occur. We found that the ring finger produced the majority
of coactivations observed, as shown on the left. We also found that the majority of coactivations
are caused by a finger lateral to the intended one. For example, for intended use of the
middle finger, coactivations of the ring and pinky occur most frequently, as shown on the right.
The review was also used to identify an initial set of features distinguishing coactivations
from intended activations, of which two are shown here. On the left, the time interval
between activations and on the right, the averaged correlation of finger velocities. In both
plots, orange indicates coactivations and blue indicates intended activations. These
features clearly show how coactivations can be identified and separated from intended activations.
The features were then refined to retain the ones most suited to being implemented into
an existing online system. These features were evaluated by training a neural network
on a single feature and recording its performance. Low performing features were removed leaving
a final set of four, the velocity of the finger producing the current key activation, the
finger depth of the previous key activation, and the interval and correlation between the
current and previous key activations. Three models were trained using the final
features, a neural network, a support vector machine, and a naïve Bayes. The performance
of each is shown here, with all models achieving a high level of accuracy in detecting coactivations.
The neural network achieved the highest F1 score and so was chosen to be implemented
into an existing statistical decoder.
The existing decoder calculates the probability that an input character was intended and uses
the result to make more accurate autocorrections. The model's coactivation prediction is incorporated
into this probability and if the prediction exceeds a certain threshold the character was removed.
An example is shown here. First, 'n', 'o', 'p',
is entered using the index, middle and ring fingers. The user is trying to type the word 'notes',
but has caused a coactivation with their ring finger and inserted the 'p' into the input field.
The coactivation module subsequently identifies and removes the coactivation before the remaining
characters, 't' , 'e' , 'e' , are entered. Finally, the input is corrected to the desired word
'notes' by the decoder. With the addition of this model a reduction
in character error rate by approximately 10% relative and 0.9% absolute can be achieved.
This investigation has shown that identification and removal of coactivations from a text input
stream can be completed with simple lightweight features. Four features have been identified
that can aid the detection of coactivations. Combined, these features can achieve a detection
accuracy exceeding 97%. The module integrated into the decoder also demonstrates how coactivation
detection can effectively reduce character error rates.