we propose silent spelling as an
alternative to silent speech to enable
text entry in situations where speech
may be difficult or socially
inappropriate
we characterize our silent speller
prototype and compare its use in live
text entry studies to participants
smartphone text entry skills
we conclude with plans to create a
device that improves sensing and is
fully contained within the mouth for
more discrete use
voice control provides hands-free access
to computing and has become part of
everyday life through smart speakers and
smartphones
however there are many situations where
interaction through audible speech is
not appropriate for example talking
using voice-based interaction while in a
meeting or a classroom is disruptive and
considered socially rude in addition
mobile users often avoid using speech
interfaces in public places to avoid
disturbing others or calling attention
to themselves instead they type on their
smartphone which can affect situation
awareness while on the move silent
speech recognition could help address
these problems for the general public
however the greatest benefit may be for
people with movement disorders who rely
on speech recognition
our work is motivated by the following
scenario composited from situations
described to one of the authors while
consulting kim is a project manager for
a large software company that maintains
an open office plan where many employees
work in the same room with adjacent
desks unfortunately kim's muscular
dystrophy has weakened her hands so that
she can no longer type her emails and
documents speech recognition could help
but it would distract her colleagues at
neighboring desks and raise privacy and
security concerns given her managerial
role kim is now searching for a text
entry method which will maintain her
efficacy and allow her to sit with her
team silent speech would be an excellent
text entry method for kim
however silent speech recognition is a
very difficult task
in most previous work vocabulary has
been limited to less than 100 words
another difficulty is that most previous
silent speech systems are sensitive to
movement which may make them unusable
for people with uncontrollable tremors
or movements such as occurs with
multiple sclerosis
on-the-go use of these systems is
impractical
many previous silent speech systems also
require wired interfaces that can draw
undesired attention to the user
silentspeller is an alternative to
silent speech
the user spells words but instead of
saying words audibly they spell without
voicing
silent speller enables large
vocabularies is highly tolerable to
movement and future designs have the
potential to be completely contained in
the mouth
our first question was whether silent
spelling might be useful and usable as
an interface
we employed six users to enter text from
the mackenzie phase set using silent
speech emulated silent spelling and
standard smartphone text entry with an
on-screen mini qwerty keyboard
as we expected emulated silent speech
was by far the fastest with an average
of 115 words per minute
emulated silent spelling averaged 38.7
words per minute and mini qwerty
averaged slightly lower at 36.6
the 95 confidence interval for the
difference in means showed silence
spelling between 7.5 words per minute
slower and 11.8 words per minute faster
than mini qwerty
we also compared subjective ratings on
the nasa task load index
the graphs compare the difference
between the methods for each subscale
negative values indicate that the first
technique in the comparison was rated
with a lower workload than the second
again silent speech dominates but silent
spelling has similar ratings as normal
smartphone typing
this result seems especially promising
as our motivating scenario involves
people who otherwise may not be able to
type due to motor disorders
given these results we decided to
continue developing silent speller
we selected a dental retainer with 124
captive touch sensors to track tongue
movement
the device is commercially available and
is custom made based on the user's
dental impression
it fits tightly to the top of the user's
mouth
one benefit of this custom fit is that
we saw a little effect of session
dependency
data for each person in the following
experiments was collected over the
course of months in many different
locations and scenarios and no
calibration was needed
silentspeller did not seem to suffer
from the repeatability issues often seen
due to sensor placement with other
methods
we used data from two early participants
to tune a hidden markov model user
dependent recognizer
to create a training corpus we used the
dictionary from the mackenzie soccerof
phrase set which is used in many text
entry experiments
two participants collected all
1164 words twice for a total of
2328 samples
ten-fold cross-validation testing with
1164 word dictionary revealed that best
results occurred at 100 hertz sampling
and with 16 principal components
after 1500 example words there seems to
be little improvement in recognition
accuracy peaked at 97 per character and
surprisingly good results can be
obtained with just 500 examples
we also tested on unseen words by
holding out both examples of 100 random
words from the
1164 word dictionary
results suggest that the recognizer may
generalize well to larger dictionaries
both participants tested silent speller
while walking and sitting
surprisingly both participants showed
higher accuracy while walking and both
could demonstrate live text entry with
the system many months after their first
training data was collected
the following videos show silent speller
being used in environments where speech
recognition would be inappropriate or
would be inaccurate due to too much
noise
the recognition in these videos is being
performed live and in real time
formal testing of our live test entry
system involves prompting users with a
phrase which they try to enter by silent
spelling one word at a time
the system provides a five best list for
each attempt the user can accept the top
word simply by starting to spell the
next word
otherwise the user can scroll down the
end best list by swiping the tip of
their tongue from the front of the
palette to the back
pushing the tongue against the entire
palette erases a word
seven participants compared the live
silent speller system to smartphone
texting in three 20-minute sessions
participants were prompted with 107
random phrases from the mackenzie phrase
corpus
these participants first provided 1 000
words of training data and we limited
the live text entry dictionary to the
321 words that compose the 107 phrase
subset
the participants used their own
smartphones for normal text entry which
included auto spell but not auto
complete
silent spellers speed averaged 37 words
per minute with 87 percent character
accuracy while participants mini qwerty
rates averaged 48 words per minute with
93 percent accuracy
average text entry speed was very
similar between the live silent speller
system and the emulated system described
in the pilot study
however the 48 words per minute for mini
qwerty is more than 30 percent faster
than the pilot study
this difference may be attributed to the
fact that the live text entry
participants were more experienced mini
qwerty texters
even so
silent speller speed seems promising
especially given that a recent survey of
thirty seven thousand mini qwerty
texters showed an average speed of 36
words per minute
an analysis of silent speller's accuracy
per character showed that letters
pronounced with an ee sound such as b
p
t
v and z were most likely to be confused
this result is unsurprising in that lip
movement helps differentiate these
letters and the smart palette does not
sense lip movement
current hardware improvements seek to
add sensing for the lips and to create a
wireless version of silent speller that
can be placed entirely in the mouth
thank you for your attention