Have you ever imagined directly telling
your CAD software what to do, but it's
just too silly talking to the screen?
Have you ever wanted to control music
hands-free, but it's not appropriate to
speak out loud at a quiet place?
Silent speech allows you to speak
silently without disturbing the
environment, like [silently: nice to meet you],
which is suitable for these scenarios.
In addition, silent speech can also be
used in noisy situations where it might
not be efficient to communicate with
voice.
My name is Ruidong. I'm a PhD student
from SciFi Lab, Cornell University. I'm
excited to present our group's work, EchoSpeech:
Continuous Silence Speech
Recognition on Minimally-obtrusive
Eyewear Powered by Acoustic Sensing.
EchoSpeech uses speakers and
microphones on a commercial glass-frame to
recognize silent speech continuously.
Silent speech recognition is drawing
increasing attention lately. Before
EchoSpeech, researchers proposed many
solutions.
Traditional approaches put cameras in
front of the user to capture lip
movements. Researchers from the wearable
community tried putting skin-contacting
sensors to track major articulator
movements such as the tongue.
More recently, researchers put sensors at
less-obtrusive locations to capture more
subtle articulator movements such as the
chin, the jaw, or the ear canal.
Many of these work achieve impressive
performance. However, existing solutions
are still limited in many ways.
For example, video-based solutions are
usually not portable.
Many work uses obtrusive form factor for
better performance, which might not be
comfortable to wear especially for long
hours.
Most work can only recognize phrases
spoken separately instead of
continuously.
Some work uses cameras for better
performance. However, they usually consume
a lot of power and are not privacy-aware.
In addition, many work require users to,
for example, "sit still" or "speak slowly" in
order to ensure good performance.
These issues limit wider applications of
existing silent speech solutions.
To address these challenges, we propose
our solution: EchoSpeech.
So, how does EchoSpeech work?
We notice that, facial movements
especially lip movements are highly
informative in inferring speech content.
We put two pairs of miniature speakers
and microphones on either side of a
glass-frame.
The speakers emit inaudible signals
which travel across the face in
different paths through reflection and
diffraction.
We analyze the received signals from
different paths and calculate the echo
profiles as the representation of facial
movements.
We then stack the multi-channel echo
profiles together and feed them into a
customized deep learning model.
The model has a ResNet-18 backbone
followed by a one-dimensional average
pooling. The pooling is only conducted on
the spatial axis to preserve temporal
information.
Then a fully-connected decoder with CTC
loss is applied to enable continuous
recognition.
Predictions are first made on 192
milliseconds level.
Blank labels are used so that the model
can automatically detect when the user
is speaking without prior segmentation.
Raw predictions are combined to generate
continuous predictions.
In addition, we designed a two-step
training scheme where we first pre-train
a model with other users' data. When a
new user wishes to use the system, we
only need to fine-tune the existing
model with the new user's data. This not only
saves training time but also improves
the performance.
EchoSpeech can work when the user is
sitting still or walking around.
To compensate for the noises caused by
environmental reflection when the user
is in motion, we applied data
augmentation.
We take the echo profiles calculated
when the user is static and the echo
profiles of pure motion noises without
speech. We linearly combine them to
generate the synthesized echo profiles
of speech when in motion.
We then use the synthesized samples to
train the model.
We also applied a similar protocol on
environmental sounds to improve
robustness against acoustic noises.
Specifically the signals used by EchoSpeech
are all inaudible.
Low frequency ranges containing human
voices and most environmental sounds are
filtered out before being used by EchoSpeech.
This preserves piracy.
We implemented the sensing system onto a
commercial glass-frame. In addition, we
implemented a low power variant with
only one speaker with the sensing and
data transmission system operating at 73
milliwatts.
we use BLE to transmit data from the
glass-frame to a smartphone in real-time,
where the entire data processing and
deep learning pipelines are implemented.
Compared with previous work, EchoSpeech
has a lot of advantages. It uses a
minimally-obtrusive form factor while
operating at low power and preserving
privacy.
In addition, EchoSpeech can recognize
continuous speech spoken at a normal
speed without pausing.
We also demonstrate that EchoSpeech is
robust to various noises.
So, how well does EchoSpeech work?
For evaluation, we mainly focus on the
performance, robustness and training
effort.
To evaluate EchoSpeech, we designed two
command sets. The first one includes 31
words or phrases in four categories:
music control, smart interaction, digits
input and voice assistant.
To examine how EchoSpeech handles
continuous speech, we designed the second
section: connected digits, where we
randomly generate three to six digit
sequences for users to speak
continuously at normal speed, such as 123,
67295 - but silently, of course.
We conducted two user studies with 24
participants in different scenarios.
Results show that EchoSpeech achieves
4.5% word error rate in
recognizing discrete speech. As backward
knowledge, the acceptable word error rate
in face-to-face human conversations is
usually considered to be around 5%,
- but of course with a much larger
vocabulary, which is the next step for
EchoSpeech.
For continuous speech recognition, EchoSpeech
still achieves around five
percent word error rate.
What's more, we demonstrate that the
performance is consistent for speech at
different speeds and lengths.
In the user study, participants spoke at
as fast as 113 words per minute, while
EchoSpeech still achieves 6.1 percent
word error rate.
We then evaluated the robustness of EchoSpeech
in different scenarios. We
demonstrate that with proper data
augmentation, EchoSpeech achieves
consistent performance when the user is
sitting, walking or in different noisy
environments.
In order to achieve such performance, how
much training data is needed from each
new user? We examine the performance
curve when different amounts of training
data from the same user are provided. We
demonstrate that the performance
of EchoSpeech improves with more data.
However, with only 6-8 minutes
of training data, EchoSpeech can still
achieve 10 to 15 percent word error rate.
This gives users flexibility in
customizing their own system.
Finally, we demonstrate how EchoSpeech
works with real-time demos.
In this demo, less than two minutes of
training data was collected from this
user to recognize eight commands for
music player control.
The model was pre-trained using other
people's data and fine-tuned with this
user's data.
In this demo, less than 5 minutes of
training data was collected from this
user to recognize digits of any
combination, any length spoken
continuously without pausing.
With such capabilities in discrete and
continuous silent speech recognition,
EchoSpeech can be used in various
real-life applications. For instance, EchoSpeech
can be combined with stylus input
to use CAD software smoothly without a
keyboard.
In addition, EchoSpeech can support
hands-free music player control,
or silent passcode input.
we believe that EchoSpeech takes a
solid step towards a minimally-obtrusive
wearable silent speech interface for
real-life deployment.
This work is supported by the
Information Science Department at
Cornell University. We also thank all
participants and reviewers for their
support. Thank you for watching! For more
details, please refer to our paper.