Hello Everyone, I am Zixiong Su from the University of Tokyo.
I am very happy to present our paper, LipLearner
customizable silent speech interactions on mobile devices.
We propose a few-shot
transfer learning framework that enables private communication between users
and smartphones with customizable vocabulary in arbitrary language.
Over the past decade, voice assistants have become incredibly popular,
offering an intuitive way to access devices in different situations.
However, despite their convenience, voice-based interactions still face
social acceptance issues,
and they can struggle to recognize commands in noisy environments.
Other problems, like fatigue
and inaccessibility, also make them less user-friendly.
Silent speech interfaces have emerged as a promising
alternative to tackle the challenges in voice user interfaces.
Researchers proposed various approaches to exploit non-acoustic signals,
such Neuromuscular Signals,
Electropalatography, ultrasound imaging, and RGB cameras.
Among these, we are particularly interested
in lipreading because it can be easily implemented on smartphones
without requiring any obtrusive head-mounted devices.
Additionally, its rich visual
information holds great potential for achieving high accuracy.
However, most previous lipreading systems, which uses train-from-scratch
models, require the user to provide a large amount of data.
This process is tedious and time-consuming, and the model
can only be trained on GPU servers, which is also expensive.
As a result, the user would not be able to use custom
commands, leading to very a limited vocabulary.
Furthermore, such biased datasets that are collected intensively
from laboratory settings make the model vulnerable to environment changes,
which means its performance can degrade significantly as the data domain shifts
over time.
Therefore, we think that
there are three key challenges for real-world lipreading systems.
First of all, we need to minimize user burden of data collection, making it
much easier for new users to start and allow them to add custom commands.
Finally,
the system should be highly robust
to different environments and interaction factors,
because mobile phones have been every-day devices and can be used in various
scenarios, bright outdoors or dark indoors, seated or walking.
Therefore, We propose to combine pre-trained models
with in-situ few-shot transfer learning.
First, we pre-train an encoder model on a
large lipreading dataset, LRW.
Using contrastive learning, it learns efficient
visual speech representations in a semi-supervised manner.
Specifically, by using a similarity matrix
to maximize the similarities between samples of the same words,
and minimize the similarities between samples of different words,
the model learns an embedding space in which similar samples are close
to each other while dissimilar ones are far apart.
Moreover, the diverse data in LRW covers a wide range of lighting
conditions and view angles, therefore, the learned representations not only have
strong generalization ability,
but are also robust to various environment
configurations.
To evaluate the model’s performance, especially in complicated environments
configurations, we collected a real world dataset covering different
lighting conditions, user postures, and holding gestures.
We visualize the latent space using UMAP for dimension reduction.
As the figure shows here, the model effectively
captures speech-related visual information,
and the recording condition only has a subtle influence on the features.
To minimize the user effort in command registration,
we introduce a voice2lip technique that incorporates speech recognition
to automatically learn lipreading from the voice input.
When the user speaks the new command aloud,
the system will remember the lip movements and prompt
the text recognized from the voice signal as the label.
With the efficient representations extracted from the pre-trained encoder,
we leverage a simple logistical regression classifier to learn and update
the decision boundary
in the embedding space every time the user enrolled a new command.
In real-time recognition, the user’s speech is projected
to the same latent space and predicted by the classifier.
Since we freeze the weight of the pre-trained encoder all-the-time,
it is also trivial to fine-tune the model by incrementally
only updating the linear classifier with the new data.
To evaluate the model’s performance, we conducted
a leave-one-condition-out test on the collected dataset.
The result shows that our approach can robustly achieve high F1-score
in unseen environments, outperforming previous supervised method by
a huge margin with much less data.
We also investigated the effect of the number of commands
and the number of shots.
The cross validation was performed in a leave-six-condition-out manner,
which means we only use data from one condition for training,
and test the model on all other conditions.
As a result, the 1-shot model can recognize 25 commands
with an F1-score of 89%,
and it learns efficiently from more shots.
We also observed that more command would be more
challenging to achieve a high accuracy, but this gap
can be filled by adding
a few more training samples.
We also leverage the lip embeddings to enable
visual keyword spotting for hands-free activation.
By enrolling two special commands, keyword, for example
hello Alexa, and non-speaking lips, we can use a slide window to
continuously detect when user started
and ended saying a command by computing the cosine similarity
between the real-time input and the keyword samples.
In this way, the user can activate the system
in a hand-free manner by saying the keyword.
Here we demonstrate the similarity when the user is saying different phrases.
The model can efficiently detect the keyword and reject negative samples.
We demonstrate seamless and reliable silent speech interaction with a prototype
on iPhone.
LipLearner can cope with shaking videos captured in bright sunlight.
It is also robust to camera distortions and different view angles
performing well on dark and low resolution videos in hands-free and noisy scenarios
To achieve a improving performance,
we propose an Incremental Learning scheme to gradually learn form
new available samples.
To initiate the system, the user first record keyword
and non-speaking samples to enable the keyword spotting function.
In the registration mode, The user speak aloud to define new command.
In the active learning mode, the system will prompt to
solicit confirmation for each recognition and use that knowledge to improve the model.
Since we fine-tune the model using a light-weight linear classifier,
it can be updated on-device in no more than 2 seconds.
In the on-demand learning mode, the user can correct misrecognitions at any time.
Therefore, the user can start to use silent speech with minimal data
and trade-off the performance and effort by switching between the learning mode.
In the user study, we recruited 16 participants
of different native languages.
The participants first decided their command set, which consists of 10
pre-defined commands, 10 user-described commands with an illustration
as hint, and 10 user-created commands without any guidance.
Then, they registered the commands in a one-shot manner.
We were also curious about the implications
of using voiced speech for command registration,
so we asked the participants to perform a voiced trial before 5 silent trials.
After all trials,
they were able to use the system freely for 10 minutes
for collecting subjective feedback.
We observed that a bunch of interesting commands appeared in the user study.
The diversity of the commands suggests that
expressiveness of speech interactions is an essential demand, and our user study
has a good coverage of different languages, phrasings and purposes.
Quantitative result shows that LipLeaner has good and improvable performance
in both keyword spotting and command recognition, and can achieve high accuracy
regardless of the phrasing and language.
Finally, we collected subjective feedback using a system
usability score questionnaire.
The result indicate that our system is easy to use and easy to learn.
Since our system
demonstrated strong ability to generalize to different view angles,
we also explored the its feasibility in wearable contexts.
We devised a prototype for a preliminary test.
by collecting a dataset from one of the authors
using the same command set in the model test,
the cross validation results suggest that our method
can also adapt to profile faces captured using a fish eye camera,
showing the potential of various VR applications.
In the user study, we also observed a participant tried to registered
4 kinds of facial expressions as commands,
and 9 of 11 tries were correctly recognized.
We envision that LipLearner can also be used to monitor
emotional health by detecting various facial expressions.
Like any other paper, our work has limitations.
First, the number of concurrently available commands is still limited.
Second, there are several command pairs that so similar
that it is difficult to differentiate even with multiple samples.
Future work should investigate efficient solutions
to disambiguate similar commands and enable large vocabulary lipreading
That's all.
Thank you for listening.