Good afternoon everyone, My name is Zhang Qian
from Shanghai Jiao Tong University
I am very honored to present our work
SoundLip: Enabling Word and Sentence-level Lip interaction for Smart Devices
Nowadays, the voice user interface or VUI
is becoming more and more indispensable to smart devices
because it can provide an intuitive and convenient interactive way without any touching or typing.
For example, we can use voice commands to control smart appliances
make a phone call and send a message
However, VUI is not ready for increasing demands
Since It still suffers several limitations
For example, it is not suitable to issue voice commands in some scenarios
such as in a library
or environmental noise will significantly degrade the performance of speech recognition
What‚Äôs more, voice-based interaction in public areas also has the risk of privacy leakage
To overcome these limitations, we present SoundLip, a non-invasive end-to-end lip input.
The key aspects of our system are ultrasound sensing and lip input.
As shown in the picture, when people are speaking
SoundLip leverages the speaker to transmit the ultrasound
and simultaneously uses the microphone to receive the signal reflected by moving lips
Then the system can analyze the reflected signals to recognize commands
Moreover, SoundLip can not only recognize isolated word command
but also recognize continuous sentence-level command
Here is the system design of SoundLip, which consists of three modules
Data collection, Signal Preprocessing and Lip Command
In data collection, we use the IQ modulator to get the phase and amplitude values.
Moreover, we simultaneously transmit sound waves at multiple frequencies
to mitigate frequency selective fading as well as enhance the capability to profile multipath environments.
When it comes to preprocessing module, we employ two simple but effective measures in this stage.
First, we calculate the gradient of received signals to remove static multipath inferences
Then, inspired by time warping technique
we apply data augmentation by wrapping the differential data along both the horizontal and vertical axis
Lip command recognition can be split into two different parts, word-level recognition and sentence-level recognition.
In word-level recognition, we design a Hierarchical CNN
which contains two individual CNN and one merged CNN
. The two individual CNN are constructed to learn the intra-modality information of phase and amplitude respectively
while the merged one is to find the cross-modality association between these two modals
In sentence-level recognition, , since it‚Äôs hard to segment for each word
we treat the sentence-level lip reading as a sequence modeling problem
and adopt a multi-task encoder-decoder network
to directly map the acoustic signals into sentences in an end-to-end way.
The encoder and decoder are both based on RNN with GRU units
And we apply a multi-task learning strategy which combines CTC and attention together
The next part is the implementation and evaluation.
We develop a data collection app on a commercial smartphone
and invite 12 volunteers to construct two datasets which are word-level and sentence-level.
The locations where volunteers perform the experiments contain four settings
laboratory, meeting room, bedroom and living room
Firstly, we evaluate the performance of SoundLip on the word-level dataset
In the user-dependent test
We can see that SoundLip achieves a very high accuracy
In the user-independent test, SoundLip can achieve about nighty percent
which means our system is robust enough to recognize the words from new users
Then, we evaluate the word-level micro-benchmark under the user-independent test
From the table, we can clearly see that each component in our pipeline boosts the recognition performance to some extent.
We also evaluate SoundLip in different situations
It can be seen that SoundLip is robust to different noises.
and also robust to different distances and angles.
Secondly, we evaluate the performance of SoundLip on sentence-level dataset
The test results show SoundLip is robust enough to recognize the sentences from new users
In unseen sentence test, we evaluate the sentences not included in the training set.
We can see that SoundLip can also deal with this problem
We also conduct the same comparison and robustness test as those in word-level evaluation
At last, we carry out system delay, impact of phone type and wild performance evaluation
As a result, from these results, we can see that SoundLip works very well in real time
In a conclusion, We present SoundLip, a first-of-its-kind acoustic sensing system
which enables end-to-end lip reading at both word-level and sentence-level
We propose a CNN model, and a multi-task encoder-to-decoder network to enable word-level and sentence-level lip reading, respectively
We construct two high-quality dataset and achieve state-of-the-art performance under a broad range of evaluations