Hi, everyone, I will introduce Endophasia. Utilizing acoustic based imaging for issuing contact-free silent speech commands.
This is a joint work from Shanghai Jiao Tong University, National Taiwan University, and National Chiao Tung University.
Using speech interface is a natural way for interaction on mobile and wearable devices, which has received growing attentions from researchers.
However, this method suffers from several limitations. It may be easily interfered by background noise or voices from others.
It may also raise privacy concerns or disturb others.
Therefore, the goal of this paper is to develop a silent speech interface on mobile devices.
Several works have been proposed for speech processing, but without an intelligible acoustic signal.
Including invasively place sensors within human body.
Or non-invasive solutions, but require attaching, specialized and bulky sensors.
Or camera based solution, which is sensitive to ambient light and power consumption. Therefore, we proposed our acoustic based imaging solution.
There are several challenges to implement acoustic based silent speech interface on mobile devices.
Some mechanisms have been proposed to track the motion of finger or hand, but they cannot extract the 2D movement of facial components.
Limited mics and speakers make traditional acoustic imaging based on mic or speaker array, not applicable to mobile devices.
Also, it is not feasible to ask users to move the device.
Finally, the facial movement is user dependent, and we do not have large open source dataset for training.
The basic idea of Endophasia is that we use a mobile device to emit inaudible sound.
When a user issues silent commands, the facial movements will change the channel impulse response.
We can estimate the CIR at the receiver side.
Then extract the 2D motion profiles to indicate the synthesized movement of facial components.
Finally, we can infer the commands with motion profiles as shown in the right, where we show examples of 2D motion profiles, of 8 commands.
To estimate 2D motion profiles, we first generate the transmission signal with GSM sequence.
And up-convert the signal to inaudible band, which is then played by a speaker.
At the receiver side, we convert the acoustic signal to base band and perform cross correlation to estimate the channel impulse response.
Before extracting 2D motion profiles by computing the differential CIR and segmenting the moving period.
For classification, we proposed a three step training scheme to reduce required data size and adapt to user difference and users' habits change.
In the cold start, we train a model from a large number of users.
In the warm start, we transfer the model to an unseen user, with little effort required.
In the online training, we continuously improve the performance of the customized model.
We developed an Android app powered by Endophasia.
We also deploy Vicon cameras to provide the relative position between user's mouth and the mobile phone.
We collected data from 14 participants. They were required to issue twenty silent commands and repeat thirty times per each.
Generating seventeen hours of audio signals.
To provide micro benchmark, we first evaluate the performance of different CNN architectures in terms of the accuracy and training time.
We choose resnet18 as our base network.
We then tuning the system parameters, such as data augmentation ratio and windowing time of segmentation, to achieve the best performance.
In the cold start, we evaluated the within user accuracy, and leave-one-user-out performance.
Although within user accuracy reaches 90%, it shows only 40% accuracy for unseen users, indicating significant user difference.
A warm start scheme can greatly improve the performance, with few training data from unseen users to mitigate the user difference.
Our online learning scheme can gradually improve the accuracy and adapt to user's habit change.
The overall accuracy of twenty commands is 92.5%.
We also evaluate the impact of angle and distance between user's mouth and the mobile phone.
Although these two parameters have nonegligible impact, the tolerance is enough for practical implementation.
This is a demo video of our system.
That's all. Thanks for your watching.