Hello Everyone. I am Yue Qin from Tsinghua University.
Today, I'm glad to introduce our work -
ProxiMic: Convenient Voice Activation via Close-to-Mic
Speech Detected by a Single Microphone.
As we all know, voice interaction is convenient, efficient, and intuitive.
But voice input still faces some challenges.
First, the voice wake-up process can be lengthy.
Second, such interaction can be tedious and annoying
especially during multiple rounds of voice input.
Third, users risk exposing their privacy as their voices
may be overheard or eavesdropped by a third party.
Therefore, we propose ProxiMic, a close-to-mic speech sensing
technique for handheld and wearable devices. With ProxiMic, users can keep the device close
to the mouth and speaks directly to the device without wake-up phrases or button presses.
In addition, users can also whisper, directly interrupt and realize multiple rounds of dialogue.
In recent years, several works have been done to improve the voice activation process.
They use a variety of sensors and device-related gestures
to shorten the wake-up time for specific devices. The difference is that we only use one microphone.
To implement a close-to-mic speech detector. We first conducted a pilot study to find what
feature can be used. We found the most significant feature is pop noise caused
by air being blown onto the microphone while speaking. This is an example.
Nice to meet you (2 cm normal)
Nice to meet you (10 cm normal)
Nice to meet you (2 cm whisper)
First, in the time domain, the impact of the airflow on the microphone brings high amplitude,
so the amplitude of close-to-mic speech with pop noise is usually higher than background noise.
Second, in the frequency domain, the pop noise area containing low frequencies is visible.
So we adopt a two-stage approach to detect close-to-mic speech in time domain and frequency
domain respectively. The first stage detector is based on the adaptive amplitude threshold with
minimal calculation required. If the amplitude of audio signal exceeds the dynamic threshold,
we will extract a one-second audio clip and hand this raw signal input over to a
tiny spectrogram-based Convolutional Neural Network for refined detection.
We use smartphone to collected three audio data sets. They are voice commands data set,
background noise data set and 7-days daily records.
We used Recall Rate and False Accepts per Week per User as metrics to evaluate our model.
The result shows that the most false accepts occurs when a person speaks from a distance
and superimposes high-energy noise similar to pop noise.
Recall rate is reduced in noisy environments such as the subway.
In Study 2, we conducted four analyses to
understand the performance and boundaries of ProxiMic.
First, in order to understand what CNN has learned,
we calculated the Saliency Map and Occlusion Sensitivity Map
of voice input. We found that our CNN model mainly responds to pop noise in the spectrogram.
Second, we tested different form factors. The results show that watches and phones have similar
performance due to similar structures. For headphones and recording pens, there
was less obvious pop noise due to their outer shells, and the performance has been reduced.
We also used a state-of-the-art ASR system to perform speech-to-text and
calculate the word error rate of the voice command data set.
The result shows pop noise didn't significantly reduce the accuracy of speech recognition.
For the privacy part, we invited 6 users,
ask them to talk to ProxiMic and make sure the ASR recognition is correct.
When one person speaks, others sit on chairs at different distances, eavesdrop carefully
and write down what they hear. We found that whispering can effectively protect privacy.
Finally, we compare ProxiMic to other traditional activation methods.
Result showed that our method was significantly faster than the traditional methods,
and watches is faster than phones and headphones. For subjective feedback,
our method has advantages in terms of speed and multi-round interaction.
In summary, we present ProxiMic, a novel sensing technique that uses pop noise and close-to-mic
features to perform efficient, user-friendly, and practical voice input by a single microphone.
With this, I would like to end my talk. Thank you to your attention! Happy to answer any questions!