I am Karan Ahuja a fourth-year PhD student at the
Human-Computer Interaction Institute of Carnegie Mellon University.
Today I am going to talk to you about my work on Direction-of-Voice Estimation
for intuitive speech interaction with smart device ecosystems.
This work was done with my collaborators Andy Kong, Mayank Goel and Chris Harrison at CMU.
Today, voice user interfaces have entered the mainstream via smart assistants such as
Google Home or Apple Siri in products such as smart speakers, smartphones and smartwatches.
They have proliferated to almost all devices around us.
However, these devices simply receive and process spoken content
irrespective of if they were the intended target.
In these cases, the Direction of Voice would provide a natural mechanism for disambiguation.
This innately enables voice commands with addressability, in much the same way as gaze,
but without the need for cameras. This not only allows users to dispense with wakewords
but also more readily and naturally interact with increasingly-dense ecosystems of smart devices.
What this inherently boils down to, is predicting where the user is facing or
the head orientation of the speaker based on the audio data. That way we can tell whether
the user is facing the device or not for the intents of interaction or other applications.
We are not the first one to think along these lines. Let’s first
take a quick look at the prior work in this area.
Prior systems have looked at placing multiple microphones around the room of known geometry.
Recently, in parallel to our approach, Yang et al. made great strides in this area and
presented SoundR at CHI 2020. They make use of raw audio from a single microphone array and employ
a CNN+LSTM based deep learning architecture to estimate the speaker's position and orientation.
We provide a complementary approach, using a single microphone array
wherein we pursue a featurization approach that is informed by the
physics of sound propagation and the human vocal tract.
Our setup consists of a 4 channel mic array, with each channel sampling audio at 48 kHz.
Our approach relies on two fundamental properties.
The first is that built environments introduce characteristic multipath effects.
When a user speaks towards a device, the first,
loudest and least-distorted signal to arrive is the original sound, which took a direct path.
All other sounds – which have echoed and scattered
off of various surfaces in the environment – are delayed, quieter and more distorted.
The second property on which our approach is built is that the distribution of human
speech frequencies varies by spoken angle. The figure
here shows the frequency-dependent polar plots from 160 Hz to 8 kHz.
In particular, higher frequencies are more directional, and also tend to be absorbed by
the environment, leaving the more omnidirectional, lower frequencies to reach the microphone.
This creates a characteristic imbalance between higher and lower
speech frequencies that you can see on the FFT.
An end-to-end demonstration of our system can be seen here wherein the user faces
the mic array placed on the countertop and speaks to it from different body orientations
the predictions are displayed on the laptop where the processing is happening.
We conducted a human benchmark study, based on which we came up with three definitions
of “facing”. For both brevity and clarity in all subsequent slides,
we are only going to look at one of them which is the forward facing criterion wherein the
notion of facing spans from -45 to 45 degrees and everything else is considered not facing.
In order to analyze the efficacy of our system across different factors,
we collected data from 10 participants in two back-to-back sessions wherein the user
spoke two utterances across two rooms and two device placements. We then collected the data
across three user distances as can be seen on the image on the right: 1m, 3m and 5m,
and each distance there were three polar positions and each of this position had eight orientations
starting from 0 to 360 degrees at 45-degree increments. This resulted in a total of
11,520 recordings being collected with over 350 minutes of data.
We designed our study procedure in order to
systematically isolate and analyze different factors.
This graph showcases the accuracy of our system across different factors.
There are two key takeaways from this plot.
The first is that our accuracies are in line with human benchmarks showcased here in purple stripes.
The second is that we do not see a huge accuracy shift across different placements,
people or utterances which is great for the generalizability of our model.
Thanks for viewing this talk. For any questions please email me. For further
details please refer to the paper pdf and the full 15-minute talk.