Good evening everyone, I am Karan Ahuja, a fourth-year PhD student at the
Human-Computer Interaction Institute of Carnegie Mellon University.
Today I am going to talk to you about my work on Direction-of-Voice Estimation
for Intuitive Speech Interaction with Smart Device Ecosystems.
This work was done with my collaborators Andy Kong, Mayank Goel and Chris Harrison
at CMU. Where a person is looking is an
important social cue in human human interaction,
allowing someone to address a particular person in a conversation
or specify a focus of interest. Prior computing systems
that have used gaze location have either used sensors worn on the head
unlikely for consumer adoption such as the tobii eye tracker
or cameras which have significant privacy implications in settings such as
homes. In this research we explored whether
speech could be used as a directional communication channel much like visual
gaze. This can be seen in the demonstration of
our system here.
Today voice user interfaces have entered the mainstream
via smart assistants such as Google Home or Apple Siri
and products such as smart speakers smartphones and smartwatches.
They have proliferated to almost all devices around us.
However, these devices simply receive and process spoken content
irrespective of if they were the intended target. In these cases the
Direction of Voice will provide a natural mechanism for disambiguation.
Note this is different from existing Direction of Arrival algorithms
which calculate from where a voice originated.
In this example the Direction of Arrival between the speaker and the smartphone
would be the same irrespective of the user's orientation
to the device.
In contrast, our approach calculates the direction along which the voice was projected.
This innately enables voice commands with addressability in much the same way
as gaze but without the need for cameras.
This not only allows users to dispense with wake words
but also more readily and naturally interact with increasingly dense
ecosystems of smart devices. What this inherently boils down to is
predicting where the user is facing or the head orientation of the speaker
based on the audio data. That way we can tell whether the user is
facing the device or not for the intent of interaction
or other applications. We are not the first ones to think along these lines.
Let's first take a look at the prior work in this area.
Prior systems have looked at placing multiple microphones around the room.
Nakajima et al. makes use of 96 microphones embedded in the walls of a
room. Works by Segura et al. further reduces
the number of mics needed using six T-shaped microphones clusters distributed
in a room of known geometry. Recently in parallel to our approach
Yang et al. made great strides in this area and presented SoundR at CHI 2020.
They make use of raw audio from a single microphone array and employ a CNN+LSTM
based deep learning architecture to estimate the speaker's position
and head orientation. We provide a complementary approach
using a single microphone array wherein we pursue a featurization
approach that is informed by the physics of sound propagation and the human vocal tract.
Our setup consists of a four-channel mic
array with each channel sampling audio at 48 kHz.
Our approach relies on two fundamental properties.
Let's go back to the room that we took
as an example to see what they are. The first is that built environments
introduce characteristic multipath effects.
When a user speaks towards a device the first,
loudest and least distorted signal to arrive is the original sound
which took a direct path. All other sounds which have echoed and scattered
off of various surfaces in the environment
are delayed, quieter and more distorted. We measure this multipath effect by the
crispness of the first wavefront and narrow down on three key types of
features which capture this. First and most straightforward is to run
autocorrelation on the sound. If we receive a facing sound that came
first and directly to the microphone there should be no echoes on which to
correlate against in the first few milliseconds of audio.
As a measure of reverb we calculate the speech reverberation modulation energy
ratio. Finally we also compute the cross
correlation using the generalized cross correlation with phase transform
algorithm between all the six pairings offered by
our four-channel mic array. You can see the difference in
correlation between facing and not facing
on the image on the slide. The second property on which our approach is built
is that the distribution of human speech frequencies vary by spoken angle.
The figure here shows the frequency dependent polar plot from 160 Hz to 8 kHz.
In particular, higher frequencies are
more directional and also tend to be absorbed by the environment
leaving the more omni-directional lower frequencies to reach the microphone.
This creates a characteristic imbalance between higher and lower speech
frequencies that you can see on this FFT.
We encode this data by capturing the sum power of low and high frequencies,
the ratios between them and also by fitting a polynomial to the FFT
to capture its shape and taking its coefficient as its features.
An end-to-end demonstration of our system can be seen here where the user
faces the mic array placed on the countertop and speaks to it from
different body orientations. The predictions are displayed on the
laptop where the processing is happening.
In the future, the processing could be entirely done on a smart speaker
as our system is lightweight and computationally tractable.
Now I've used the word facing the microphone a lot till now.
What does it actually mean? There is no universally accepted definition of what
constitutes as facing a direction. Therefore we ran a pilot human benchmark
study to understand what directions constitute as facing and
also provide a preliminary insights into how well humans can estimate direction
of voice. For this, we recruited multiple pairs of
participants. For each pair one listening participant sat
blindfolded mimicking a microphone and the others participants spoke
towards them from different angles. For each of these angles the listener
predicts whether the speaker is facing them or not.
Here you can see the accuracy for each of these angles
for which the listener thinks the speaker is facing them.
It is interesting to note that even when the speaker is unambiguously
facing the listener 6% of the participants think that the speaker is
not facing them. This helps us to establish a human
baseline and also shows the complexity of direction of voice estimation.
Based on our findings we modelled three definitions of facing.
Directly facing where 0 degrees is facing and everything else is not facing.
Dorward facing where this notion of facing is between plus-minus 45 degrees.
And mouth line of sight where the notion of facing expands to plus-minus 90 degrees.
In order to analyze the efficacy of our
system across different factors we collected data from 10 participants
in two back-to-back sessions wherein the user spoke two utterances
namely "hey assistant" and "the quick brown fox jumped over the lazy sheep"
across two rooms: a classroom and an office; and two device placements:
one against the wall and one away from it.
We collected this data across three user distances as can be seen on the image on
the right: 1m, 3m and 5 m.
Each distance there were three polar positions
and each position had 8 orientations starting from 0 to 360 degrees
at 45 degree increments. The direction of voice arrow is highlighted in black
at each of these positions. Such a setup allowed us to bin
outputs based on the different definitions of facing that we discussed.
This resulted in a total of 11,520 recordings being collected with over 350
minutes of data. For our machine learning algorithm we
used an Extra Trees Classifier. We trained our classifier across all
8 angles of direction of voice to mitigate class imbalance and bin the
output predictions based on the facing definition we were testing on.
We designed a study procedure in order to systematically isolate and analyze
different factors. These graphs showcase the accuracy of
our system across different factors. As this is a lot to process, for both
brevity and clarity in all subsequent slides
we are only going to talk about forward-facing criterion.
For full results please refer to the paper.
There are two key takeaways from this plot.
The first is that our accuracies are in line with human benchmarks,
showcased here in purple stripes.
The second is that we do not see a huge accuracy shift across different placements, people or utterances
which is great for the generalizability of our model.
We started off by purposefully engineering our machine learning
features to be speech invariant. To explore how
successful we were we reduced the dimensionality of our
feature set to two components using a Linear Discriminant Analysis (LDA).
You can see on these plots there is little difference in the distribution of
different spoken angles a positive result suggesting
generalizability. This means that you can train our model
on a few utterances like we do for Siri or Google and then
it can make a voice profile for you that generalizes to different utterances.
Another perk of our approach is that apart from utterances
it can work across different languages as well.
Here we make use of our model trained on data we collected in English and tested
on utterances in Chinese.
and Spanish.
We found that distance has a profound effect on the accuracy of our system as
can be seen in these LDA plots. Results show the accuracy at 1m away is
97 %, dropping to 93% at 3m and finally
88% at 5m. Looking at information loss
we find that features derived from voice frequency distribution drop the most in
value at longer distances. It is important to note while our system
has promising results there are several key limitations before
it can be ready for consumer adoption. First is the accuracy of the system
itself. Our model can distinguish between speakers facing and not facing a device
at about 90% accuracy. However, it does fall short of the 99% +
accuracy consumers have come to expect, though they are perhaps more forgiving
with voice interfaces. We also acknowledge our current
implementation does not attempt multiple speaker direction of voice estimation
which would preclude its use in highly social settings.
Currently, voice interfaces that rely on wake words are susceptible to false
triggers. This can be due to external media
sources such as your TV saying the wakeboard such as "Hey Alexa"
or incorrect wake word prediction for example when someone says "Hey Sarah"
and your Siri gets activated. These cases can be as high as 10%.
In such cases direction of voice estimation can be combined with
wakeword confidence to increase robustness to false triggers.
Thank you for listening. This concludes my presentation. For any questions
please feel free to mail me at the email id shown on the slide.