Hi, everyone. I'm Jackie, from Sanford University.
Today, on behalf of my coauthor,
Grurab Banerjee, Vishesh Gupta
and my advisors Monica Lam and James Landay .
I'm here to present Soundr head position
and orientation tracking using a microphone array.
So this all started with the idea of all smart speakers.
So I wonder whether can future smart speakers
understand me more than just my voice
can it understand me more like a human speaker.
So let's start by talking about one of the most common applications of
smart speakers, which is controlling smart appliances.
For example, this is the living room of my parents house.
I spend a lot of time during last holiday
season upgrading each one of those light to be a smart light.
And then I also set up a Homepod as a smart speaker.
However, my parents had one complaint.
It's not really easy to control those smart lights.
The question is, how do you refer to each one of them?
Currently, you have to give each one
of them a name and they use that name precisely in your voice command.
But remembering the name is really hard,
with a human assistant, then it will be a much easier problem
because you can just point towards those light and then say, turn on that light.
So how about we also do the same thing with smart speakers without adding any
extra hardware? So that's the idea behind our project.
Soundr provides a framework for
head tracking with hardwares
that already existed on smart speakers.
Here's a demo
[User] Turn on this light
[Soundr] The light is on.
As you can see, I'm standing over there,
but I can control those lights just by talking towards them.
Let's see a few more examples.
[User] Turn on this light.
[Soundr] The light is on
[User] Turn on this light.
[Soundr] The light is on
[User] Turn on this light.
[Soundr] The light is on
[User] Turn off this light.
[Soundr] The light is off
[User] Turn off this light.
[Soundr] The light is off
OK.
So let's see how it works.
So this is how smart speakers these day work.
They capture the audio with them onboard microphone.
They usually actually have a microphone array.
So each individual microphones
on a smart speaker will capture the audio with slight differences.
And then what smart spekaers today are
doing is that there aggregating all of those audio streams and then align them
to produce a amplified audio and then use that amplified audio for speech recognition.
during this process all the alignment.
and details in individual soundtracks are lost.
Instead with soundr, we pass the raw audio into an neural network and use the nerual network
to produce, head position and orientation.
We're not the first work that is trying
to predict head position and orientation our from the audio, for example,
for this work, they're enumerating through a set of different position of the users' head
and then they compute an energy signal from that.
And then this signal supposedly will have
local maximum at the position of the sound source.
So, for example, this is one of the visualizations.
As you can see, there's five spikes off
this energy, which probably means that there's five talkers.
However, for this algorithm, it assumes that the audio received
by the microphone is coming directly from the user.
So this doesn't account for reverberation in the room.
And random noise in the room may also affect the performance of this model.
On the other hand,
there's also work trying to predict had orientation from the collected sound.
So, for example,
this work could compute the same energy from multiple microphone arrays
distributed across the room and then find the one with a highest energy.
And then they think that's probably
the one microphone array that the user is talking towards
And for this one, it assumes that the microphone that is
the user is facing towards will collects more hight band audio component.
For both of them,
they need multiple microphone arrays distributed across a room to work
as comparing to a single microphone array that is embedded in one smart speaker.
Similarly, those are reason that prone to noise.
So people are trying to solve this problem by using a machine learning method,
which usually has a much better tolerance on noise.
For example, this work is train on
simulation and then fine tune on real data.
However, it actually produced worse result than traditional method
So the problems is why can't prior work do head tracking accurately,
especially for those machine learning based prior work?
So the prior work with machine learning model is not even as accurate as physics
based method, which can achieve 0.5 meters of
positional error and 44.48 degrees of rotational error
which is achieved with multiple microphones,
One reason could be that they're only processing a short amount of input.
Even if one of the papers suggested longer input improves accuracy?
The reason is you don't really want the processing latency to be long
because you don't want your voice assistant to respond to your command
after a few seconds or so.
The other potential reason is that they are only outputting
they are outputting either position or orientation
because both of the task are really similar.
So we think training for both could be beneficial
And most importantly, they really suffer from lack of data.
The best prior work only have 900 seconds of hand labeled data.
So in our work,
we attack each one of those problems and then achieves a better accuracy.
So this is the training set of the best prior work, as you can see.
they only have 668
seconds of data collected with five speakers.
Among them, only one is a female speaker.
And then they only collected data in one environment.
So there's really no way to know whether
the result will be generalizable to other environments.
So how do we collect more data?
Let's first look at prior work.
In prior work, they usually use a microphone array to collect raw audio data.
And then for head position, orientation, it is usually hand-labeled data.
They usually label it in a interface
similar to this, where they can point out the positions of
individual speakers and then their head orientations.
Obviously, this process is time consuming and also in accurate.
On the other hand, for Soundr to tackle this problem,
We invent a new data collection
method for soundr that leverages commercial VR headsets.
We first used an VR application to guide the user to different positions and orientations
across the room so that we can get it evenly-spread dataset.
Similarly, we collects raw audio with a microphone array.
And then we collect head tracking groundtruth with a VR headset.
So this is me collecting some data for Soundr.
As you can see, the program is guiding me
through different positions orientations across the room.
And what I'm seeing in VR is that I can
see this triangle thing that is guiding me to a different positions orientations.
And also the text is propmting me to speak different commands.
So with this Soundr collects 751 minutes
of training data, which is 50 times of prior work.
So this is the dataset we collected
It has 751 minutes of recording, 16 speakers, among them nine are female speakers.
And then we collected in five different environments.
Compared to prior work, we have collected much more data.
So for the machine learning model,
we adapt to the CNN model similar to prior work, but wider and deeper.
And then we process te audio with a LSTM
module, which allows us to possess longer audio without extra latency.
Finally, we append a few linear layers so
that we can output positions and orientations at the same time.
So this is the result.
[User] Turn on this light.
[Soundr] The light is on,
So we mark the position of the user that we predicted here
and we mark the orientation of the user we predicted here.
[User] Turn on this light.
[Soundr] The light is on.
[User] Turn this light.
[Soundr] The light is on.
[User] Turn on this light.
[Soundr] The light is on.
[User] Turn on This light.
[Soundr] The light is on.
[User] Turn off this light.
Yea. So we can achieve much better position and rotation or error in this way.
So this is the model performance of Soundr
with data from different user and different room.
We can achieve positioning error of 0.57 meters,
which is comparable to the baseline of 0.5 meters.
If we add more training data within
the same room, then we can get even better positioning error and orientational error
The positioning error is already better than the best baseline,
And then the orientational error is
already comparable with results achieved with multiple microphone arrays.
And finally, if we add some training data
of the same user, the orientation error is even comparable
with the best baseline achieved with multiple microphone arrays.
So how do we use Soundr to control our IoT devices?
So, for example, if the microphone is here, the user is here,
and then there's a few lights in the room
What the user can do is the user can
can face towards the light and then say, turn on this light.
Then the system will compute a logical distance for each individual one of those lights
And then find a light with the lowest logical distance and then turn that one on.
Our system can also solve the problem of ambiguity.
For example, the users talking direction
is between two lights and then Soundr will compute the same
logical distance for both of the lights
And then it will figure out that two
of the lights are having similar logical distances.
Because we know the user's position and orientation.
We can we can then ask, which one do you mean?
And then the user can reply.
"The left one."
And then we can figure out which one is to
the users left because of the high tracking.
Configuration of smart lights with Soundr is also very easy.
User can just hop towards light four times like "this is the light."
And then the system will able
to recognize where the light is
This is a demo.
[User] This is the light
[Soundr] Position registered three more times left.
[User] This is the light
[Soundr] position registered two more times left.
[User] This is the light
[Soundr] position registered one more times left.
[User] This is the light
[Soundr] device configuration successful.
[User] Turn off the light.
[Soundr] The light is off.
[User] Turn on the light.
[Soundr] The light is on.
Other than just talking towards a light.
The user can also express their need, for example, in this case, say,
"make this area brighter."
Because now we know where the user is,
we can just we can satisfy their needs
without them specifying which device they're talking towards.
For example, in this case, we are turning on both of.
We're turning on those two lights, which are close to the user.
For configuration, the user first make the device
enter configuration mode and then say I'm in the range.
Now, then the system can add this into the effective range of this light.
And when the users say,
making this area brighter, this system will know that the user want to turn this light.
For the evaluation, we'll have 12 participants.
None of them has participated in the data collection part.
We asked them to do the same task in three different conditions.
The first one is using an AR app to control the lights.
The second one is basic voice control lights, like voice assistant these days.
And the third one is soundr voice control where the user talk towards light.
That tasks include configuration and control.
So this is the room where I conducted the study.
We have five groups of smart lights.
For application of Soundr,
we have already talked about controlling IoT devices.
What's more, it can also be used for querying IoT
devices, for example, in a laundry room with a lot of washers.
The user can say, "call support for this washer."
And the system will be able to figure out
which one to washer the user need support with.
Similarly, in a printing room with a bunch of printers,
the user can also say "what is the IP address of the printer?"
And then the system will be able to figure out.
will be able to tell the user to the IP
address of that exact printer, even if there's a lot of printers in that room.
Finally soundr can be used for configuration,
for example, the user can say
"controlled this light using this button"
and this system will be able to figure out
exactly which light and which button the user is referring to.
Soundr can also be used as a meeting assistant.
Because we know the position the positons and orientations of every speaker,
we can use that to enhance speaker recognition.
It can also be used to figure out verbal references.
For example, the user says
"I will send you the files afterwards" and that the system will be able to figure
out which forms a user to file will be sent after.
Soundr can also be used for indoor guidance.
For example, in a room that is
that with soundr deployed, the user can ask, "where's the restroom?"
And then the system will be able to figure
out a position and orientation of the user and then give precise guidance.
For example, "turn right, walk ten steps and the restroom will be on your left."
Similarly, since Soundr can be used
for remembering location of objects
the user say, "remember the keys is here."
similar to how you can figure an IoT device.
And later to use, the user can ask the system to recall where a key is by saying,
"where's the key?" And then the system will be able to give very precise guidance.
So with Soundr, future smart speakers can finally understand your spatial references
and better serve your need for paper and dataset and source code.
You can find out on this Web site.
And then if you have any questions, feel free to send me an email.
Thanks for watching.