hi, my name is Xinlei and today I'll be presenting Aware:
intuitive device activation using porosity for natural voice interactionS.
And this research was jointly conducted with Zixiong Su and Jun Rekimoto, and it was done
when I was a researcher at The University of Tokyo and now I'm a researcher at Huawei.
So voice interactive devices are becoming more and more popular
and there are voice assistances smart speakers and even edge computing devices.
And to use such a device the first thing we need to do is to call its name.
And such method using keyword to activate the devices is called kws or keyword spotting.
However, although this is the de facto approach it has many problems for misactivation such as it
may respond to keywords that are not intended for device activation and also other researchers find
that it can also respond to totally irrelevant keywords such as unacceptable may also activate
alexa and this kind of mis activation can happen rather frequently so it's a serious problem.
And there are also previous other work that trying to help with this situation
such as proposing gaze-based approach for device activation however because they are
using cameras they can raise other privacy concerns and also for voice-only approach
someone proposed to use customizable keywords but still, it's using KWS And also some others
propose to start the speech with filler and higher pitch but it can be very unnatural
so for us human actually, we can identify when people are calling our names if they're actually
calling us or just mentioning us and also in this kind of daily conversations do you know Leon? no
who is he? He is a new guy in our office i think you should talk to him. Oh there he is hey Leon
Leon i want you to meet Alex. hi hi nice to meet you nice to meet you too.
So in this case although the keyword Leon appeared three times when they are spoken in different
intentions their prosody is kind of different and also there are other researchers showing
similar results for this. So based on those facts we wonder if there are some patterns in
prosody that can be used to differentiate calling voice and the otherwise not calling
voice. And if there are then maybe we can use that to develop much more intuitive and secure
device activation method and such is the motivation for this work.
So we set off to collect some voice data to explore possible features to describe this.
we recruited 11 participants and asked them to talk to the device and sometimes talk to people
by following a script with different keywords and also we ask them to speak
in different directions and distances to the device to investigate the effects of those
factors and after extracting the keywords and do some analysis with them, we find some
interesting results so for pitch variations we find that calling voice are much more varied
than the not calling ones in that the calling ones has a small rise in the first and then drastically
drops in the middle whereas the not cutting ones are much more flattened.
And also calling keywords tend to be much more longer compared to non-calling ones When we look
at the voice quality we find that the calling voices tend to be much more clear than the not
calling ones and they are louder. Also to investigate the effects of these distances
and different directions to the features we're using we run a linear discriminant
analysis and find that our features are robust to different distances and directions. So based
on these results we create a classification model based on CNN and then run a series of
tests across different keyword distances and directions and find it still stable and based on
this classifier we build a prototype smart speaker that only responds when people are speaking
a certain prosody and here is a demonstration for this do you know the device called Alexa?
you can call Alexa and ask questions. Those are not calling voice
Alexa what is the weather? this is calling
right now it's 34 degrees celsius with mostly sunny skies
Alexa These are single words with different prosody. Hi, how may I help you. Alexa.
Different keywords assistant hi how may I help you computer
hi how may i help you computer and without visualization so it's faster.
Assistant and this also works with unseen users with different keywords assistant
so we also conducted a user study to compare Aware's
response accuracy and usability to Amazon Echo. Specifically, for aware we compare
the model's accuracy with completely unseen voice and also with the trained prosody patterns that
are closer to what we find in the data collection experiment. And as for the results we find
that Aware has a better response accuracy compared to amazon echo
especially if you look closer to the data we find that amazon Echo is responding too many times
to not calling voice so its causing the recall rate for not calling voice to be very low and such
is the problem we are raising at the first place and is actually happening. And also for usability
we ask participants subjective opinion over how does the device responded if it is to their
expectation or if it's easy to use or easy to learn for using prosody to activate the device
and if it is intuitive and natural to use and people feel things that it yes it is. Because we
also tested awareness performance with untrained words uh we find actually unseen calling prosody
in our experiment in that people actually are talking in a much shorter duration when speaking
the keyword compared to what we find from our data collection experiment so we used those uh samples
actually to retrain the model and actually in this time the accuracy improved much better
than the initial results so this problem can be solved simply by collecting more data.
Even more interesting because our model is giving a rather good performance across
different keywords we want to investigate if our model also works with completely and same keywords
so we pick up four words: Two in English, Two in another language and to see if it can
pick up calling voice and not calling once correctly in random order and it's actually
working pretty good so here's a short demo of this. Do you know the device called Zhang san?
You can call Zhang san and ask questions. Zhang san, what is the weather?
Okay so although these is the early results we think this
result indicates that prosody-based activation may work independently from specific works and this
direction is totally worth more exploration. For more details please see our paper thank you!