Hi Everyone, my name is Rebecca and in this talk, i'll be presenting our work on acoustic activity
recognition when bounded by conversational assistant interactions.
Nowadays, voice assistants serve as a natural interface to a wide range of services ranging from home automation tasks to media players.
And so they help people perform several tasks with ease such as setting timers and playing music etc.
Now, while these devices offer a wide range of useful capabilities, they are still passive and unaware of the human context surrounding them.
And so there's still room to integrate them into the flow of everyday life more seamlessly
by enabling them to become more aware of people's behavioral context.
And this in turn will enable much richer applications.
So, for example, imagine a voice assistant helping a person cook a new dish for the first time.
It could observe and make sure all recipe steps are being followed in the right order and it could
and it could keep track of cooking times, set cooking temperatures, etc.
Another example, imagine a voice assistant monitoring routine activities of daily living of an elderly relative.
So instead of resorting to nursing homes or paid home care,
imagine being able to remotely monitor a loved one living alone,
without having to use obtrusive wearables or instrumenting the household with any additional sensors.
So, in this work, we focus on the observation that these acoustic interactions
are often accompanied by background environmental sounds
that can be a rich source of context, such as conversations taking place,
cooking sounds, kids playing, etc.
And so, as a first step, we focus on background sounds that reflect what people are doing
and focus on a set of activities of daily living.
So more specifically, we explore how off-the-shelf conversational assistants
can be enhanced with acoustic-based recognition of activities of daily living.
In order to achieve our goal, the first step was to develop an add-on accessory device
that would ultimately accompany a Google Home without affecting its main functionalities.
So our device was developed using a Raspberry Pi, a ReSpeaker 4-mic array, and a camera module.
Now contrary to previous audio-based ADL-related work,
our system does not passively and continuously record data
but instead is controlled by the user's interaction with the Google Home.
And so it only records when the voice assistant is activated which is indicated by the LED lights on top.
And so the Raspberry Pi is programmed to continuously monitor the LED light activity using the camera module.
In order to explore further this problem, we ran a semi-naturalistic user study
with 14 participants in their own homes.
And so we collected audio data of user interaction with a voice assistant
while users performed 19 activities of daily living across 3 locations, kitchen, living room, bathroom.
You can see the list in the table.
We captured 30-seconds of audio per interaction, and users were asked to perform each activity twice
so we captured 2 interactions per activity.
Next, we set out to develop a framework that recognizes the activities from the acoustic data.
And so we extracted 64-bin log-mel spectrograms of our audio clips.
And the spectrograms served as inputs to a Convolutional Neural Network, inspired from the VGGish architecture,
as you can see here in the figure.
Now due to the limited amount of data that we were able to collect,
we leveraged transfer learning and pretrained our model on AudioSet.
So we finally used this model as a feature extractor by freezing its parameters after pretraining
and only trained the last fully connected layer with our own collected audio data to classify 19 activities.
So this is a sample audio. And recall that the system was set to capture 30 seconds of audio data.
And so it includes some before and after background sounds as well as the interaction with the voice assistant
which typically entails a user query, "Hey Google what time is it?", and the assistant's answer.
And so as is the case with all voice assistants such as Google Home or Alexa,
the actual amount of public audio you could say that can be retrieved from an interaction is restricted to a short segment which we define as the mid-interaction segment
falls between the user query and the voice assistant's answer.
And, on average, the length of the mid-interaction segment is around 2.3 seconds.
So, another important question we tried to explore was
can we recognize activities when we're restricted to this mid-interaction segment?
So, we ran a series of experiments where we investigated location-free and location-specific modeling.
We conducted each experiment using both the whole audio clip (30 sec) and the mid-interaction segment.
And given that our study was conducted in the wild in people's own homes,
so our data contained a lot of variability in the living environment or the placement of the device
we performed a Leave-One-Participant-Out evaluation to effectively measure our model's performance.
And using the whole 30-sec audio clip, the model was able to identify the set of 19 classes
reaching an average F1-score of around 84%
in a location-free setting.
And, taking into account the fact that users can specify the location of the voice assistant during setup,
we can build location-specific models and so this further limits the classes to their respective contexts.
We can see the performance listed in the table.
Now when using mid-interaction segments, the model was able to identify the set of 19 classes reaching around 57% average F1-score in a location-free setting.
And as expected the performance dropped because also the amount of acoustic information captured was reduced.
The audio clip was reduced from 30 seconds to about 2 seconds of audio.
And so when put in perspective, this is approximately an order of magnitude reduction
and this in turn caused around 25% drop in performance.
With voice assistants, users have the ability to specify the location as previously mentioned.
But sometimes,
it's often the case that users fail to specify the location and so another question we looked into was
whether we're able to automatically infer the physical context of the voice assistant
which is beneficial for enabling location-specific classification
without user intervention.
And in order to achieve that,
we used the activity classes predicted by our system
as proxies for automatically inferring the physical location of the voice assistant.
And we simulated such an experiment and observed promising performance
reaching 95% F1-score.
So in conclusion, this research represents a step forward in leveraging off-the-shelf
voice assistants for building acoustic activity recognition systems.
able to recognize real-world activities of daily living.
And again, we leveraged the key idea that these voice-based interactions with these voice assistants
often capture background environmental sounds that can be a rich source of context.
So, in this paper, we investigated the potential capitalization of such data to recognize essential activities of daily living.
And to do that, we also collected audio-based interaction data with voice assistants
from 14 participants in their own homes
while they performed 19 common home-related activities.
And we make the dataset public to the community at the link provided in the PowerPoint.
We ran a comprehensive quantitative evaluation of our system on the data collected in people's own homes
and showed very promising performance.
Thank you! I'm happy to answer any questions.