Hello everyone! Thanks alot for taking interest in our research. I am Vidit Jain and I will
be presenting our work titled Exploring Semi-Supervised learning for predicting listener backchannels
I would like to briefly describe what listener backchannels are and then move onto why they
are important.
Basically, A listener backchannel is a frequent and short response provided to the speaker
in a conversation that signals that the listener is paying attention to what the speaker is
saying. Such responses are quite important for sustaining a conversation. Some common
examples of BCs are head nods, head shakes, short utterances like hmm, right or a combination
of both.
As we all know, personal robots and interactive AI systems are increasingly becoming a part
of our lives. Social robots that we have currently, have shown immense potential in sectors like
education or healthcare where they can serve as personalised learning and social companions.
One important requirement for such robots is that they must be good active listeners.
That is why we focus upon modelling listener backchannels.
While going through the previous literature on backchannel prediction, we found a few
gaps: Prior works rely upon a large amount of annotated
data to train backchanneling models. We felt that this was a huge bottleneck as annotating
large datasets for backchanneling would require a lot of time and effort.
Secondly, most works focus on the task of backchannel opportunity prediction i.e. when
should the backchannel occur. But to design active listeners that mimic human behaviour
we require that the BCs be expressed as closely resembling to human behaviour as possible.
Thirdly, we found that all previous works focussed on English conversations.
Through this research, we explore the efficacy of using semi-supervised learning techniques
to automate annotations by comparing them against conventional supervised learning approaches.
Semi-supervised learning techniques are advantageous as they require only a small fraction of the
data to train and they are able to generalise really well, thus saving time and human effort.
Secondly, we propose techniques to predict the modality for backchanneling. We call this
task as backchannel signal prediction. And lastly, we use Hindi conversations.
For this paper, we use the Vyaktitva dataset, which is a multimodal Hindi conversation dataset.
The dataset contains 25 Hindi dyadic conversations.
3 annotators manually annotated the dataset for both the presence and the signals used
for backchanneling. We chose to annotate nods, head-shake, mouth i.e. whether the person
is smiling or not, eyebrow movements and short utterances for signals.
We extract several visual and acoustic features from the conversations to train our models.
For task formulation, we divide our approach into 2 categories: identification and prediction
The identification task encompasses identifying backchanneling instances and the corresponding
signals. This task included the manual annotation of the dataset to get the ground truth labels
and training semi-supervised models to learn to identify and annotate BCs in the data.
We refer to these generated labels as pseudo-labels.
The prediction task involved using DL models to predict when should a listener backchannel given the speaker’s
video and audio. Also if a BC was predicted, then we predict what modality to be used to
express that backchannel. The prediction task was performed on both the manually annotated
and semi-supervision annotated labels.
Some of the key findings from our research were
The semi-supervised models were able to identify BC instances and the signal types very well.
We achieved high accuracies given that we only used 25% of the ground truth data was
used as seed to train the semi-supervised learning models.
The key highlight of our study was that the opportunity and signal prediction models trained
on the pseudo-labels achieved close to 94% of the performance achieved by the models
trained on manually annotated data.
To further strengthen our results we conducted a blind
user study where participants were asked to rate a conversational agent trained on manually
annotated data and another agent trained on semi-supervision annotated data. 20/27 participants of our study found our
proposed system to be more or equally natural to the one trained on manually annotated data
Now I would like to present a short demo of our system
Human: in the US... umm...
Human: US has a diverse terrain
Human: it is a different continent but... | Agent: ok [nods][smiling]
Human: but for our case
Human: in India we can find deserts, grasslands and... | Agent: [head-shake][smiling]
Human: but in the US also there are many different kinds of areas | Agent: Hmm
Human: like glaciers can be found in Alaska and...
Human: Nevada has a desert-like terrain and... | Agent: Yes
Human: then many places like Washington have wetland-forests | Agent: Ok [nods][smiling]
Human: my uncle used to live there so...
Human: they went to Alaska | Agent: [nods][smiling]
Human: so... he showed me some beautiful pictures of glaciers there... | Agent: [nods][smiling]
Human: the blue colour of the glaciers is very different
Thank You for your time. I’ll be happy to take up any questions now