Hello.
This is a presentation of improving social awareness
through DANTE.
A deep affinity network for clustering
conversational interactants.
I am Mason Swafford.
And I'm John Peruzzi.
We will be presenting this paper on behalf of our colleagues.
Conversational group detection is
important for enabling interactive displays.
Socially aware robot navigation.
And security applications.
Displayed here is the system we created,
which utilizes an Azure connect in Robot to simulate a socially
aware interactive display.
From the visualization in the middle and bottom right,
we can see that the robot is cognizant.
That it is part of the group formed by the other members
of the scene.
Prior work on conversation group detection
utilize models from social psychology as in three.
Created graphs to represent a social scene
as in two and five.
Or leverage data to learn mappings
from participant features to groups as in one
and prior work from our lab in four.
Our approach leverages techniques
from graph based and data driven methods.
To perform group detection, we leverage an existing graph
clustering pipeline which works as follows.
Each social scene is captured.
And the 2D position and orientation of each participant
is extracted typically using an off the shelf
computer vision algorithm.
This scene is modeled as a complete edge-wheighted graph.
Where nodes correspond to participants.
And each edge between two nodes corresponds
to the affinity between those nodes which
is the probability they are in a group.
The weighted graph is then converted
into a symmetric affinity matrix where
element ij is the affinity between person i and person j.
A graph clustering algorithm is then
used to cluster the groups according to their affinities
into dominant sets.
A general-- generalization of maximal
clicks to edge-wheighted graphs.
The key step in this pipeline is calculating affinities
between participants.
Prior work calculates affinities as a function
of the position and orientation of the nodes connected
by the edge.
This is either via heuristics or more recently,
through data driven techniques.
However, it is impossible to accurately determine affinities
between pairs of participants without also
considering the social context of the rest of the scene.
Therefore, our affinity function takes an additional context
and puts C which represents the rest of the participants
in the scene.
We implement this function using a neural network.
Here, you can see how our affinity calculation
network, DANTE fits into the overall group detection
pipeline.
Shown as an example of DANTE, calculating the affinity
between persons 4 and 5.
And this would be repeated for all dyads in the graph
in order to construct the affinity matrix.
Zooming in, DANTE is composed of a dyad transform
and the context transform in a multi-layer perception.
Now, we'll go through how these pieces fit together
in the context of predicting affinity ij.
We start with the dyad transform.
Which takes features from our dyad of interest as input.
The dyad transform uses a multi-layer perception
to find a high dimensional encoding of the inputs.
We also input the feature vectors
for the rest of the participants into the context transform
which outputs a high dimensional representation of the context
features.
Well, a standard multilayered perception
could not handle variable number of context participants
and abuse meaning on their ordering.
The key idea behind the context transform
is the use of the symmetric max function to aggregate context
participants high dimensional features
into an overall context feature vector.
A final multilayered perception combines
the outputs of the dyad and context
transforms to output a predicted affinity.
Here are the results of our group detection
pipeline on standard data sets labeled ground groups.
Our method is shown in red.
And as you can see, we outperform prior work
on the cocktail party and SALSA data sets.
However, on the coffee break data
set we perform some similar to prior work.
Because the participant input features
are noisy which limits the performance of our machine
learning based approach.
Qualitatively, we can see the importance
of the context features via an ablation study
where we removed the context transform.
Here, the predicted group membership is shown by color.
And the predicted affinities are shown by line weight.
For the scene shown, everyone should be in one large group.
However, DANTE-NoContext produces
lower affinities between person four
and the rest of the group because of his distance
from them.
However, DANTE is able to correct
for this by using the social context to understand
that persons will be farther away from each other
when in large groups.
In summary, we found that our data driven methods effectively
improved upon prior work.
In particular, our incorporation of context information
allowed our method to deal with complex spatial patterns
without an ad hoc post processing step.
In the future, we'd like to incorporate
temporal information into the model
as this can improve accuracy.
We would also like to build upon the robot setup
we presented by applying this technology
to the creation of interactive displays
in mobile social robots.
Please see our website for further results
and demonstrations.