Hi All, and thank you for your interest in this work.
My name is Amal Nanavati, and today I will be talking about our work modeling social
interaction logic for one-to-many interactions.
This is joint work with Malcolm Doering, Dražen Brščić, and Takayuki Kanda, and was done
at Kyoto University and ATR International.
As we are aware, service robots are being increasingly deployed in human spaces, such
as stores, airports, restaurants, and nursing homes.
As these images show, such deployments inevitably involve robots coming into contact with human
groups.
How these robots should handle such one-to-many interactions, where there is one robot and
multiple humans, is an open question.
At the same time, we are moving towards an Internet of Things world, where cameras, microphones,
and other internet connected sensors will passively collect data about how humans interact
with one another.
This brings us to the core question of our research:
Can we use the passively collected human-human data to learn one-to-many social interaction
logic, without explicit manual labeling?
The HRI community has had many past works in both data-driven learning and multi-party
interactions.
However, these works typically focus on learning low-level components of a robotic system,
such as gaze and turn-taking, and typically require someone to manually label and analyze
the data for the robotic system.
Past work from our lab has focused on learning the high-level interaction logic directly
from human-human data without explicit manual labeling.
This work builds on our past work in the shopkeeper-customer scenario, but extends it from a one-to-one
to a one-to-many scenario.
We began this work by collecting data in a makeshift camera shop.
Our store had three cameras and a shopkeeper’s counter.
A network of 6 Lidar sensors tracked participants as they moved around the store, and they used
a headset and smartphone app to specify the start and end of their utterances.
Each scenario had 1 shopkeeper and 2-5 customers , who were given various group assignments
and objectives to increase the diversity of interactions.
Overall, we collected 266 scenarios, or 28 hours of data, across 51 participants.
Our goal is to develop a system that takes in customer actions online and outputs how
the shopkeeper should respond to them.
Therefore, we must process our dataset into discrete customer and shopkeeper actions.
During the data processing step, we convert our raw data, which consists of position and
velocity per participant over time as well as participants’ recorded audio, into discrete
movement and speech actions.
Movement actions consist of participants moving from one discredited location to another,
and speech actions consist of their vectorized utterances.
The paper has more details about our data processing pipeline.
As we mentioned, our main extension to past work involves moving from the one-to-one to
the one-to-many scenario.
Let us watch a video to understand the challenges in one-to-many interactions.
In this scenario, the shopkeeper must determine that the utterance is not directed to her
but rather to another customer, and therefore she should not respond.
In this scenario, the shopkeeper must balance between the needs of multiple customer groups
and determine who to help.
Let us keep watching.
In this scenario, customers are acting asynchronously, and the shopkeeper must be able to decide
which customers to respond to and which to ignore.
As we saw, one of the challenges in this scenario is the asynchoneity of customer actions.
Consider this sample interaction, where there is a group at the Canon and one at the Sony
camera, but their actions are interspersed.
One way of modeling the actions and reactions is in a Global Sequence, where every customer
action has a shopkeeper reaction.
In that case, consider this scenario before Time 9.
The Global Sequence would assume that the shopkeeper at time 9 is responding to the
customer at Time 8, when that action is actually just a byproduct of the asynchroneity.
However, since the shopkeeper acted, we do know that they are responding to at least
one customer.
Therefore, instead of a Global Sequence, we use a Per-Customer Sequence, where we separate
actions based on the customer who performed them.
Now, for each shopkeeper action, there are up to 5 candidate actions.
However, we are guaranteed that at least one of them is the action that caused the shopkeeper
reaction.
This leads us to the following two-part neural network for predicting shopkeeper reactions
to customer actions.
The Attention Network takes in all recent customer actions, as a Per-Customer Sequence,
and outputs which one customer is most important for the shopkeeper to pay attention to.
This is then passed to a filter.
If the customer who acted just now is not the one the shopkeeper should be paying attention
to, their action gets filtered out.
On the other hand, if the customer who acted is the one the shopkeeper should pay attention
to, that action is passed on to the interaction network, as part of a global sequence that
consists of only important actions.
The Interaction Network then outputs how the shopkeeper should respond.
However, we cannot train this network end-to-end, since the filter in-between the two networks
picks out one action from five contenders, which is not a differentiable function.
In terms of training the networks separately, we do have ground truth shopkeeper actions
to train the Interaction Network, but we do not have the ground truth for who who the
shopkeeper should pay attention to since we cannot get into the shopkeeper’s head.
Therefore, how do we get ground truth attention labels?
Although we don’t have ground truth labels for who the shopkeeper was paying attention
to, we know that the shopkeeper action should be most highly correlated with the customer
action that caused it.
Therefore, if we consider the five contender customer actions that could have the shopkeeper
reaction, only one should be highly correlated with the shopkeeper reaction.
This brings us to the Causal Inference Network, which is a helper network intended to learn
this correlation.
The Causal Inference Network is only trained on the five contender actions at every timestep
before the shopkeeper acts.
It is similar to the combination of the Interaction and Attention Networks, except we make it
fully differentiable by replacing the argmax and filter with a differentiable Gumbel-Softmax.
This allows us to train it end-to-end.
After training this network, the outputs of the Gumbel Softmax layer are used to create
the labels for the Attention Network.
In summary, we have a multi-step procedure for training our proposed system.
First, we train the Causal Inference Network only on the timesteps before the shopkeeper
acts, to learn attention labels.
We then extrapolate those attention labels to all timesteps, which is explained in-more-detail
in the paper.
We then use those Attention Labels to train the Attention Network.
Once that is trained, we filter out the unimportant customer actions, and finally train the Interaction
Network.
We evaluated this network offline, using hold-one-out cross validation on a balanced test set consisting
of 5% of the data.
We ran two evaluations, one to determine the correctness of the predicted shopkeeper actions,
and the other to determine the accuracy of the attention predictions.
We used human coders for both evaluations.
For both evaluations, we ran McNemar’s test for paired binary data to determine statistical
significance.
This is a picture of the coding tool used.
In our first evaluation, we wanted to determine whether the Attention Network improved the
correctness of the predicted shopkeeper actions.
Therefore, we compared two networks.
The Proposed system is the full network we presented earlier, and the Without Attention
system is that system without the Attention components.
In other words, the Interaction Network of that system takes in all customer actions,
not just the important ones, and predicts shopkeeper reactions.
We look at the correctness of predicted shopkeeper actions, as determined by our human coders.
Our results show that the Proposed system is, in fact, better than the Without Attention
system, with 77% correctness compared to 65.2%, which is a significant difference.
In the second evaluation, we investigated whether using the Causal Inference Network
to learn attention labels improved attention accuracy.
Again, we compared two systems.
Our proposed system is the same system presented earlier, where the Causal Inference Network
learns the attention labels.
Our Without Causal Inference system, on the other hand, jointly learns the attention labels
and shopkeeper actions, by replacing the non-differentiable filter with a differentiable Gumbel-Softmax
layer.
This time, we compare the systems’ predictions of which customer is important to pay attention
to, compared with the human coders’ labels.
Our results show that the Proposed system is, in fact, better than the Without Causal
Inference system with 90.7% attention accuracy compared to 61.5%, which is a significant
difference.
In conclusion, in this work we developed a system to learn, without explicit manual labeling,
how a shopkeeper should respond to multiple human customers.
We then demonstrated the benefits of first learning whom to respond to, the Attention
Network, and then learning how to respond to them, the Interaction Network.
Finally, we demonstrated the benefits of a novel technique to learn which customer likely
caused shopkeeper actions.
Thank you, and we’re happy to take any questions via email.