Hello everybody.
My name is Hooman Hedayati.
And I do human-robot interaction.
In this video.
I'm presenting our work which done with Dan Szafir at
the University of North Carolina Chapel Hill,
And ATLAS Institute at the University of Colorado Boulder.
In the next 10 minutes, I'll show you how robot can predict the position
of undetected people in the conversational group.
Look at this example,
three people are in the conversation on group with the
robot and another person,
Joins.
The robot can detect the three people with their special hats,
but cannot detect.
The fourth person. Using our method,
the robot is able to predict the position of undetected
person and look in their Direction.
My vision of the future is that people accepted robots
in their daily life.
In order to achieve that Vision,
I'm focusing on human-robot conversation groups.
The definition of F-formation by Kendon is: An F-formation arises
Whenever two or more people sustained the special and orientational
relationship in which the space between them is one to
which they have equal, direct, and exclusive access.
Detecting F-formations is easy for us and
almost hard-coded in our brains.
Here's an exercise for you.
If I show you a scene like this one and
ask you to select the people who are in the
conversational group with each other.
You easily can do that.
Let's give it a try ready.
Probably you come up with the answer.
We have it F-formation of size 8,
size 4, size 3,
size 2 and that's all.
F-formations can appear in different Arrangements.
In general, F-formation that have more than two people tend
to appear in the Circular Arrangement,
While the F-formation of size 2 can appear in
three different arrangments,
L arrangement.
Vis-a-vis Arrangement and side-by-side arrangement.
Each of these Arrangements can give us information about
the activity happening in the F-formation.
For example,
when people tend to stay on the side side-by-side Arrangement,
they are either sharing a resource like a map or
Looking at,
an artifact,
like a poster.
Imagine you are in an F-formation with a robot and
ask a question. In order for the robot to interact with you.
It relies on its sensors like RGB cameras and depth sensors.
This information then passes to human detection algorithm.
And then the F-formation detection algorithm.
All these data then pass to the interaction engine
that choose the robot action.
In this case.
It is going to be responsive to you,
responsive to you.
The problem is that the first three components are prone
to errors cameras can be noisy because of the lighting
or they can simply mask by an object because of
the noisy input the output of human detection.
Algorithm might not be accurate.
Same as to F-formation detection algorithm.
The faulty outcome of F-formation detection,
algorithm can harm the human-robot in Interaction in two
ways.
The first scenario is that F-formation detection
algorithm detects.
A person who is not part of the conversation.
For example,
the person who is just passing by,
then the robot action can harm the interaction in many
ways,
such as not respecting people's privacy and sharing private information.
The second scenario is that the F-formation detection algorithm detects,
not all the members in an F-formation.
In this case,
robot cannot see the user.
So,
the user felt ignored and this harms the
human-robot interaction.
To overcome this problem.
We designed and implemented the method that checks the outcome
of F-formation detection algorithm.
This helps for robot to detect possible,
missing people and plan for them,
for example,
choose and more cautious actions like asking for clarification from
the users.
Our approach Works based on an observation,
we had when we were comparing it F-formation of the
same size.
We select all the F-formation of the same size in
a dataset.
And then we laid those F-formations on top of each other.
First,
we select an F-formation as a frame of reference.
Then for all other F-formation,
we choose a random person in that F-formation and
we used transformation matrix.
So the two selected people have the same position and
orientation.
Like that.
after plotting,
the heat map of all,
observed F-formations,
We find out that in a standing free-standing information,
people tend to stay on the predefined position,
which can be predicted.
You can see the heat map generated for three different
data sets.
Using this property,
we trained a classifier that checks the position of detected
people in the F-formation and Compares it with the training
data set.
So in this example because all the people are staying
in the highlighted position,
The Robot knows that everything is fine.
If an undetected person joins,
the F-formation,
people change,
their positions slightly, to be welcoming to the
the newcomer.
When people change their position then the robot knows,
Although,
It detected only four people in this Arrangement.
The correct number of people in this F-formation is five.
So it detects the error and also predict the possible
position of the newcomer.
We trained and tested our model on two publicly available data sets.
Salsa and Babel data set.
Babel consists of 35 minutes of recording conversation.
Interaction between 7 people,
It boasts a large range of conversational groups of different sizes,
and Arrangement,
and it's publicly available in the link below.
We train our model on 80% of data and tested
on the remaining 20%.
Here are the Confusion matrices for the SALSA dataset
and Babble dataset.
The diagonal of confusion metric,
shows the correct predicted classes.
These confusion matrices show that our approach is reliable way
for detecting error.
Generated by F-formation detection,
algorithms.
The accuracy for the salsa.
Dataset is a slightly less than the Bible data set.
The reason is that the salsa dataset is more noisy,
especially for the orientation of the people compared to the
Babel dataset.
I conclude my presentation with another example,
implemented. An undetected person.
Join the conversation of group and the robot can detect,
the error and look at her Direction.
So in this presentation
I introduced a method that can detect error generated by
F-formation detection algorithms,
which can be helpful in a human robot conversation of
pipeline.
I would like to thank James Kennedy,
who helped a lot with the earlier version of this work.
Moreover.
I would like tanks Shanel.
Niloofar
Elliot and Nick for the footages showed in this
work.
If you have any question or comment,
please email me at
Thank you so much for watching this video.