Hello, my name is Angelique Taylor.
I am a Visiting Research Scientist at Meta Reality Labs,
a Postdoctoral Researcher at UC San Diego
, and I’m an incoming Assistant Professor at Cornell University starting this summer.
Our research explores how to design
robots that can work in teams with groups of people.
To achieve this goal,
we developed a robot perception system called REGROUP
which enables robots to detect and track groups of people
in real-world, human-centered environments.
There are several reasons why group perception is important
to enable human-robot teaming
Humans are social and they often
interact in groups in real-world settings.
By understanding social relationships between people,
robots can both predict people’s motion trajectories, and gain contextual information
about the environment (for example, when it’s appropriate to enter or exit an interaction with people).
However, it is challenging for robots to interact with multiple people in an automated way.
For example, people are unpredictable and sometimes behave in unexpected ways.
For example, consider this environment,
recorded from an ego-centric RGB-D sensor mounted on our mobile robot.
As you can see, this is a pretty challenging space
the robots is in motion, people are in motion,
and people have dynamic and chaotic motion trajectories.
As such, we make no assumptions about prior knowledge,
but instead, use intuition about how we as humans identify groups as a driven factor for our system design
In particular, we were curious,
How can we solve perception problems in everyday environments?
when we cannot rely on overhead cameras nor a priori knowledge,
and must solve these problems onboard a mobile robot
with limited computational resources.
We are particularly interested in ego-centric (or first-person) perception methods
for mobile robots,
that can perform unsupervised in real-world environments.
We address this problem
by exploring two research questions.
First, how can a robot track groups of people
from ego-centric perspective
on a mobile platform in crowded environments,
where people are in motion and the robot is in motion?
Then, how can we use person re-identification Convolutional Neural Networks (CNNs)
to address the group data association problem?
We built REGROUP
an ego-centric group detection and tracking system that runs in real-time.
This is a high-level overview of how the system works.
The goal of a group tracker is
to preserve the identity of groups of people over time.
It takes as input, RGB data or video stream
and it outputs a group track ID for each pedestrian in the scene
Unlike prior methods that require multiple sensors and high computational resources
our system enables robots to detect and track groups
in real-time from a mobile platform, using a single RGB sensor.
Now I will discuss the technical details of each step.
The first thing REGROUP needs to do
is detect pedestrians in the scene.
Rather than reinvent the wheel, we employ YOLO
which provides real-time performance on a GPU.
We divide the image in a grid, which generates
a class probability map and bounding boxes with class confidences.
Then, we perform regression on these data
to infer bounding box coordinates of people in the scene.
REGROUP tracks pedestrians using Kalman filtering
It uses two distance metrics which represent distances
in terms of motion and appearance
then we combine them using a weighted sum of the two
The intuition here it that people's appearance
does not change much from time from one timestep to the next
Therefore, by leveraging appearance features
from person re-identification Convolutional Neural Networks,
we can use this as a distance metric to re-identify people
for the data association problem
Now, I'll talk about our new ego-centric group detection system
Our goal is for this system to be
deployed on a mobile robot from its onboard sensor
Thus, the group detection system uses three distance metrics
which characterize groups, particularly
particularly for ego-centric perspective sensing applications.
We construct three adjacency matrices,
D_w, D_h, and D_gp,
to capture whether two pedestrians are in the same group.
These matrices capture the
height ratio, ground plane distance, and width
between pedestrians respectively
We apply thresholds \beta, \alpha, and \kappa
to these matrices to exclude group candidates with pedestrians
that are not physically close to each other.
Where \beta$is the mean width
of two pedestrians' bounding boxes
\alpha is a number ranging between 0 and 1
and is the height ratio between two pedestrians
\kappa is a number that ranges from 1 to the height of the image
and is the ground plane distance (in pixels)
between two pedestrians
We determined \alpha and \kappa through experimentation
We aggregate the distance matrices
into a single adjacency matrix which identifies group candidates
In the final step, we use depth first search
on this matrix to find cycles where cycles indicate groups
A novel strength of REGROUP is that it introduces
a new way to perform group re-ID
to address the group data association problem.
REGROUP generates group appearance descriptors
by merging pedestrians' appearance descriptors
We employ the Hungarian algorithm
o address the assignment problem for group detections and tracks
followed by Kalman filtering to track groups over time
One challenge that we encountered while developing REGROUP
is that the distance metrics
often converged to zero
for short periods of time
when multiple groups pass each other
To address this challenge
we developed a new method called a Crowd Indication Feature
to detect this situation by using
past track states to preserve group track IDs over time.
For example, the left image shows
an example of REGROUP detecting no crowd
and the right image shows an example
of a crowd being detected
For more details on how this works, please see our paper
To test REGROUP
we collected a large-scale RGB dataset
which shows a crowded, sunny, outdoor environment.
This resulted in many computer vision challenges
such as occlusion, shadow, and varying lighting illuminations
which is actually great, and what we want to test against
We mounted a ZED stereo vision sensor
on a Double Telepresence robot
and let the robot roam around a large public park
We annotated the dataset and validated our annotations
consistent with popular computer vision benchmarks
Here is a demonstration of REGROUP
running on a mobile robot in a park
The red boxes show the individual pedestrian detections.
Groups are identified with the same color bounding box
and track ID
Even with camera ego-motion
we see that REGROUP performs pretty well
In order to experimentally validate
REGROUP’s group detection method
we compared it to four state-of-the-art methods
We compared REGROUP to Spencer by Linder et al.
NCuts by Shi and Malik
Self-Tuning-Spectral Clustering by Zellik-manor and Perona
and RoboGEM, the prior version of our system
We evaluated these methods using
standard metrics from the pedestrian detection literature
including precision and recall vs intersection over union (or IOU)
For those not familiar
IoU measures the overlap between two bounding boxes
where an IoU of 0 means the boxes don’t overlap at all
and an IoU of 1 means the bounding boxes overlap perfectly
The standard IoU is 0.5 in computer vision
in which case identifies a correct detection
This Figure shows the group detection results.
Overall, REGROUP outperformed all other methods
in group detection by up to 40% in terms of precision
and up to 18% in terms of recall
Self-Tuning-SC outperformed Spencer
and Spencer outperformed NCuts+K in terms of precision
Overall all methods outperformed RoboGEM
In order to experimentally validate REGROUP’s
group tracking method, we compared it
to three state-of-the-art methods including
Spencer by Linder et al.
Group-LSTM and Group-LSTM-Obst by Bisagno et al
We evaluated the group tracking methods using
standard classification of events, activities, and relationships (CLEAR)
which are standard metrics from the multiple object tracking literature
We report a more comprehensive list
of metrics in the paper but we will summarize here
for the sake of time
We report multiple object tracking accuracy
which combines false positives, false negatives, and ID switches
between group tracks
and we compute the multiple object tracking precision
which is the misalignment in bounding boxes
between ground truth and predicted tracks
Overall, these metrics measure the accuracy
of tracking methods and their ability to track long-term
Overall, REGROUP outperformed all comparative methods
in group tracking by up to
66% in terms of tracking accuracy
and up to 20% in terms of tracking precision
Now, we can start thinking about human-robot teaming
in situations where there are distributed teams
in the same environment
For instance, applications in manufacturing, healthcare, and autonomous vehicles
can use REGROUP to track groups
to enable more safe and socially aware robots
In the future, we are excited to continue building
upon REGROUP and deploying it in conjunction
with navigation systems to enable
real-time human-robot teaming
There are several contributions of this work
First, we introduced a new semi-supervised
egocentric group detection and tracking system
called REGROUP
We showed that REGROUP outperforms four
group detection methods
and three state-of-the-art group tracking methods
and we showed that REGROUP is robust to real-world vision challenges
such as occlusion, camera ego-motion
shadow, and varying lighting illuminations
and we also showed that REGROUP runs
in real-time on real-world data.
I would like to give a special thanks
to my collaborators and those that supported my research
For those interested in deploying REGROUP
on their own robot
our code is provided on GitHub
Thank you