Hello my name is Mariah Schrum
and Erin Hedlund-Botti and I will be presenting on our paper MIND MELD:
Personalized Meta-Learning for Robot-Centric Imitation Learning co-authored by Nina Moorman
and Matthew Gombolay.
For robots to operate in the real world they must be capable of learning novel tasks from non-expert
humans. However, prior work has shown that human are suboptimal when providing corrective
demonstrations to robots which makes it difficult for robots to successfully learn from humans.
To overcome this limitation, robots must take into account human suboptimality when learning
new tasks.
Learning from demonstration can be broken down into two main types: human-centric LfD
and robot-centric LfD. In human-centric LfD, the human is in control and demonstrates the task to
the robot. An example of human-centric LfD is behavioral cloning in which the demonstrator
provides a series of demonstrations and the agent learns via supervised learning.
While human-centric LfD is intuitive to the user, it suffers from covariate shift problems and
performs poorly in stochastic environments. Robot-centric LfD attempts to overcome the
covariate shift problem. In robot-centric LfD, the robot rolls out a policy and the human
provides corrective feedback while observing the robot. The current state of the art for
robot-centric learning from demonstration is based upon Dataset Aggregation or DAgger. In DAgger,
the human’s corrective feedback which is aggregated each iteration and the agent is retrained on the new
dataset. While this algorithm works well if the labels provided by the human are high quality,
prior work has shown that humans often provide poor labels. To overcome this limitation,
we propose to learn to map poor human labels to better labels, thereby improving on the
performance of robot-centric LfD.
To improve upon the deficiencies in robot-centric LfD, we introduced Mutual Information Driven
Meta-Learning from Demonstration or MIND MELD. To better learn from suboptimal
corrective labels, MIND MELD meta-learns an individual-specific mapping from human labels
to predicted optimal labels via a Long Short-Term Memory (LSTM) based neural network architecture.
We demonstrate our architecture in a driving simulator domain. In this domain, the robot rolls
out its learned policy and the human provides corrective feedback that may be suboptimal.
Based upon the feedback the human provides we learn a personalized embedding in
d-dimensional space that describes their style of corrective feedback.
In this example, we see that this individual is an over-corrector, meaning that they tend to turn the
wheel too far when providing corrective feedback. Based on their learned style,
we then map their corrective labels to better labels so that the LfD agent can more effectively
learn from the human.
We employ calibration tasks to learn an individual’s style of providing
corrective feedback. These calibration tasks consist of wizard-of-oz policy rollouts,
representative of the task space with known optimal labels. We determine the optimal
labels via RRT* and model predictive control. For each calibration task, the agent rolls out
a wizard-of-oz policy as shown on the right and the demonstrator must provide corrective feedback.
We train our MIND MELD architecture on the optimal labels and corrective actions provided during the
calibration tasks to learn the personalized embedding and mapping from suboptimal labels
to better labels.
Here we show our architecture which is trained on the calibration tasks. Our network
architecture simultaneously learns the style of the demonstrator via a personalized embedding w,
and learns the mapping from demonstrator label a sub t to the difference between the demonstrator
label and the ground truth. Our Bi-directional LSTM learns the embedding z which encodes
temporal information about the demonstrator’s suboptimality. The subnetwork, f sub theta maps
this encoding to the difference between the human provided label and the ground truth.
We include the subnetwork g_phi which approximates the probability distribution from which w is drawn
and is necessary to maximize mutual information.
To test the ability of our architecture to learn suboptimal demonstrator styles and improve
suboptimal demonstrations, we create a synthetic experiment with synthetic policy rollouts,
demonstrators and corrective demonstrations which we utilize to train our MIND MELD architecture.
In our experiment, we show that similar demonstrators tend to cluster together,
suggesting that our architecture is able to learn meaningful embeddings.
Additionally, we find the we are able to improve upon the suboptimal corrective labels
by 55%.
We next conduct a human-subjects study using the driving simulator domain,
Airsim and an Xbox steering wheel. Autonomous driving has an intuitive interface for humans and
is a common domain used in prior work to evaluate LfD algorithms. The objective in this domain is to
teach the car to drive from a start location to the orange ball while avoiding collisions.
We recruit 76 calibration participants to complete the calibration tasks and learn the
network parameters for MIND MELD. We next freeze the network parameters
and conduct a test study in which we compare MIND MELD to DAgger and BC.
We recruit 42 participants for the within subjects study. Note that our population pool consists
mostly of Georgia Tech students.
After completing the calibration study and learning MIND MELD’s network parameters,
we next conduct the test study comparing MIND MELD to baseline LfD algorithms.
In our study, participants first provide demographic information and fill out
a survey on prior experience with driving and video games.
Participants then complete 16 calibration tasks to learn a person’s embedding.
Note that one of MIND MELD’s assumptions is that a person’s embedding is static over time.
In our study the calibration tasks are meant to capture the distribution of
tasks that may arise when teaching a car to drive from point A to point B.
Each participant then teaches the car on a different test task via either MIND MELD, DAgger, or
BC and provides 12 demonstrations. After teaching an agent, the participant fills out surveys on
workload, likeability, perceived intelligence, and trust. All three conditions are experienced
by each participant in a random order.
To evaluate MIND MELD specifically, we measure percent improvement, which is the
amount that MIND MELD improved participant labels compared to ground truth.
Additionally, we wanted to determine what metrics impacted a person’s style
of providing feedback and if they correlated with the learned personalized embeddings.
To measure how much a person over- or under-corrected and if their feedback was delayed
or anticipatory, we measured a participant’s stylistic tendencies. We used dynamic time
warping to temporally match the participant’s feedback signal and the ground truth signal.
We also collected demographic information such as age and gender and a participant’s prior
experience with driving and video games. We found that a person’s embedding correlated with
how much a person over/undercorrected, the amount that their feedback was delayed or anticipatory,
and their experience with video games.
To evaluate MIND MELD compared to DAgger and BC, we measure the objective performance of each
algorithm by measuring the distance from the goal at the end of each trial. Additionally, we measure
the number of times the agent reaches the goal, the probability of the agent reaching the goal,
and how many demonstrations are needed for the agent to reach the goal or sample efficiency.
To evaluate how people experienced MIND MELD compared to DAgger and BC, we measure
participants’ workload to teach each agent and their rating of the agent in terms of likeability,
perceived intelligence, and trust.
Here we see results for the average distance from the goal for each of the agents after each
demonstration. We see that MIND MELD achieves a lower average distance from the goal overall. When
comparing the last trial, we find that MIND MELD achieves a significantly smaller average distance
from the goal compared to DAgger and BC.
Next are results depicting the probability of an agent reaching the goal once over the course of
the 12 demonstrations. MIND MELD has nearly a 100% probability of reaching the goal after training is
complete, while DAgger and BC are significantly less likely to reach the goal.
Here is the probability of an agent reaching the goal twice,
three times and four times. We see that MIND MELD achieves nearly 100% probability
of reaching the goal four times whereas the other agents achieve less than a 50% chance
after training is complete.
If a person provides good feedback, then DAgger may be able to learn well.
We hypothesized that MIND MELD would outperform DAgger the more that a person is suboptimal.
We found that MIND MELD’s advantage over DAgger does increase as a person’s feedback is further
from optimal.
Next we investigate subjective metrics. We find that participants
rated MIND MELD as significantly lower workload than DAgger and BC.
Additionally, participants rated MIND MELD as significantly higher in terms of likeability,
intelligence and trust.
To summarize, MIND MELD has the ability to improve corrective labels to be closer to optimal.
MIND MELD’s learned embeddings correlate with a person’s stylistic tendencies and their
prior experience with video games. MIND MELD outperformed DAgger and BC in
terms of ability to reach the goal and consistency in reaching the goal.
MIND MELD has a greater advantage over DAgger when a person’s feedback is more suboptimal.
MIND MELD was rated better than DAgger and BC in terms of workload, likeability, perceived
intelligence, and trust.
In the future, we want to demonstrate that MIND MELD can generalize across platforms, so we will
evaluate MIND MELD on a robotic arm domain. Additionally, we want to investigate how to map
a person’s embedding across platforms or domains.
Lastly, we plan to learn how to estimate how a person’s embedding changes over time
and how to provide a person feedback on how to be a better
demonstrator.
We would like to acknowledge our sponsors, Georgia Tech, the National Science Foundation, Konica Minolta, NASA,
and MIT Lincoln Laboratory.
Thank you for listening and we would be happy to answer any
questions.