Researchers from MIT and elsewhere have developed a technique that enables a human to efficiently
fine-tune a robot that failed to complete a desired task— like picking up a unique
mug— with very little effort on the part of the human.
Imagine purchasing a robot to perform household tasks.
This robot was built and trained in a factory on a certain set of tasks and has never seen
the items in your home.
When you ask it to pick up a mug from your kitchen table, it might not recognize your
mug (perhaps because this mug is painted with an unusual image).
So, the robot fails.
Researchers at MIT, New York University, and the University of California at Berkeley created
a framework that enables humans to quickly teach a robot what they want it to do, with
a minimal amount of effort.
When a robot fails, the system uses an algorithm to generate counterfactual explanations that
describe what needed to change for the robot to succeed.
For instance, maybe the robot would have been able to pick up the mug if the mug were a
certain color.
It shows these counterfactuals to the human and asks for feedback on why the robot failed.
Then the system utilizes this feedback and the counterfactual explanations to generate
new data it uses to fine-tune the robot.
Fine-tuning involves tweaking a machine-learning model that has already been trained to perform
one task, so it can perform a second, similar task.
The researchers tested this technique in simulations and found that it could teach a robot more
efficiently than other methods.
The robots trained with this framework performed better, while the training process consumed
less of a human’s time.
This framework could help robots learn faster in new environments without requiring a user
to have technical knowledge.
In the long run, this could be a step toward enabling general-purpose robots to efficiently
perform daily tasks for the elderly or individuals with disabilities in a variety of settings.
The research will be presented at the International Conference on Machine Learning.
Robots often fail due to distribution shift — the robot is presented with objects and
spaces it did not see during training, and it doesn’t understand what to do in this
new environment.
One way to retrain a robot for a specific task is imitation learning.
The user could demonstrate the correct task to teach the robot what to do.
If a user tries to teach a robot to pick up a mug, but demonstrates with a white mug,
the robot could learn that all mugs are white.
It may then fail to pick up a red or blue mug.
Training a robot to recognize that a mug is a mug, regardless of its color, could take
thousands of demonstrations.
The researchers’ system determines what specific object the user cares about (a mug)
and what elements aren’t important for the task (perhaps the color of the mug doesn’t
matter).
It uses this information to generate new, synthetic data by changing these “unimportant”
visual concepts.
This process is known as data augmentation.
The framework has three steps.
First, it shows the task that caused the robot to fail.
Then it collects a demonstration from the user of the desired actions and generates
counterfactuals by searching over all features in the space that show what needed to change
for the robot to succeed.
The system shows these counterfactuals to the user and asks for feedback to determine
which visual concepts do not impact the desired action.
Then it uses this human feedback to generate many new augmented demonstrations.
In this way, the user could demonstrate picking up one mug, but the system would produce demonstrations
showing the desired action with thousands of different mugs by altering the color.
It uses these data to fine-tune the robot.
Because their work seeks to put the human in the training loop, the researchers tested
their technique with human users.
They first conducted a study in which they asked people if counterfactual explanations
helped them identify elements that could be changed without affecting the task.
Then they applied their framework to three simulations where robots were tasked with:
navigating to a goal object, picking up a key and unlocking a door, and picking up a
desired object then placing it on a tabletop.
In each instance, their method enabled the robot to learn faster than with other techniques,
while requiring fewer demonstrations from users.
Moving forward, the researchers hope to test this framework on real robots.
They also want to focus on reducing the time it takes the system to create new data using
generative machine-learning models.