When we deploy learning-based robots in the real world,
they need to be able to autonomously recover
when their policy fails, without fully relying on experts.
I’m delighted to present our work on correcting high-level robot policies
using non-expert feedback.
Failing policies can be due to changes in the environment
that the robot has not been trained to handle.
The availability of objects might change.
For example, the robot might fetch an item that is no longer available.
Or actions that were previously successful can
suddenly lead to undesirable outcomes.
For example, the robot may get stuck on a carpet,
if it has never seen this before during training
and hence, never learned to avoid it.
Typically, such failures require costly expert intervention,
or substantial time or data to retrain the policy.
Yet, certain failures, especially when they happen in everyday context,
are also understandable by people who do not necessarily have robotics
or programming experience.
In such cases, these people, or non-experts,
could provide additional rules to the robot
that can help it prevent the failure on future occasions.
We are motivated by the question: “how can we automatically correct robot failures...
... after deployment using non-expert feedback?”
Prior work has looked into intuitive ways
for humans to teach robots.
Learning from demonstration, for example,
lets non-experts quickly teach robots new behaviors
through kinesthetic guidance.
Interactive reinforcement learning is used
to teach robots sequential decision-making tasks,
for example, using positive or negative rewards, corrections or preferences.
These works highlight the potential of human
feedback in reinforcement learning, yet, often require teachers
to constantly monitor the entire training process.
We, instead, query people only after a failure has occurred.
So, when we have obtained human input:
How can we enforce correct robot behaviors?
Constrained reinforcement learning enforces
desired behaviors by restricting the state
or action spaces, providing penalty rewards,
or by interfering with the policy update.
However, it requires technical expertise and
can often not formally ensure that undesired
behaviors are always avoided.
Such formal guarantees are crucial for robots’
real-world deployment and trustworthiness.
Temporal Logics are a popular tool to define
formal constraints on robot behavior.
With their resemblance to natural language,
they balance the trade-off between rigorousness
and simplicity for non-experts.
While such temporal logic-based approaches are effective,
they often interfere with the underlying reinforcement learning problem.
For example, they prune actions and paths
in the model, which potentially limits the agent’s exploration.
Rather than directly changing the policy,
shielded reinforcement learning restricts the agent's actions.
This way, it minimally interferes with the model while still enforcing desired behaviors.
Shields can be constructed from temporal logic specifications
to define a set of constraints.
However, experts are often required to construct
these specifications or to turn them into shields
and often, an automaton with all possible
states needs to be constructed to conduct safety checks.
Instead, we apply shields only to the states
the agent actually visits, which reduces computational complexity.
Taking the prior efforts from both research fields
we propose to leverage non-expert
input in a policy repair approach that automatically
synthesizes correction mechanisms, or shields,
to correct high-level robot behavior.
Specifically, the robot is trained as a reinforcement learning system
and deployed in a different environment.
Upon failure, it queries a non-expert
for a correction that is then translated into a shield,
with which the system is retrained.
This way, robots do not need to fully rely
on experts to resolve failures and we can
formally verify the corrected behaviors.
We investigated the following two research questions:
“Can we use shielding to correct high-level robot behaviors?”
Which we tested through a series of experiments.
And, “If/how can non-experts provide input to correct failures?"
When the robot queries them only after it
detects a failure state, which we evaluated
through an online study.
I’ll start with the experiments to answer
the first research question and will then
talk about the online study to answer the second.
We conducted a series of simulated experiments,
in which we construct shields to correct high-level actions
in a cooking task from ground-truth feedback.
We also create shields from incorrect feedback
to gain insights into their effect on the
failure recovery.
This feedback was obtained from the online study,
which I will discuss in more detail in a moment.
We showcase our approach in a 2D simulated
kitchen environment which is adapted from prior work.
In three experiments, we collected input for
the following types of failure corrections:
action refinement, alternative item, and forbidden action.
I’ll go through each in detail now.
Action refinement refines a failed action by executing another
action or sequence of actions instead.
The agent has been trained to assemble and
deliver a salad, however, after deployment
a tomato slice is no longer available.
We apply a correct shield that refines the
action of ‘fetching a tomato slice’ with
fetching a whole tomato and cutting it into slices first.
The incorrect shield refines the failed action
by fetching a plate and then fetching a tomato slice again.
Our approach converged 30% faster on average
and obtained less negative rewards than without a shield.
With the incorrect shield, the task is not learned,
because the shield calls the failed action,
without solving the issue first.
When the agent tries to fetch a tomato slice,
it fails again, resulting in an infinite loop of corrections.
The alternative item shield corrects the failed action by
replacing the missing item with an alternative.
The agent has been trained to bake and deliver a cake.
However, after deployment
wheat flour is no longer available.
The correct shield replaces wheat flour
with the alternative: almond flour.
The incorrect shield refines the failed action
with the following action sequence:
fetching a bowl, fetching eggs, and then fetching
wheat flour again.
Our approach avoids the failure
while being more data-efficient.
It learns the task successfully and converges
after 104 episodes on average.
Without a shield, the policy is unable to
robustly learn the task or obtain positive
rewards, even when trained for longer.
The incorrect shield again resulted in an
infinite loop of corrections, like in the
salad experiment that we saw earlier.
The forbidden action shield removes undesired actions.
The agent has been trained to deliver a coffee.
However, after deployment one floor
tile is occupied with a carpet.
We apply a correct shield that constraints
the agent’s behavior such that it never
moves onto a carpeted floor tile.
The incorrect shield refines the failed action
by instead moving one tile down, one right, and one up.
Our approach learns the task
successfully and converges after 24 episodes on average.
Without a shield, the policy does not learn
the task and only obtains negative rewards.
The line denoted as ‘penalty’ in the graph shows the
results for providing an additional penalty
of −1 for moving to carpeted tiles.
This extra expert knowledge helped the policy
to converge, yet it still requires 25% more episodes
and has a less stable learning curve,
compared to our shielded learning.
The incorrect shield performed well.
The agent learned that with activating the shield,
it actually ends up at the delivery tile.
So, to summarize, this series of three simulated experiments
showed that we can use shielding
to correct high-level robot behavior.
Now, let’s dive into how we obtained the
feedback that we used to generate these shields.
We conducted a within-subjects online study,
in which all 18 participants went through
the three experiments in random order:
the salad, the cake, and the coffee experiment.
Participants were 5 female and 13 male, and
between 20-63 years of age, with little to
no prior experience with robots and programming.
They were provided with an introduction
to robot failures and rules to correct them.
They saw the agent in its environment and
a list of all possible objects, actions, and locations in the environment.
They were asked to help the robot by describing
what caused the failure and by giving a rule
that would help prevent the failure on future occasions.
The results showed that the majority of the
participants could successfully understand
and identify the failure, as well as correct the failure successfully.
To answer our second question, we found that
non-experts can indeed provide the input that
is needed for the generation of the shields.
So, what does this mean?
This work is a first step towards a full non-expert policy repair pipeline
and our proof-of-concept implementation validates for the first time
that shields can be used to repair high-level policies after failures.
However, we have not yet automated the shield synthesis from human feedback,
and we acknowledge important avenues of future work.
For example, automatically identifying what
is incorrect and correct feedback
can be challenging.
In the paper, we describe an initial clustering
of the open answers based on text similarity.
Moving forward, multiple non-experts can be asked
to judge whether corrections are sensical
before sending them to the robot.
Additionally, our results showed that incorrect
shields might still work well for one specific case,
but fail to generalize.
We are interested to further investigate how
we can automatically test how effective and
generalizable corrections are.
For example, by validating corrections in a set of benchmark scenarios.
Moreover, we conducted simulated experiments
and in next steps we want to test our approach with a real robot.
This means we need to investigate what level of abstraction
is needed for the robot’s actions and states,
with the aim of creating an intuitive interface
between low-level robot control and high-level corrections.
This also means that we need to explore how
we can leverage sensor fusion to provide richer
representation of the environment and failure
in the robot’s queries, so that we can evaluate what
information representations help non-experts
repair the robot’s policies effectively.
Our approach is applicable to any sequential
decision-making robot that performs common tasks.
For example, a robot in a table-setting task
might place objects according to a set of constraints.
Non-experts can specify rules to define desired
relations between the objects over time
to ensure the correct task configuration.
For example: at some point, place the sandwich on the plate.
Robots can use on-the-fly non-expert feedback
to synthesize controllers to adhere to such rules.
To wrap up, we showed that non-experts are able to
provide corrections from which shields could
be generated and that shields can be used
to repair failures of learning-based robots in a verifiable fashion.
As a first answer to our overall research question,
as posed at the beginning of this talk:
what this work showed is that combining
non-expert feedback and shielding has the
potential for automatic verifiable failure correction.
Thanks for listening!
If you have any questions, feel free to contact me
via the e-mail address listed right here.