Does the HRI community currently have sufficient
knowledge of what makes a system trustworthy to be able to design robots as such?
The authors of this paper, who all have different disciplinary backgrounds ranging from philosophy
and AI to mathematics and logic, met at a Dagstuhl seminar and discussed exactly this.
In this paper, we present the outcome of this discussion: A taxonomy of trust-relevant failures
and mitigation strategies, based on literature as well as empirical data from a real-world use case.
So if we talk about trust in a robot we also
have to discuss how trust can be violated. Trust repair can be understood as the activity
of rebuilding trust after one party breaks the trust of the other.
But what causes these trust violations in HRI, can we distinguish different failure
types? And how can trust be repaired after it was broken, which mitigation strategies
are worthwhile? The inspiration for our taxonomy stems from
so-called risk tables. It presents an overview for failure situations in HRI that can harm
trust in the robotic system, and offer robot designers mitigation strategies to (1) avoid
or (2) recover from failure and reestablish trust.
As a starting point for our discussions, we defined trust as “a person’s willingness
to rely on a robot to carry out its duties”. As HRI involves two different actors, namely
the robotic system and the human interacting with it, we base our taxonomy on a first fundamental distinction:
who performed a type of action which caused a break of trust
(1) the user or (2) the robot. Next, we distinguished the failure type.
We identified four different failure types with respect to their impact on trust
and the related mitigation strategies:
(1) Design, (2) System, (3) Expectation, and (4) User
So let’s go into detail with our failure categories:
In a Design failure: the System does what it’s been made to do,
but in retrospect the system should not actually behave this way.
Imagine you have designed a robotic system in a specific way (in terms of behaviour, appearance, dialogue and so on) to the best of your knowledge.
While in the real world, the system behaves exactly the way you intended it to do,
it turns out that you made design choices that were not ideal for the people interacting with the robot.
This is a failure by design.
In a System failure: the System doesn’t do what it’s been made to do. For example,
the robot stops in the middle of a room during a navigation task without a reason, or stops
a scanning task because its scanner malfunctions. In other words, the system does not do what
it should, for example because of a system crash. Here, we can distinguish into hardware and software failures.
In an Expectation failure: the System does what it’s been made to do,
but users expected something different to happen.
In retrospect, the system should still behave this way. For example, when the user expects a robot to turn
while observing a room, but the robot does not need to do so, the system performs as it should,
but confuses the user. This is an example of an omission failure:
the robot does not act when the user expects that it will. The opposite of this is a commission failure:
the robot does something the user does not expect, such as starting to move
in the middle of an interaction, because it needs to charge its battery.
A User failure happens, when users behave in a way they are not supposed to, which might lead
to another failure type. For instance, if the user is intentionally disturbing or
sabotaging the robot or unintentionally standing in the robot’s way so it cannot move.
A user failure can be caused either by a Design failure or an Expectation failure which influences
its impact on trust and potential mitigation strategies. While an Expectation failure deals
with what the user expects the robot to do, a User failure is about what the users themselves do.
Of course, Expectation failures could lead to unintentional User failures.
Depending on the type of failure that has taken place, there are different possible
mitigation strategies that can help regain the trust of the user. We compiled this information also from related literature.
For Design failures the following mitigation strategies can be used:
First, we can try and fix our interaction design; second, the robot
could offer an explanation what went wrong, and third the robot could apologise.
For System failures, the robot could also give an explanation or an apology, but mainly
we need to fix the failure for sustainable success. Depending on the type of system failure
it could also be an option that the robot offers an interaction alternative to the user.
For Expectation failures, the same options are available as for Design failures:
Interaction Design, Explanation, and Apology. Our discussions on User Failures revealed
that there is little to no research on how to mitigate this type of failure. We consider
the following strategies as promising: Clearly, we should again try to improve
our interaction design, but we could also ask the human for justification, to raise the
awareness of the intentional or unintentional misuse. The robot could show emotions such
as sadness or anger when treated wrongly. For robot bullying, involving an authority figure
could be an option, for instance the robot saying: “I will notify my owner now”.
And finally training end users to avoid future misuse, could be a strategy.
Given the interaction between different failure types, mitigation strategies for the
initial failure type should be applied first. For example, if an Expectation failure was
caused by a Design failure, the Design failure should be considered first.
To support our taxonomy with empirical evidence we analyzed data of an autonomous mobile robot
that has been deployed in a care home for a total of over a year. We used data on task
and error logs, user demographics, and navigation failures and analyzed it to generate a risk score.
The probability score can range from 1:
1 occurrence in about 1000 interactions to 4: likely to happen in every interaction episode.
Similarly, the impact score can range from 1: minor impact to 4: fatal impact, with which
we mean potential loss of trust and further use. The risk score is finally calculated
by multiplying probability and impact. So for an expectation commission failure
a probability of "2" multiplied by an impact of "4" leads to an average risk score of "8".
For the example of an intentional user failure, we counted the number of forceful robot pushes
and deliberate tampering. We observed about 1 out of 200 interactions, which leads to
a probability scoring of "2"., however, those have minor impact on future use and thus lead to a low risk score of "2".
You find more details on this also for the other failure types in the paper.
To summarize, in this paper, we presented
a taxonomy with the aim to help fellow researchers developing trustworthy robots according to the state of the art.
We presented failure types, mitigation strategies, as well as risk
scores derived from a case study. We hope that fellow researchers build on our work
and extend it with empirical evidence.