Hello everyone, I’m Xinru Wang from Purdue University.
Today I’m happy to present our work:
Are Explanations Helpful?
A Comparative Study of the Effects of Explanations in AI-Assisted Decision-Making
In recent years, many AI-driven decision aids have been developed to assist people in making
better decisions in diverse domains
such as healthcare, finance, and criminal justice.
Many of these decision aids are based on complex, black-box AI models,
which means humans usually cannot understand how these models turn input features into
the final algorithmic decisions.
To overcome this problem,
various methods have been designed to explain AI models and their decisions.
The rapid development of AI explanation methods
raises a key question in evaluating these methods:
What properties constitutes a good AI explanation?
Here we posit three desirable properties that ideal AI explanations should satisfy:
Understanding: Explanations of an AI should improve people’s understanding of
how it makes predictions.
Uncertainty awareness: Explanations of an AI should help people recognize the uncertainty
underlying an AI prediction;
in particular, AI explanations should nudge people to rely on the model more on those
high confidence predictions.
Trust calibration: Explanations of an AI should empower people to trust the AI properly;
so with explanations, people should be more likely to trust the AI when it’s correct,
and less likely to trust it when it’s wrong.
Given these three desiderata, we ask:
Do established AI explanation methods satisfy these properties?
To answer this question, in this project,
we conduct an experimental study to understand
how various AI explanation methods influence people’s understanding of the AI,
people’s ability to tell AI’s confidence,
and people’s levels of trust calibration in the AI.
Before we start our experimental study,
we have reviewed some recently published studies on empirical evaluations of AI explanations.
This table briefly summarizes the results of these studies.
Overall, we find the current empirical evaluation results are incomplete and quite mixed.
For example, some desiderata like uncertainty awareness and trust calibration have rarely
been examined by existing studies.
Moreover, different studies often examine
different types of AI explanations in the context of different tasks,
but some properties of the tasks,
such as people’s level of domain expertise in the task,
may largely affect the ways that people utilize explanations and therefore affect the effectiveness
of the explanations.
In light of this, we find it necessary to carry out a comparative study
to systematically examine
the effectiveness of different AI explanation methods on
different types of decision-making tasks.
So we start by selecting two types of decision-making tasks:
The first task is recidivism prediction, in which participants are asked to review a profile
of a criminal defendant,
and to predict whether this defendant would reoffend within two years.
The second task is forest cover prediction.
In this task, participants are shown a geological profile of a wilderness area,
and then they need to predict whether this area
is primarily covered by the spruce-fir forest.
We conjecture that for most people, they would feel that they have more expertise in the
recidivism prediction tasks,
because they could apply their day-to-day, common sense knowledge in such predictions.
Indeed, through a pilot study, we found that 83% of participants reported themselves
to be more knowledgeable on making recidivism predictions than in making forest cover predictions.
Regardless of which type of tasks a participant works on, in each task,
they go through the same three steps:
They first make an initial independent prediction,
Then they will review the AI model’s prediction,
with or without explanation depending on the treatment they are assigned to,
and lastly, make a final prediction.
Now let me tell you a bit more about our experimental treatments.
In total, we create 5 treatments by varying whether and how the AI model is explained:
In the control treatment,
participants would not receive any explanation on the model’s prediction.
In the Feature importance treatment, we provide a global explanation of the model by showing
the overall “importance” of different features.
Since this is a global explanation, participants would see the same explanation on all tasks.
In the Feature contribution treatment, on each task, we show to participants the contribution
of each feature to the model’s prediction on that task.
This is a local explanation, so the explanation participants get varies from task to task.
In the Nearest neighbors treatment, given the profile in a task,
we show to participants the most similar two profiles in the training dataset that the
model either makes the same or different prediction as the current profile.
Finally, in the Counterfactuals treatment,
we explain the model by either showing the smallest change that is needed on a feature
to flip the model’s prediction,
or we tell participants that changing a feature’s value will not affect the model’s prediction.
We conduct our study by posting tasks on Amazon Mechanical Turk and recruiting MTurk workers
as our participants.
Upon arrival, participants are randomly assigned to one of the 5 experimental treatments.
They first complete a survey on their background.
Then, they go through a tutorial to get familiar with the task interface.
After completing the tutorial, the participant then move on to
make predictions on a set of 32 tasks.
On half of these tasks, The AI model has a high confidence (of 0.7 or higher),
and on the other half, AI’s confidence is low.
at the end, participants complete an exit survey,
which includes some multiple-choice questions testing
their objective understanding of the model,
and some questions collecting their self-reported understanding.
In the end, we are able to recruit 782 participants to complete the recidivism prediction tasks,
and 561 participants to complete the forest cover prediction tasks.
Now let me highlight some of our results to you.
I’ll focus on only the results about trust calibration.
You can find more results regarding understanding and uncertainty awareness in our paper.
We quantify people’s trust calibration in the AI using three measures,
appropriate trust, which is the frequency for a participant to
either take a correct AI prediction
or override an incorrect AI prediction;
undertrust, which is the frequency to override a correct AI prediction,
And overtrust, which is the frequency to take an incorrect AI prediction.
Ideally, We hope to see that AI explanations help people to
increase their appropriate trust in the AI
while decreasing their undertrust and overtrust, or at least not increasing them.
Based on the data we get, we find that on recidivism prediction tasks,
compared to not providing any explanations,
providing feature importance and feature contribution explanations do seem to help increase people’s
appropriate trust and decrease people’s undertrust.
And when we dig deeper into the data, we find
these two explanations are particularly helpful on tasks that AI has high confidence,
but the feature importance explanation may also have the risk of increasing people’s
overtrust on AI’s low confidence predictions.
In contrast, the nearest neighbor and counterfactual explanations do not have any obvious effect
on influencing people’s trust calibration.
Interestingly, when we conduct the same analysis on the forest cover prediction tasks,
we do not see any explanations have a conclusive effect.
To summarize, this table concludes the main findings of our study.
Notice that the effectiveness of AI explanations are largely different
on different types of tasks,
and very strikingly, on tasks that people have limited domain expertise in,
almost none of the desiderata is reliably satisfied.
In light of this, we recommend that in future evaluations of AI explanations,
researchers should communicate the results along with
sufficient contextual information about the tasks,
so that people can better understand the generalizability of the results.
Finally, we also notice one explanation can score high on some desideratum
but not the others.
This implies that instead of looking for a one-size-fits-all solution,
we’d better to understand the strength and weakness of different AI explanations,
and learn to select the right type of explanation for the right purpose.
So That’s all about our study.
Thank you for listening and looking forward to seeing you in the Q&A session!