Hi, my name is Gagan Bansal.
I am a graduate student at the University of Washington.
Today I will talk about an important issue with the effectiveness of current methods
for explaining AI systems.
There are many reasons to explain predictions of AI models,
Including helping ML developers to debug and train better models or because of legal reasons
such as GDPR
But another crucial reason is to help create better human-AI teams.
For example, one common for of teamwork is AI-assisted decision making, where
the AI provides its predictions as recommendations to users who retain the agency to make the
final decisions.
While the model’s recommendations by themselves may be helpful,
one would hope that explaining and conveying the model’s reasoning to the user will offer
further improvements by increasing appropriate reliance.
Ie, by letting the user better decide when to trust the system’s recommendations.
But do current AI explanations improve AI-assisted decision making?
Many prior works have shown that explanations improves the performance of users at making
decisions, For tasks ranging from detecting deceptive
reviews, to predicting loan defaults, to patient care.
In each case prior works have shown that team where explanations were shown to the users
performs better than humans by themselves
But our survey of these works revealed another commonality--- all prior works observed improvements
from explanations only when the AI itself was much much more accurate than solo users
and in fact the human-AI team.
And if accuracy were the sole objective, in each of the cases performance could have been
further improved by simply getting rid of the people in the loop or by soothing the
users so that they become more compliant.
But either of any of such approaches to develop augmented intelligence seem deeply unsatisfying
or ethically fraught.
As a way forward, in our paper we propose that research on explainable AI should focus
on achieving complementary performance
i.e., a level of performance that is greater than what solo users or AI can achieve, as
indicated by the gray region on the horizontal axis.
Such gains are only possible if explanations improve appropriate reliance.
But can AI explanations lead to complementary performance?
Since all prior works did not observe complementary performance when there was a significant gap
between AI and user accuracies,
Using large scale studies, we investigated the case where the AI and humans have comparable
accuracy.
Despite extensive efforts we observed no improvements from showing current explanations methods
over simply displaying the model’s confidence.
And while explanations increased trust, they did not increase appropriate trust.
Our study uses a between subjects designs:
Three datasets: two binary classification datasets and another harder LSAT question
answering dataset.
Multiple explanation strategies and sources
The complete details of the experimental design can be found in our paper, but here is a
screenshot of our study interface for one of the classification.
On the right the interface communicates both the prediction and confidence our model.
And on the left the system explains its recommendations using inline-highlights
Here are results on one of the sentiment classification dataset but the results were similar on other
two domains
First, we observed that a baseline team where the AI simply shows its recommendation and
confidence achieved complementary performance, i.e. it performed better than humans or AI
acting alone
Second, despite extensive efforts none of many explanation conditions we tried improved
team performance over simply showing confidence.
Finally, when the AI was incorrect its predictions lowered the team’s performance, indicating
that showing explanations did not help increase appropriate reliance on AI.
Please checkout our full paper for quantitative results on the other two datasets
And qualitative analysis such as how people collaborate with AI.
For example where the use the AI explanation as a prior guide or as post check.
And the difference in distribution of errors and expertise of human and AI models.
Our study has several limitations
We are not claiming that explanations will never lead to complementary performance, but
rather that existing AI explanations may be insufficient and that none of the previous
works, including ours, has observed significant improvements into the complementary performance
zone.
We only studied a limited number of domains with lay users, so its possible that there
exists other domains for which explanations lead to complementary performance.
In fact, very recently
we put out another study showing that open-domain question answering is one domain where explanations
significantly provide value to end users!
Finally, we only focused on accuracy of decision and did not investigate whether explanations
improve the speed of decision making.
In summary, we invite contributions from the community to find domains and settings where
explanations lead to complementary performance
And develop new explanations algorithms, representations, and interfaces that promote appropriate reliance
on AI systems.
For instance recent work from Zana Buccinca at Harvard showed that using cognitive forcing
functions to make users pay more attention can decrease over reliance but at the cost
of user satisfaction.
So future work should also develop explanations for optimizing multiple objects, such as speed
and user fatigue or satisfaction.