Who of you knows how a bicycle works?
Chances are high that if I ask you to prove if by drawing one
including its functional elements it will results in something
like this, this,
or this.
We humans have a tendency to confuse our understanding of how to use something with understanding
how and why it works.
I our work with my collaborates, we found evidence that such a cognitive bias also occurs
when non-technical users are exposed to Explainable AI.
We explore how non-technical end users build their understanding, and if what they think
they understand matches what is explained.
The term XAI is about an AI system’s ability to explain or present in understandable terms
to a human. But to what kind of human?
In our work, we focused on the non-technical stakeholders of XAI,
namely end users in decision support situations that use ML predictions for their decisions,
but have no expertise in ML. Various explanation approaches for XAI have
been proposed.
Global explanations aim to provide holistic insights about the model behavior to understand
its typical behavior while local explanations aim to explain an
individual prediction of the ML model. In our work, we focus on local Shapley explanations.
Shapley explanations are widely used in practice.
They perfectly distribute the prediction outcome (for example a probability) among the features
that contributed to it.
However, the challenge with local explanations is that they leave it up to the user to construct
a global understanding of the model behavior. Especially Shapley explanation simplify much
of the probabilistic model complexity into easy looking contributions.
By this, they abstract for example from feature interactions and may leave users with a false
belief about their gained understanding. However, we think they are promising, because
their mathematical properties allow to approximate a model from multiple perspectives.
However, in their raw form they hide the intrinsic complexity of probabilistic ML models behind
simple values.
If unguided, we believe users might form an false or incomplete belief about the understanding they gain from Shapley explanations.
This illusion is a robust cognitive bias of overconfidence which has been reproduced in many context
We humans often form an inaccurate understanding of our interaction with complex systems and
often overrate the depth of our understanding how they work
The IOED is more pronounced for explanatory knowledge that involves complex causal patterns
between components (such as ML systems).
It applies less for for factual or procedural knowledge.
For example, we know when we don’t know the capital of a country.
Some researchers in XAI already speculated that an IOED may be at play when users are
confronted with XAI.
As such, we investigated if non-technical users of XAI systems based on Shapley Explanatins
are prone to an IOED.
We examined how robust the self-reported subjective understanding of users actually is.
We hypothesized that when exposed to Shapley explanation, their perception of understanding
should increase. Further, we hypothesized that if an IOED
applies, their perception of understanding should decrease when we examine their actual understanding.
Further, we were interested in what users actually do and their reasoning when constructing
their mental model of global model behavior from local explanations.
We developed a prototype that we used as an apparatus
SHAPTable is an accessible spreadsheet-like explanation interfaces that embeds Shapley
explanations We centered our prototype in a loan application
scenario and used a black box ML model to predict each loan request’s probability
of defaulting.
The features that contributed to this probability were shaded based on their Shapley contributions.
Further, our explanation interface followed the XAI guidelines for contrastive, selective
and interactive explanations.
For our study we used a mixed methods approach to account for the dual modes or reasoning.
Our study procedure was inspired by the original IOED studies.
In consisted of a phase of an exploration phase, in which users build their understanding
without and with SHAPTable. In a second phase of reflection, users needed
to write down what they actually understood following guiding questions.
And in a final phase, we examined their objective understanding through 8 forward prediction
tasks in which they needed to apply their understanding and simulate the AI prediction
behavior.
In the end, we confronted them with their results.
We conducted the study in two different settings.
In the moderated online setting, the presence of a moderator motivated participants to apply
high-effort rational thinking throughout the procedure
While in the unmoderated crowd-sourcing setting, participants were guided more by low-effort
heuristic thinking. We used their self-ratings of subjective understanding
as a measure.
We took them at five points throughout the procedure on a calibrated scale of understanding
Further, we used their test results as a measure for their objective understanding
The results show that participants in both settings significantly decreased their subjective
understanding throughout the procedure.
Indicating an IOED.
Regarding our first hypothesis: The provided Shapley explanations in SHAPTable were considered
as useful and significantly increase their subjective understanding.
Regarding our second hypothesis we have seen a significant decreased after participants
have been examined.
In the moderated rational thinking setting the reflection tasks already had some calibrating
effect as the ratings dropped significantly.
For the unmoderated setting, participants did not change their self-perception until
confronted with their actual objective understanding at the very last stage.
Regarding their objective understanding, we observed that participants in the moderated
setting were less confident but gained a more accurate understanding.
The objective understanding was better than a random guess in both settings, indicating
that participants gained a general sense of the ML model behavior.
Interestingly, participants in the unmoderated setting spend less time on building their
understanding, less time on reflecting it, were more confident in their answers, expected
more correct answers of themselves, but objectively score significantly worse than their moderated
counterparts.
We measured their performance in terms of correct answers as well as their distance
from the correct answer.
Our work highlights the need for XAI to account for cognitive biases when exposing explanations
to non-technical users.
In the unmoderated setting participants did not realize the incompleteness of their understanding
until they saw their objective performance.
We observed that they were less often aware of irregularities in contributions and that
features interactions were present in the ML model
Further, in our paper, we report insights from think aloud protocols and observations
about the reasoning and interactions participants used when interpreting Shapley explanations.
These may inform XAI designers on how to design more human-centric explanations. Overall, the large majority of participants
found that the procedure helped them “to better assess their understanding of the prediction
behavior”.
As such, we believe that XAI must not only provide understandable explanations, but must
also calibrate and guide the process of understanding. With that, we invite you to explore our paper
and are looking forward to the discussion.
Thank you for your attention and the opportunity to share our work.