Hello my name is Sunnie. I'm a PhD student at Princeton University, and I'm excited to share
our work "Help Me Help the AI: Understanding How Explainability Can Support Human-AI Interaction."
This work was done with Elizabeth Anne Watkins,
Olga Russakovsky, Ruth Fong, and Andrés Monroy-Hernández.
With the growing adoption of AI systems and growing demand
for being able to understand how these systems work and produce their outputs,
the field of Explainable AI or XAI has rapidly grown over the past years.
Yet, little is understood about real AI end-users' XAI needs, uses, and perceptions.
This is because much of prior work has focused on developing new XAI methods without humans,
and studying them with automatic evaluations or short experiments with MTurkers who are
placed in a hypothetical AI use setting. To address this gap,
we conducted an in-depth qualitative study with real end-users of an AI application called Merlin.
It's a bird identification app, and we chose it because it is a widely used app that would
allow us to connect with a diverse set of active end-users. More concretely,
we conducted 20 one-hour interviews with end-users of varying levels of domain and AI background.
Our study aimed to answer these three research question. First,
what are end-users' XAI needs in real-world AI applications? Second, how do end-users intend
to use XAI explanations, specifically explanations of individual AI outputs?
And third, how are existing XAI approaches perceived by end-users?
In this short talk, I will summarize the key finding for each of these questions.
First, regarding XAI needs, we found that system details such as information about
the AI's performance, training data, and inner workings were only wanted by some.
High-AI background participants, such as AI researchers and developers,
had high needs for these details. They were super curious about the app's AI,
and to get this information were willing to go to the extent of emailing the app developers
and playing with the relevant data and model themselves. On the other hand, low-AI background
participants said they were curious but wouldn't go out of their way to learn more about the AI.
Some even preferred to keep it as a black box, saying they don't want to ruin the mystique.
An exception were participants with a notably high interest in birds. They were very curious about
the AI, in particular how it tackles challenging bird identification problems, such as mockingbirds
or "little brown birds" that even experienced human birders have difficulty in distinguishing.
While participants' needs for AI system details differed based on their AI background and domain
interest, all participants expressed the need for practically useful information,
such as AI's capabilities and limitations, confidence, and detailed outputs.
And participants wanted this information to improve their collaboration with the AI,
to work together with it to achieve the best possible outcomes.
Next, to gather concrete and detailed data on participants' XAI uses and perceptions,
we mocked up explanations that could be potentially embedded into the app.
As with most AI applications, Merlin does not provide explanations of its outputs.
Hence, we designed mock-up explanations for three real outputs that we obtained by using the app.
And for each of these outputs, we created mock-up explanations from four distinct XAI approaches.
Here, I'm showing two as examples: heatmap and concept-based explanations.
When we showed participants these mock-up explanations, they were excited to use them
for various purposes. To understand the AI's outputs, calibrate their trust in the AI,
learn from the AI to perform the task better on their own, change their own
behavior to help the AI perform better, and give feedback to developers to improve the AI.
These demonstrate the broad range of user goals that should be considered in XAI development.
And we particularly want to highlight four and five, which are uses that haven't been discussed
much in the XAI literature. They suggest that participants viewed the AI as a collaborator,
and wanted explanations to help them help the AI perform better.
We view these as intriguing repurposing of explanations which are typically
developed to help people understand and calibrate their trust in AI systems.
Finally, regarding XAI perceptions, we found that participants' perceptions
of existing approaches varied based on their AI background.
For heatmap-based explanations, high-AI background participants found them intuitive and helpful,
while low-AI background participants didn't and didn't want them as an explanation.
Similarly for concept-based explanations, high-AI background participants overall like the details
and wanted even more information, whereas low-AI background participants found them overwhelming.
In summary, our findings reveal a creator-consumer gap in XAI research and development. Regarding XAI
needs, creators of XAI, such as AI researchers and developers, are known to want AI system details
and an understanding of AI's inner workings. However, when we talk to end-users in our study,
we found that only those with high-AI background wanted such details. A more frequently expressed
need was for practically useful information that can enhance their collaboration with the AI.
Regarding XAI uses, typically XAI methods are created to help people
understand and calibrate their trust in the AI.
However, again when we asked end-users about their intended uses of explanations,
we found a much broader range of user goals that should be considered in future XAI development.
Regarding XAI perceptions, we found that while existing XAI approaches are deemed as satisfactory
by creators and end-users with high-AI background, end-users with low-AI background perceive the
same XAI approaches very differently and were dissatisfied with some of the popular approaches.
Finally, while participants were overall excited to see explanations, they expressed concerns
as well, about explanations potentially being not faithful to how the AI works,
being difficult to digest, and engendering over-trust in the AI. To mitigate these concerns
and to design XAI that address end-users' needs, we advocate for the following.
First, explanations should be designed with end-users.
Second, explanations should answer "why" questions not just "what" questions.
Third, explanations should use multiple forms and modalities if warranted.
And fourth, explanations should be evaluated rigorously to mitigate potential negative effects.
Thank you for listening, and please check out our paper for more details!