Hi, this video is to present our CHI 2020 paper titled questioning the AI
informing design practices for explainable AI user experiences. My name
is Vera Liao. I'm a researcher working at IBM Research. This is joint work with my
colleagues Dan Gruen and Sarah Miller
Artificial Intelligence technologies especially those using machine learning
are increasingly used everywhere including high-stakes tasks such as
supporting decision making in healthcare, finance and social justice. Accompanying
this trend is a quest for explainable AI. That is, making AI more transparent and
understandable by people so people can feel more confident or better exercise
cautions with AI's decisions, and ultimately better work and live together with AI
But Explainable AI is hard. It involves a lot of technical work. There are many
different machine learning algorithms and not all a straightforward explainable. It
is also commonly recognized that popular, high-performing algorithms such as those
using neural networks are especially complex, opaque and almost impossible to
explain the inner working directly. So often a separate set of techniques or
algorithm have to be used to generate explanations that can be consumed by people.
Developing these techniques is the main focus of the research field
explainable AI, or XAI, which started as a sub-field of AI research. These are just
some recent review papers on the topic. There are at the scale of hundreds if not
more papers published on the topic. Another notable trend is that very
recently a growing number of toolkits are making these XA techniques
accessible for practitioners. So we are seeing explanations becoming an
instrumental part, and likely increasing so, in real-world AI systems.
Explainable AI is also hard because explanation has to be meaningful to the
people demanding it. Simply exposing a model's inner working does not
guarantee the information is understandable or useful to the users.
There are also many different type of user who may have different reasons and
requirements to seek explanations. So creating explainable AI systems should
follow a user-centered process to bridge technical capabilities and the fulfillment
of user needs. In product practices this kind of work often falls on design and
user experience practitioners, whose job involves identifying user needs
communicating with data scientists, developers and stakeholders and creating
design solutions based on demands and constraints on both sides. Therefore in
this work we look into the design practices around the current and
upcoming explainable AI technologies. Our goal is to identify opportunities to
better support such work hence the creation of user centered explainable AI.
By talking to design practitioners across many AI products we also hope to
understand real-world user needs for explainable AI and the design space,
which could help inform future directions for the research field of XAI.
But very quickly we ran into the challenge of how to talk about a
technical space that people are not quite in there yet, since we just began
introducing what's produced in the academic world to practitioners. Our informants
may not be familiar with different XAI algorithms or have an established
framework to talk about user needs for explainability. So we decided to create
a study probe--a list of algorithm informed XAI questions. The probe is
based on the following assumptions: First, user needs for explainability can be
represented by prototypical questions such as Why, What If, How. This is both
based on prior work on explaining computing systems,
and fundamentally, the definition that explanation is an answer to a question.
We also assume that a prototypical questions can be answered by one or
multiple XAI methods, which can be implemented by one or multiple XAI
algorithms. The key idea here is that even though there are hundreds of XAI
algorithms many of them produce the same kind of output or method to explain, so
there is a much more finite list of XAI methods. For example, a why question
asking about a particular AI decision can be explained by a popular method
called local feature importance, which highlights the most important features
of the instance that contributed to the AI's decision. In this case, pixels that
the image classifier used to determine this is a wolf by mistake. Meanwhile
there are many popular algorithms to implement this local feature importance
method. Based on this reasoning we started with a literature review of
technical XAI work and arrived at a taxonomy of explanation methods. Then
we map them to a total of six categories of prototypical questions such as Why
What If, How. We also decided to take a broad view of explainable AI and added
three more descriptive categories of questions based on prior work. These are
questions regarding AI's input, output and performance. For each category of the
prototypical question we created a question card. It includes the leading
question and comment examples. Then we brought these question cards to the
informants. In total we interviewed twenty design
practitioners working on 16 different AI products. In each interview we asked
the informant to first walk through the AI system, then discuss what are the common
questions users might ask to understand the AI.
After that we walk through each question card and discuss whether they apply and
if there are any missing questions. We closed the interviews by discussing what
are the challenges design practitioners and their teams
face to create explainable AI systems. Through step two and step three we
gathered a list of user questions. We performed content analysis on these
questions, details in a paper. The outcome, and one of the main contributions of the
paper is this XAI Question Bank. It includes both
algorithm informed questions we started with and new questions identified from the
interviews, marked by asterisks. This Question Bank represents a quote unquote
designer source space of user needs for AI explainability. It also helped grounding
our discussions to understand these questions in real-world contexts. We'll
come back to the Question Back later and talk more about its functions. But first
I want to discuss the main themes identified in the interview analysis. They
point to the challenges for design practitioners to create explainable AI
user experience. The first challenge designers face is the variability of
users' explainability needs. First of all there are diverse motivations for people to
demand explanations. We summarized five main categories with more details in the
paper. For example some users demand explanation to gain further insights for
their decisions. As the example given by this informant, it was not enough for the
AI to give a prediction to supply chain management workers that a delivery might
be delayed. More importantly users want to understand the reason for the
potential delay, whether it is because of the weather, or somewhere they could
make a quick call to. So they can take actions based on the explanations. In
this context user may ask a why-question, why the delivery is predicted to be late,
and how to reduce the delay. A different motivation for demanding explanation is to
appropriately evaluate the capability of the AI. For example a doctor is introduced
to a new AI system for medical imaging. The doctor may want to understand both
how well the system performs and how the system makes its judgment.
These are two out of the five types of motivation we identified. We also
identified other factors mentioned by the informants that could vary users' explainability
needs including user group, usage points, algorithm and data type, and the
decision contexts. A second challenge that we saw
informants struggle with is the gaps between explanation as a common element
in human communication and what current AI technology can produce. They mentioned some
common criteria of human expansions, thus what users might expect to see, such as
being selective, contrastive, interactive and tailored for the recipients. However
it is often not possible to generate explanations that satisfy these criteria
both constrained by current technical capabilities and the inherent discrenpency
between how human and AI makes decisions. Lastly, informants discussed some
process-oriented challenges for AI product teams to invest on or
prioritize explainability. One challenge is for designers to
navigate the technical capabilities to explain AI, partly due to the skill
gaps, and also it is challenging to keep track of the quickly growing landscape
of XAI. There are also barriers for designers to effectively communicate
with data scientists and other stakeholders around expandability, even
though explainable AI solutions should be jointly sought from a user perspective
and modeling perspective. This communication cost combined with
prototyping cost to explore design solutions often discourage AI product
teams to prioritizing explainability, which could be at odds with other product goals.
Many believed that these problems can be mitigated by having structured guidance
that helps the team efficiently navigate to desired explainability solutions. We
summarized the desired support in two areas:
One is to support the needs specification work. What are the explainability needs
specific to a product, a user group, even a particular interaction. Second is after
needs specification, guidelines or recommendations to address the needs,
ideally with example artifacts such as model output or UI patterns to save the
prototyping cost. You may consider the two areas as answering what to explain,
and how to explain, respectively. Now coming back to the XAI Question Bank. We
suggest that this list of prototypical user questions can be leveraged for needs
specification work, whether as a checklist or in user research to
enumerate on and prioritize different types of user needs for AI explainability,
for example through a car sorting exercise. In our paper the Question Bank
and informants' discussions around it also served two other functions, which I will not
go into detail but point you to the paper. One is to ground our understanding
on why these use questions are asked in real world contexts. Based on that we
derived a set of guidelines to address each categories of user needs for
expandability. Second, understanding where these
questions emerge, especially the newly identified questions that were not in the
algorithm informed question list, we identify gaps in current technical work
of XAI and opportunities for future work. Back to tackling the
process-oriented challenges, at end of the paper, we propose a question driven
design process the centers the design of the explainable AI around user
questions. We suggest, from user research, to identify key user questions
for understanding AI, and also the requirements to answer these questions
then work with data scientists to map these questions to candidate XAI
techniques that can answer these questions. Starting from what these
techniques produce, iteratively design and evaluate the explainable AI user
experience against the user requirements. Besides
providing an XAI Question Bank to support identifying key user questions,
the kind of mapping we did in the beginning, from user questions to XAI
methods and algorithms can be used to suggest candidate solutions to data
scientists. Essentially this process allows user questions to become the
boundary objects to support joint discussions between designers and data
scientists. As follow-up work for this paper, we're currently practicing and
validating this question driven design process with AI teams, and we hope to
share the results in the near future. I want to thank you for viewing this
presentation. To summarize our 2020 paper by interviewing 20 design
practitioners, set out to understand real world user needs for nine categories of
AI explainbility, we produce insights to address them and inform opportunities
for future XAI work. We also identify key challenges faced by design practitioners
to create explainable AI user experiences. To help tackling these
challenges we suggest an XAI question Bank and a question driven design process
I would like to invite you to read our paper and I'll be happy to answer your
question and hear any feedback you have. Thank you for your time and attention