In this talk, we describe AI explainability practices that arise is real-world AI-based
products, specifically within nlp applications.
I’m Shipi Dhanorkar and I’m excited to be presenting our work on behalf of all of
my co-authors Christine Wolf, Kun Qian, Anbang Xu, Lucian
Popa and Yunyao Li
Our talk is titled: Who needs to know what when?
Broadening the Explainable AI (XAI) Design Space by Looking at Explanations Across the
AI Lifecycle
Our daily lives are filled by encounters with AI-infused products.
As contemporary AI products’ become sophisticated, so do their complexities.
One problem that confronts us is the comprehension of the inner workings of the model, and an
understanding of why a model did what it did.
The growing realization of the substantial roles AI models play in our lives brings explainability
of AI models to center stage.
Our main goal was to understand how various actors in industrial AI projects engage with
explainability, the material qualities explanations take, and the tensions that arise as situated
actions unfold.
We build our study on a range of research that describe algorithmic developments for
building explainable models and the human centered perspective on Explainable
AI that explores these advances in relation to their human and societal implications.
Our empirical study focused on AI projects related to text data, a key application area
within industrial AI projects.
While language has an underlying grammatical structure; in use, it is rich, expressive,
and messy.
These complexities make computational modeling of language challenging.
Our contributions are three-fold:
First, we build an in-depth empirical understanding of explainability practice in real- world
projects.
This generative research offers a rich description of the collaborative sensemaking surrounding
AI models.
Second, we spotlight the AI lifecycle as a lens to characterize explanations as situated;
describing the actors that interface with the AI model at the different locations which
we call AI touchpoints.
Third, in taking this complimentary view, we expand the ambit of explanations moving
beyond binary ways of thinking whether an AI model “is explainable” or “not”.
To capture a variety of experiences and perspectives across text analytics projects, we conducted
an interview study with 30 participants
Our informants represent a range of roles across a variety of industrial AI projects,
including, technical AI research and development roles like data scientists and AI researchers,
those who are in strategic development as well
as those focused on user experience design and infoviz research
They also represent a range of AI expertise, some who are relative newcomers to the world
of AI on the one hand and several years of experience or grad-level expertise in AI on the other.
One product manager talked of the “brutal, steep learning curve” to get himself acquainted with AI
Participants were working on a variety of domains for example, document analysis of
legal contracts, identifying key opinion leaders in the medical domain, text generation for
application in HR and many more
Our semi-structured interview approach allowed informants to contribute to an interactive
exploration of our inquiry areas namely, who touches the model, what are their informational
needs and when do explanations get warranted?
During the interviews, many informants also showed us artifacts that they developed to
explain models like slide decks, scientific papers, dashboards, and infoviz applications.
We iteratively analyzed the interview transcripts and notes beginning with open coding to develop
categories and themes
Early in the AI pipeline, during the model conceptualization and model building phase,
model trainers use explanations to peek inside the model, get a sense of why a model is predicting
what it is and what linguistic features it might have learned.
Similarly, other details like hyper-parameter tuning could be informed by explanations and
also be used to debug or brainstorm different algorithmic designs.
In the model validation stage, an important part of discussions with clients revolves
around data, and this was a dominant theme throughout our informants’ accounts.
As novel sources of data are fed into the model for an existing business process, newer
explanatory articulation work needs to get done
During the initial AI model development phase, clients and their legal teams provide high-level
privacy requirements specific to their domain.
These are fed into data pre-processing steps so as to develop appropriate masks and filters
for personal data.
During proof-of-concept stage, discussion about this data preparation and mapping between
privacy requirements to the data masks and filters comes to the forefront.
Here, clients raise questions pertinent to regulatory issues with respect to the AI model.
Explanations are geared to assure appropriate legal compliance.
Drawing from a client conversation, one product manager mentioned, quote “How do we know
the model is safe to use?
... Users will ask questions about regulatory or compliance-related factors: Does the model
identify this particular part of the GDPR law?”
When the model is in-production, an HCI researcher aptly characterized expectation mismatch
that occurs between the models prediction and users’ anticipated prediction that elevates
the need for explanations.
Another technical strategist indicated that explanations help in distinguishing outputs
manifested through model mistakes versus decision disagreements.
In discussions with our informants, we also saw explanations as being closely intertwined
with business actionability.
To meaningfully integrate with the business workflows, it is crucial for model explanations
to accommodate the constraints of the broader business.
In using the AI lifecycle as a lens, we were able to explore the intersectionality of the
social and technical elements of rendering AI models explainable
Our data surfaced a variety of explanation audiences and motivations including the need
demystify the magic in AI models.
This explanatory articulation work is also laden with challenges:
The first challenge we observed was about balancing the different levels of AI expertise
both within internal teams and external business stakeholders
Next, there is an inherent simplicity versus complexity tradeoff in designing explanations
We also encountered a constant dilemma that our informants face on the sidelines- the
concern about explanations revealing identities, or too much of the proprietary model
In closing, w’d like to leave you with a quote from one of our informants
Explainability is like an onion, there's many layers.
The algorithms and the data are one layer of that…..
You want to keep the SMEs or Subject Matter Experts out of the ditches around precision or recall, F1 scores.
But you want them to have a higher level understanding of the protocols, the methodology that went
into creating this model.
We have begun to chart out some emergent considerations for a broadened XAI design space such as:
enriching dialectical exchanges between stakeholders by using rich artifacts like Model cards,
seeking alignment with a model’s location in the AI lifecycle,
catering to wide variations in stakeholders’ AI expertise;
minimizing expectation violations during deployment and
balancing technical details with high-level understandings of the AI model.
For questions, discussions or engaging further with our research, you can contact us on the
email shown in this slide
and
Thank you very much for listening!