- So my paper is a little bit different
than the ones we've heard so far.
This paper is a design paper, it's kinda conceptual.
Meant to, sort of,
provoke our thoughts on XAI design,
kinda generative, and think about
where we might take this budding area of research next.
Alright so, as AI systems become more commonplace everyday,
the need for people to be able to understand and interpret
these systems becomes paramount
to their successful integration into real world contexts.
The issue of explainablilty has become the locus
of a rapidly growing technical field
that we frequently call XAI, for explainable AI.
Which many of the papers in this session engage with.
A considerable amount of research has advanced
round our technical state of the art here.
But there's a key challenge that remains in bridging
the technical advances and the unique requirements
that might emerge and complex settings of end use.
To help tackle the gap between technical advances and use,
this paper turns to scenario based design
and a design method that's well known on the HCI.
That anticipates and leverages scenarios
of possible use early on in system development.
So to demonstrate how this method can help us
think through on explainability requirements,
I'll walk through a case study
in the field of healthy aging.
So in this paper I introduce the concept
of explainability scenarios as resources
and interpretability, explainability to design.
Explainability scenarios focus on what people need
to understand about AI systems in order to act on
and in concert with their outputs.
Instead of asking what might an AI system
be capable of explaining,
a technology or a solution first orientation,
a scenario perspective asks what types of explanations
might users need in the course of using our system?
The distinction here is subtle,
though can render profound design implications,
as it reorients the design focus on possible practices
and even problems of use.
Which then become resources in brainstorming
and requirements analysis early on in the stage.
So this is a roadmap of what I'll cover today.
I'm in the introduction now.
I'll spend a little bit of time introducing you
to scenario based design.
Then I will talk about the case study of healthy aging
and go through one of the scenarios
that's featured in the paper.
Scenario based design is a well known method in fields
like software engineering and human computer interaction.
Rosson and Carroll define scenarios
as a narrative descriptions of envisioned usage episodes.
Scenarios provide a design space,
that is at once concrete but flexible,
provides an interment,
though imagined engagement with our users.
A key feature of scenarios is that they take
the more traditional persona based design approach.
Which focuses on who will be using our system.
And combines that with a focus on narrative,
which is a process focus, that captures the organizational
or operational focus of how those users
might use this system, in what scenario,
what setting and for what purpose.
This helps us remember that our users don't engage
our systems for their own sake or in isolation,
but instead that they use them in order to get things done
in the context of their everyday lives.
My case study focuses on domain of aging,
which is a very active area of research right now.
Changing demographics worldwide have sparked a number
of technological approaches to address
the challenges of an aging populous.
One frequent application of AI systems in this domain,
to promote healthy aging and aging in place,
which is the ability for older adults to live independently.
One proposed way to do this is to help monitor seniors
everyday activities or ADL's, activities of daily living.
So I don't have much time to go kinda
in the very specific details of this,
but the high level proposition here is that,
if we monitor people's everyday activities
we can kind of understand what's working well for them.
What a normal day looks like for them
and whether living independently is still working right.
So over time a monitoring system might alert us
to variations or changes that might be alarming
and signal a need for heightened care.
So now we'll walk through one of the see able scenarios
that's included in the paper.
The main task here is predicting level of change,
level of care changes at a senior living facility.
So our scenario setting is a quarterly meeting
where the companies executive management team
reviews various forms of data
from the companies care facilities
and makes decisions about operations management
across those facilities.
The company offers three levels of care,
independent, assisted and nursing.
And they must align various organization
and managerial concerns
with the differing care needs of their clients.
For example, knowing that a client
will likely be transitioning from independent
to assisted living, soon will help
that they won't need that transition soon,
will help the team plan staffing
and facilities resources accordingly.
Dana, the companies Data Scientist,
begins presenting her report,
which features various standard business intelligence
or BI metrics, types of data analytics
the company has used for years.
Two years ago though, the company partnered
with a local university to deploy
an experimental AI backed activity monitoring system
that helps provide a better understanding of clients ADL's.
The first phase of the project was a pilot
that focused on data collection with a handful of clients
at one of the companies facilities.
Following their ADL and level of care changes
over the course of 18 months.
This pilot data served as the training set
for the systems predictive modeling capabilities.
The project recently moved into it's data phase
which involves using the trained model
to monitor current clients ADL's and help predict
when clients might be transitioning
to a higher level of care.
For new clients this monitoring is a mandatory service
but for existing clients it's opt in.
So this means that between these two different programs
the companies currently using the system
to monitor ADL's for 35% of their clients.
Morris, the University Professor who has been working
on the project discusses the models performance.
The model trained and pilot data is nearly 90% accurate
in predicting, based on clients ADL's in months one and two,
whether that client will stay at the same level of care
or move to a higher level of care in month three.
Cynthia, the Executive VP asks for clarification
of what that 90% means.
Does that mean 10 percents the margin of error?
Cynthia wants to understand the impact
of the 10% inaccuracy rate in relation to the cost
of staffing higher level of care units.
If we rely on the models predictions
does that mean we will be over staffing
or under staffing by 10%?
Morris explains that the model is trained on the pilot data
and based on that data is able to accurately predict level
of care changes in 90% of cases.
I'm not telling you what to do with this data
or what impact it will have on your business, he says,
I'm just telling you it performs with 90% accuracy
in predicting changes.
Vineeth, a Nurse Practitioner
and the companies Director of Clinical Care,
raises a point on the limitations of the data.
The pilot only included a handful of clients,
even now we've only got 35% of units covered,
he reminds the group.
Noting the limited data sources can create issues of bias.
He stresses the need to consider
the experimental systems predictions
with their established ways of monitoring ADL's,
which is typically through self and clinical reports.
Tanner the head of HR jokingly quips,
I don't really care too much about all this fancy data,
I just need to know how many nursing hours
we need to cover next quarter.
By describing a reasonably probable scene
of an AI systems future use,
explainability scenarios like the one I've just shared,
raise a number of issues
that can help us think through the design process.
One question we see here asks,
for whom should a system be explainable?
As we can see, a range of stakeholders
during this meeting, need to be able to understand
the systems limitations, in order to appropriately use
the predictions it provides
in their respective organizational roles.
This scenario also asks, what needs to be explained here?
A lot of explainability work focuses on explanation
of the predictive tasks.
This is important, but not the only explanations
that people need to be able to start using AI systems
in real world settings.
People need to understand
how experimental high tech AI approaches,
like the one we described in this scenario,
will be integrated with existing work practices
and how partial and potentially biased training sets
impact the validity of the outputs.
In this paper I introduce explainability scenarios
which provide a grounded and intimate depiction
of a reasonably probable scene of an AI systems future use.
Such scenarios can then become design resources
that open up spaces for design work,
that can help us bridge the gaps
between XAI and HCI
in settings of use.
This ensures that when we talk about
designing explainable systems,
we're focusing on what people need to understand
about AI in order to act on it in concert
with the systems that we are developing.
Thank you.
(clapping)
- Thank you very much.
Well times up but there's room for a quick question
if anybody has?
And if not I'm afraid we'll take it offline later.
- Yes, thank you very much.
- We'll move to our next speaker.