Cross-functional teams often lack tangible boundary objects for collectively defining and developing AI.
We report on how AI onboarding ended up serving as a boundary object between
team members while developing an AI for medical diagnosis.
AI Onboarding is introductory training material for users to see prior to their use of an AI.
For example, doctors can first be introduced to an AI via onboarding,
Prior to making medical decisions with the help of AI.
Here is an example of onboarding materials for an AI for prostate cancer diagnosis.
From prior research, we found these categories of information particularly
useful for doctors to learn about an AI during onboarding:
Such as its source of groundtruth, its strengths and limitations, the ways it
might act “weirdly” compared to human ways of thinking, and an actionable tip
so that users know when to rely more on the AI, and when to rely more on their own expertise.
Here are a few examples of how the process of creating onboarding changed
our team’s understanding of user requirements.
First, we found that giving people metrics about the AI was not enough, because metrics alone aren’t actionable.
Users wanted actionable strategy for knowing what to do in cases where they disagree with the AI’s predictions.
As a result, the team needed to translate typical performance metrics like accuracy, into metrics that could
be more actionable from the end-user’s frame of reference.
Through doing this, the team developed a shared understanding of how end-users assess models during diagnosis.
We also found that users wanted to know which edge cases the AI had been designed to handle.
For example, in their own medical practice, doctors find certain types
of medical subpatterns particularly difficult to diagnosis (such as perineural invasion).
They wanted to know if the AI similarly struggled with those edge cases, or if
the AI had specifically been designed to handle them already.
However, this information was not easy to uncover, since the data did not always contain edge case labels.
Moreover, re-labeling the data could be a taxing process.
Ultimately, the team realized that labeling subpatterns and edge-cases could actually serve a dual purpose:
it could help improve the model, AND it could help provide transparency on edge cases during onboarding.
This reflection raised awareness about how labeling subpatterns could be mutually beneficial moving forward.
One aspect that’s valuable to provide upfront during onboarding is the ways
that AI can seem weird or bewildering, compared to human ways of thinking.
Explaining these idiosyncrasies can help set user expectations early on in their exposure to the AI.
However, knowledge about the AI’s idiosyncrasies was often distributed
across team members, with different people having noticed different quirks about the AI.
As a result, we aggregated this knowledge across team members, and explained those odd behaviors during onboarding.
This process of collecting that information fostered communication between team
members about AI vs. human ways of thinking, and led to a better group understanding of user expectations.
In a study we conducted with 17 pathologists, users felt that
onboarding helped them collaborate better with the AI -- most importantly,
they felt it helped them understand its strengths and weaknesses, which helped
them develop a strategy for working with it.
Users were given an actionable strategy during onboarding, to use during diagnosis with the AI.
Those who recalled this strategy also had higher accuracy overall, compared to those who didn’t.
In conclusion, future teams could consider using Onboarding as a boundary object for developing AI assistants.
To get started, cross-functional teams could jointly create onboarding materials by filling in these 4 topics together.
For example, teams could answer questions such as “how does the AI behave weirdly, relative to human ways of thinking?”
Or, “when users perform this task in their current work contexts, which edge cases do they care about?
Among those edge cases, which ones does the model perform well on, and which does it struggle on?”
Or, “If a user is deciding when to trust the model vs. themselves, what tip would you give them, in your own words?”
In sum, we found that this process of creating onboarding had a side effect of facilitating the cross-talk between
machine learning, HCI, and product management as they jointly iterate
between understanding user requirements and developing the machine learning model.
We hope future teams will consider using Onboarding for developing AI assistants.