Hi, I'm Monica, and today I'm presenting "Assessing the impact of Automated Suggestions
on Decision Making," joint work with colleagues in MIT CSAIL. Automated decision aid can accelerate
often tedious tasks across domains, but if users misplace trust in or cede agency to the
decision aid, things can go awry. And due to the ever-growing presence of automated decision aid,
we build on past work to ask "How does domain expertise mediate the influence of decision aid?"
In particular, we're interested in asking this question in a task with a complex decision space,
where we can use objective measures of trust and agency instead of relying on user perception. And
then finally we want to do so over an extended period of use so we can factor in user's fatigue.
We investigate via a user study over a clinical text annotation task that's a perfect sandbox
for these questions. In the task, the user is presented with a sentence from clinical text
and asked first to identify instances of clinical jargon in that text and then disambiguate
those instances and map them to a term in a structured ontology with over 400,000 concepts.
To conduct the study, we developed an annotation platform with built-in decision aid.
The decision aid included label recommendations which appeared when users highlighted a term,
as well as pre-filled suggestions which contained a box with both a span and a label and can be
accepted in a single click. We then analyzed how does this decision aid influence our domain
experts and impact their agency. In our study, we had 18 clinicians from nine institutions,
and novelties of our study included the fact that we jointly studied agency (what to label)
and trust (how to label) using objective measures based off of their interactions
with the platform. We had a large label space, and we had about eight hours of annotation per user.
We conducted the study in two stages, which I now detail. In the first stage, we look at
label recommendations. We have three user modes with full recommendations, no recommendations,
and artificially weakened recommendations. We analyze accuracy, speed, and search behavior,
particularly where recommendations are inaccurate. So first we ask, how does user behavior shift?
We find that users with full recommendations created more annotations than those without any,
and they did so in much less time, as you can see in the graph on the right.
Next, we ask, "Do users search when needed or do they misplace trust when recommendations do not
surface the correct answer?" And on those examples where the underlying recommendation algorithm
truly fails, they do search most of the time, 83%, and they perform equivalently. You can see
in the plot on the right, there's no significant differences in performance across the three modes.
However, they perform much worse when they expect the correct answer to be surfaced and it does not.
They only search 73% of the time; you can see the drop in performance on the right,
and this indicates it's important for users to develop proper intuition of model behavior.
In the second stage, we look at pre-filled suggestions. We have four modes with a control
as well as suggestions with various settings. In addition to the prior metrics, we also look at
additional annotation behavior: Do they annotate those spans that were not indicated by pre-filled
suggestions? We first see if users react appropriately to the suggestions, and they do most
of the time. They accept almost all of the correct labels and spans. When the label is incorrect,
they only accepted 17% of the time, and when a span is incorrect, they show slightly less agency
accepting at 33 percent of the time. However overall they tend to have higher performance
than users without pre-filled suggestions. Interestingly there was large user variability
in the accepting of incorrect labels and spans and this was not correlated with their prior
task performance. We also found that providing label confidence made no discernible difference.
Now if we look at these new annotations that they had to create and label by hand, we found
that they experienced a loss of agency compared to their prior performance as well as the control
group, and this was significant. Interestingly this loss of agency went unnoticed by users who
noted in their exit interviews that they thought they were more thorough than before. So our
takeaways are first, our domain experts properly modulate trust and mediate model errors if they
have the correct mental models. Second users lost agency without noticing, making it very important
to use objective measures for user behavior. And finally we believe that both user interfaces
and machine learning systems should adapt to account for this loss of agency. Thank you.