Hi everyone, my name is Amy Rechkemmer, and I’ll be discussing how user trust in machine
learning models is impacted when multiple indicators of model performance are presented.
This is work that I did with Ming Yin at Purdue University.
Machine learning has become ubiquitous.
Today with the use of machine learning, we are solving problems from critical societal
challenges to decision-making in everyday life.
As the population of those regularly exposed to machine learning expands, the question
of which factors impact end-users trust in these models has been a topic of interest.
Prior literature has found that the accuracy of a model, both in terms of a stated accuracy
on some held-out data and its observed accuracy in practice, along with the confidence that
a model associates with its predictions are factors impacting users’ trust in the model’s
decisions.
What remains under-explored, however, is what impact multiple performance indicators together
would have on people’s trust in these models.
To fill in this gap, we designed and conducted a randomized behavioral experiment with subjects
recruited from Amazon Mechanical Turk to determine how users’ trust in machine learning is
impacted by the model’s stated accuracy and its confidence before the model’s accuracy
is observed in practice, and how all three performance indicators impact user’s trust
after the model’s accuracy is observed in practice.
In this experiment, we asked subjects to complete a sequence of tasks on predicting the outcome
of speed dating events with the help of a pre-trained machine learning model.
Before working on any tasks, the subjects were told the machine learning model’s stated
accuracy on a large set of held-out data.
Next, the subjects completed 20 prediction tasks in Phase 1.
For each task, the subject was presented with a profile of a participant and their date
in a speed dating event.
We asked our subjects to first review the profile and make their own prediction of whether
or not the participant in the profile would want to see the date again.
Next, the subject was shown a machine learning model’s prediction for the task, along with
a confidence score between 0 and 1 representing how confidently the model believed its prediction
to be correct.
The subject was then asked to make a final prediction.
Lastly, before moving onto the next task, the subject indicated their belief in the
model’s prediction being correct as a number between 0 and 1.
Once these tasks were completed, subjects were told both the model’s accuracy and
their own accuracy in Phase 1.
Afterwards, subjects were asked to complete an additional 20 tasks in Phase 2.
Finally, subjects were asked to take an exit survey, where they provided their self-reported
trust in the model’s predictions for both Phase 1 and Phase 2 individually.
We considered a total of 8 experimental treatments with a 2 by 2 by 2 design along three factors,
each having two levels: the level of confidence that the model associated with its predictions,
the model’s stated accuracy on held-out data, and the model’s observed accuracy
on the Phase 1 tasks.
To minimize differences across treatments, subjects in all 8 treatments saw the same
40 tasks, and in treatments that had the same observed accuracy, the models gave the same
binary predictions in Phase 1.
In Phase 2, the models for all 8 treatments gave the same binary predictions.
We used four different measures to quantify the subject’s trust in the machine learning
model, and we computed the values of these measures separately for each phase: subject’s
belief in model accuracy, representing, on average how much belief they had in the model’s
predictions being correct, agreement fraction, representing how often the subject’s final
prediction agreed with the model’s, switch fraction, representing how often the subject
initially disagreed with the model, but switched their final prediction to agree with the model’s,
and self-reported trust, which is the level of trust that the subject self-reported to
have at the end of the experiment.
We start with the question of how users’ trust in machine learning is impacted by the
model’s stated accuracy and its confidence before the model’s accuracy is observed
in practice.
We considered the model’s stated accuracy and confidence level as our independent variables,
and the four trust measures detailed in the previous slide as our dependent variables.
To analyze the effects of our independent variables on the four trust measures, we calculated
Cohen’s d effect sizes along with 95% bootstrap confidence intervals for each trust measure
between subjects assigned to different levels of a given independent variable.
We interpret the significance of a confidence interval based on where it falls in relation
to 0.
For example, if the full range of the interval is greater than 0, then that implies that
trust was significantly greater for subjects in treatments with a high performance indicator,
such as high confidence.
Looking at the effect of confidence in Phase 1, the results suggest that before observing
a machine learning model’s accuracy in practice, people believe that a model with high confidence
scores is more accurate, but we found no other significant effect on the other three measures
of trust.
In contrast, for the effect of stated accuracy in Phase 1, we find that people not only believe
that a model with a high stated accuracy is more accurate, but they self-report to have
greater trust in it and follow its predictions more frequently.
We now examine our data from Phase 2 to understand how laypeople’s trust in a machine learning
model is affected by the model’s stated accuracy, observed accuracy, and model confidence
after observing the model’s accuracy in practice.
Our Phase 1 analysis method was repeated, this time with observed accuracy as an additional
independent variable to consider.
Consistent with Phase 1, we found that subjects still tended to believe that models producing
higher levels of confidence are more likely to be correct in Phase 2, but we found no
evidence that model confidence has any impact on the other three trust measures.
On the other hand, higher stated accuracy was found to lead to higher belief in model
accuracy, agreement fraction, and switch fraction, while a higher observed accuracy led to increases
in all four trust measures.
Altogether, while the level of confidence of the model had the greatest impact on subjects’
belief in the model’s accuracy, observed accuracy of the model had the greatest impact
on the other three trust measures.
Overall, our results suggest that model confidence and model accuracy play different roles in
influencing people’s trust in a machine learning model.
Model confidence mostly influences people’s belief in the model’s accuracy, but the
model’s stated accuracy and observed accuracy consistently impact how much people think
they trust a model and how frequently they actually follow a model.
We conjecture that this may be because when both accuracy and confidence are presented,
people consider accuracy as a fact, but deem confidence as only an estimate.
These findings indicate the potential need of helping laypeople better understand the
uncertainty inherent in performance calculations and calibration estimation based on a small
set of predictions, such as was the case with observed accuracy in our experiment.
It is critical for people to see the value of continued use of a calibrated confidence
score to adjust their trust in a machine learning model, even after observing a very high or
very low accuracy of the model in practice.
Thank you for listening to my talk!
Please check out the full paper in the CHI 2022 proceedings for more information.