Hello! I'm Shiye Cao, a first-year PhD student from the Intuitive Computing Lab at Johns Hopkins University.
I'm going to present my paper on "Understanding User Reliance on AI in Assisted Decision-Making".
Performance of artificial intelligence agents is surpassing that of human experts
in select tasks in numerous domains, from cancer diagnosis to recidivism prediction.
Through human-AI collaboration, we hope that the human-AI team performance will be even better than
both the human and the AI's individual performance.
However, in reality, this is often not the case.
Users frequently have trouble gauging when and how much to rely on the AI assistant,
causing the actual human-AI team performance to be superior to the individual human, but inferior to the AI alone.
Therefore, it is important to assess user reliance on AI during assisted decision-making and apply necessary strategies
to help users calibrate their reliance to achieve enhanced human-AI team performance.
Prior work mostly used subjective and task-based metrics to retrospectively evaluate user reliance on the AI.
For example, user's self-perceived reliance on or trust in the AI reported in the post-study survey,
the degree at which user response agreed with the AI suggestion, and whether the user adopted
the AI suggestion if they initially disagreed with the AI.
Few studies have also used EEG and GSR features to predict whether the user would trust or distrust an AI suggestion.
This could allow for real-time estimation of user trust in AI.
However, the special equipment necessary for EEG and GSR data make them difficult to generalize to everyday use.
The rise of real-time eye gaze tracking with regular webcams points to a new possible way of assessing human
reliance on AI for a wider range of everyday situations.
Eye gaze can provide insights into human decision-making and collaboration.
For instance, gaze duration is indicative of people's preferences in selective decision-making.
People spend more time looking at
the options they ultimately choose and prefer things that they spend more time looking at.
In this work, we explore how eye gaze may be used to gauge human reliance on the AI
in assisted decision-making in hopes to enable future adaptive AI assistance
for more optimal human-AI team performance.
We recruited 42 participants for an in-person study.
We contextualize our exploration in a spatial reasoning task
that is commonly used to study cognitive ability in cognitive psychology research.
In each trial, the participants were asked to determine what the 3D structure shown
would look like from the top view or the bottom view on a 4 by 4 grid.
The spatial reasoning skill involved in this task is of importance in many disciplines.
For example, the skill is critical for radiologists in understanding CT scans.
On the web interface, the task image was always shown in the middle,
with the AI suggestions shown to its left, and the user response grid shown to its right.
We used the gaze point GP3 remote eye tracker to capture where on the interface
the users were looking during each and every time step of the experiment.
As an example, we show a participant's eye gaze in green overlaid on top of the web interface.
Each green circle shows the location of a gaze fixation and
the size of the circle is representative of the length of the gaze fixation at that location.
The circles are connected to indicate time sequence.
In this case, the participant quickly glanced at their own response, then glanced at the AI suggestion
and is now looking at the task image.
We employed a 2 by 3 mixed factorial design with the task difficulty as a with-in subjects factor
and the AI accuracy as a between subjects factor.
We hypothesized that user reliance on and trust in the AI will be positively correlated with
the percent gaze duration on the AI suggestion.
Moreover, by increasing task difficulty, regardless of the accuracy of the AI, user reliance on the AI will increase.
By increasing the accuracy of the AI, regardless of task difficulty, user reliance on the AI will decrease.
I will now present some key findings from the study.
For full results, please refer to our manuscript.
We used three different metrics to measure user reliance on the AI.
One, the degree of agreement between the final user response and the AI suggestion
as a task-based metric for reliance.
The higher the agreement ratio, the higher the reliance.
Two, the participants self perceived reliance level on the AI as a perception-based metric for reliance.
Lastly, the person gaze duration on the AI suggestion during the task as a potential behavioral metric of reliance.
Consistent with our hypothesis,
participant gaze duration on the AI is positively correlated with user AI agreement and perceived reliance.
We observed a significant effect of AI performance on perceived reliance and user AI agreement.
As AI performance increased, the user's perceived reliance and likelihood of agreeing with the AI also increased.
This result shows that even though the participants were not informed of the AI performance,
they were able to figure out the quality of the AI suggestions and adjust their reliance on the AI accordingly.
However, no significant effects of AI performance
on participants percent gaze duration on the AI suggestion was found.
This suggests nuanced differences between behavioral reliance, task-based reliance and perception-based reliance,
demonstrating the complexity of the reliance construct.
Gaze can serve as a window to understand how users rely on or use the AI.
Here we show a screen recording of a participant completing a full trial overlaid with their eye gaze.
We observed that in Phase 1, participants spent a lot of time observing
the task image and working on the task by themselves.
The AI suggestion appears when the users
are halfway through the task, which also marks the beginning of Phase 2.
In Phase 2, participants often continue to work on the task by themselves.
They often have much more frequent gaze shifts and spend more time looking at the AI suggestion in Phase 3,
when the participants are halfway through the task time they spend after the AI suggestion appears.
This is consistent with participants accounts that
they often completed the task by themselves first and then use the AI suggestion to check their answers.
Our quantitative analysis of user gaze duration on AI is consistent with our observations.
Participants spent longer time looking at the AI in Phase 3 than Phase 2.
In addition, through analyzing the effects of task difficulty and the interaction effects of task difficulty and task phase
on the participants percent gauge duration on the AI,
we find that participants not only spend more time looking at the AI when the task is more difficult,
but also start to spend more time looking at the AI earlier.
In summary, in this work we show how gauge duration on AI is strongly correlated
with perceived reliance and user-AI task agreement.
This relationship between eye gaze and user reliance on AI opens the possibility
of real-time adaptive human-AI collaboration through gaze monitoring.
As an example, based on observed gaze behavior, an AI agent may provide timely feedback to
help its users better calibrate their reliance on the AI for more optimal task performance.
Beyond user reliance,
eye tracking may also be used to gauge the cognitive, attentional, and emotional state of the user.
A holistic view of the user state can further allow the AI to adaptively assist the user in complex decision making.
Our work contributes an initial exploration of how eye behavior may be used to
estimate the holistic user state and pave the way for enabling more productive,
real-time adaptive human AI collaboration.
I'm really excited about this line of work. Please reach out to me with any questions.
I would love to discuss this work more with you.
Thank you for listening.