Hi, my name is Michelle Brachman from IBM Research
and I'll be presenting our paper: Reliance and Automation for Human-AI Collaborative Data Labeling Conflict Resolution
Our work is motivated by the need for labeled data for supervised and semi-supervised machine learning systems
Often this data is labeled by multiple humans
to increase reliability
but if the labelers don't agree, then there's a conflict
and the hardest conflicts often need to be resolved by humans
because they are on the boundaries, but easier conflicts may be able to be resolved in more automated ways.
So our goal in this work was to understand whether human-AI collaboration could improve
the efficiency and accuracy of conflict resolution
compared to humans resolving all of the conflicts manually
and we did this by combining automated approaches with human approaches in a conflict resolution tool
We used that tool to explore the impact
of different types of information
and the ability to resolve groups of conflicts as opposed to
having to resolve all of the conflicts manually
First, I will talk through our conflict resolution tool
and the variations we used in this study
The default mode in our tool is manual conflict resolution
In which users see an individual data item, like you can see here
We show an example of a short text data item from our study
but this tool could be used for other types of data, like images
So in this example, our data item is
Who do I contact to change my contract from my old phone to my new phone?
and the goal of the user here is to select
the label that fits this data item best
For each data item, users can see the labels that were selected by the original set of labelers
And to manually resolve one of these conflicts,
they can choose one of the labels and click next
We had two types of information that we provided
depending on the condition of the user
The first was the percentage of labelers who picked each of the labels
and the second is the AI predicted probability of each of the labels
Depending on the condition, users saw one of these types of information or both of these types of information.
Our data items were grouped
either by the labeler agreement, AI predicted probability, or a combination
We provided these groups
to signify to the users which conflicts might be simpler or more challenging
For example a conflict with
a lower labeler agreement might be more challenging
whereas one where more of the labelers agreed,
and had a higher labeler agreement
might be easier to resolve
Users could choose which group they wanted to work on
Half of the users had the option to choose to review a group manually or to resolve a whole group at once
While the other half could only choose to resolve conflicts individually one by one
If a user chose to resolve a group of conflicts
they had the option to review the data and labels
that would be applied before confirming
Depending on which information they had available,
participants could either resolve using labeler's top choice, or AI's top prediction
or they could choose between them
Using this tool, we ran a between-subjects study
to understand when and why users trust human labels
and/or AI predictions to resolve groups of conflicts
We ran our study using Amazon Mechanical Turk
We provided bonuses based on a combination of time and accuracy
and our goal here was really to try to motivate users to balances these two constraints
because in a real conflict resolution context,
the person would probably be busy
so they have a limited amount of time to resolve the conflicts
but they also really need high accuracy
because otherwise you risk the negative impacts that poor data labeling could have on a machine learning algorithm
Participants' goal in the study was to resolve 100 data labeling conflicts
and afterward they compled a survey
where they responded about their choices in using group resolve or manual resolve
So we had three research questions:
First, how do automation and labeler-AI information
affect accuracy over time, time, and accuracy
Second, how do users rely on labelers,
AI predictions and the combination
of labelers and AI predictions
for conflict resolution
And third,
How do users decide whether to rely on automated conflict resolution?
We found that participants who had the choice
Between manual or group resolve
had significantly higher accuracy/time
than those with only manual resolution
This difference was primarily impacted by time
with no significant difference in accuracy
So basically what this means is, that when users could
choose to resolve whole sets of conflicts at once,
they were faster
without having lower accuracy
even though they were using an automated approach
Next, we wanted to understand how users relied on
labeler and AI predictions for conflict resolution
and somewhat surprisingly, we didn't find a difference in
the use of group resolve
depending on whether users had information about
human labeler choices, AI predictions, or both
However, those with only the labeler information
did resolve fewer conflicts manually
than those who had AI information
this meant that they resolved more conflicts
using the group resolve
which may have led to overreliance on the system
Finally, we wanted to better understand
how users chose to rely on group resolve
Participants used a combination of system information
and their own preferences
For example, they checked the high-level performance of the system
such as the percentages of the labelers or the confidence of the AI
And they also checked individual items manually and in the groups to make decisions
Some of their choices were also based on their own confidence
which was heightened after more use of the system
and also their general perceptions of whether they should trust the system
Based on these results, we made suggestions for future systems
Our results indicate that participants can often make informed decisions about how to use automation
based on the fact that participants who had the choice
between manual and group resolution were
faster without a loss in accuracy
And we also found that participants were more likely to manually resolve conflicts
in groups with lower labeler or AI confidence
and use group resolve in cases with higher labeler agreement or AI accuracy
However, not all participants were successful in making effective decisions
and likely need more support in choosing how and when to use manual and group conflict resolution
We also found that participants often wanted to check conflicts
in group resolution before applying them
or hedge, to make sure that the automation they were using was going to work correctly
In our study, groups were primarily small
less than 50 data items
but imagine a case with much more data
users would likely need to evaluate groups of conflicts in ways other than checking each item individually
such as interactive approaches that allow users to interact with large groups
in order to be able to check them in a more efficient way
Thank you so much for listening and we hope you'll reach out to us with questions or to discuss more on these topics!