Hello and welcome my name is Michael Desmond and i will be presenting our paper entitled
"Increasing the speed and accuracy of data labeling through an ai-assisted interface".
I want to start by setting some context. Many learning problems still require people to manually
examine and label data, and it's well understood that labeling can be tedious and error-prone.
Human labelers get distracted, they accidentally select the wrong label,
their interpretations of labels change over time, and in a general sense human effort is costly.
Large label spaces are a further challenge. As the number of labels increases
it gets more difficult to make decisions, and the basic mechanics of labeling become more difficult.
We wanted to understand how ai assistance could help the labeling process especially when working
with larger label sets. So what do we mean by ai assisted labeling? At a high level it's all
about using an estimator or a statistical model to predict labels so that as you, the labeler,
labels data a model is simultaneously learning to predict labels for the examples that you have not
not yet labeled. And we can use these predictions to assist the labeler, and make the labeling
experience easier. Hence AI assisted labeling. More concretely we use label predictions to Rank,
Annotate and Reduce the label space. So if you look to figure one on the bottom left here,
you have an example, and the labeler is required to choose the correct label.
Notice that each label is annotated with a blue bar. This represents the likelihood
of it being the correct label. There is also a percentage representation of the same information,
and labels are also ordered. All of this information is derived from a label prediction.
Finally there is a "View More" option at the bottom of the view. This is important. So
initially the visible label set is limited to the top five labels, the top five predictions,
to reduce what we call the labeler's decision space. The "View More" option expands the full
labeling space as can be seen in figure 2. Okay so hopefully everything is clear so far. I want
to now talk about the empirical study design. So I already talked about the research question
and I mentioned that this is an empirical study. We developed three hypotheses as a framework for
the study. H1 is essentially, AI assistance improves labeling accuracy. H2: AI Assistance
decreases labeling time, you can also think about this as increasing labeling speed. And finally H3:
Improving the accuracy of the assistance improves the accuracy of the human labeler. That last one
is an attempt to measure and understand how human labelers rely on ai assistance. We ended
up with three study conditions. The baseline condition was the simplest, This group used a
labeling interface without any assistance. No annotations, no ranking of labels, no model,
just an alphabetically ordered list of labels. Hence the baseline.
Then we had two AI Assisted groups. We had the weak AI Assisted condition
and a strong AI Assisted condition. Both of the AI Assisted groups used the Assisted interface. This
interface contains the Rank, Annotate and Reduce features that I described in the previous section.
The difference between the two AI Assisted conditions was the quality of the predictions,
hence weak and strong. The weak AI Assistance group had predictions from a minimally trained
model, and the strong AI Assistance group had predictions from a better trained model,
which was more accurate in predicting the correct labels. We went with a within subject's design
meaning that every participant would be exposed to one condition only. We designed a 30-minute
short text labeling task, and the domain was telecommunication customer service inquiries.
We randomly selected a set of 100 short text examples that each participant would label, and
there were 21 labels to choose from. Of the 100 examples the first 21 examples were interactive
training examples, meaning that the labeler would choose a label and they would get feedback as
to whether they were correct or incorrect, along with labeling guidelines. This would
allow participants to warm up and become familiar with both the interface and the labels. For the
remaining 79 examples the labeler would need to choose the correct label without feedback.
We recruited participants from amazon mechanical turk. We were very focused on keeping the
experiment realistic, so we chose only master workers and paid a small bonus for correctness.
So each participant could stand to make 1 dollar 58c on top of their flat fee, if they
got everything correct. And finally a quick note on the data that we collected during the study.
So we recorded the label selections to measure accuracy against our ground truth. Labeling
duration was recorded per example, and we trimmed any loading times etc. We recorded use of the
"View more" function and this of course applied to the assisted conditions only. So let's move on
to the analysis and the results. Let's start by looking at the results around labeling accuracy.
I have included the hypothesis H1 for reference, and this hypothesis states that essentially,
AI Assistance improves human labeling accuracy. So for the analysis we conducted a one-way ANOVA
between our three conditions, and accuracy amongst the conditions was significantly different.
You can see the mean and standard deviations in the list there so participants in the baseline
condition were approximately 72% accurate, in the weak AI Assisted condition they were 79% accurate
and in the strong AI Assisted condition they were 78% accurate. Interestingly enough
the strong AI group performed slightly worse than the weak AI group and I'll have more on
that a little later. In the post-hoc analysis we found that there was a significant difference
between the baseline and the weak AI Assistance, however we found that
the difference between the baseline and the strong AI Assistance was not significant. So
we accept h1. Moving on to labeling duration and again i've included the related hypotheses
for reference. We expect that the labeling time decreases in the assistance conditions.
So for this analysis we used a linear mixed model and we found a significant effect of
assistance condition on labeling duration. In the post-hoc analysis we found that the
weak AI Assisted group took significantly less time to label than the baseline group.
And also the strong AI Assisted group took significantly less time to label than the baseline
group. The chart on the right shows the mean labeling duration across the three conditions.
We were surprised at how small the decrease in labeling time was due to the AI Assistance,
we expected a more dramatic effect. So to better understand this we needed to look at the usage of
the "View more" function during the study. So we accept h2 but there seems to be more
going on here. Looking at the use of the "View more" feature in the AI Assisted conditions.
Remember that the use of "View more" means that the labeler expanded the full set of labels.
Presumably they could not find the label they wanted and the top five initially presented
in the AI Assisted interface. We observed a very large difference in labeling duration
when the "View more" function was used compared to both the baseline and the AI Assisted cases
where "View more" was not used. So in effect when AI assisted labelers found the label they wanted
in the initial set of five labels, they were very much faster than the baseline. However when they
could not find the correct label in the top five, they were very slow, even slower than the baseline
condition, and by a large margin. So some of this can be explained due to the mechanical
task of expanding the labels, so you have to click the "View more" button to expand. However
we determined that the bigger problem here was that the probabilistic ordering of labels per
example removed the consistent spatial ordering of labels that was present in the baseline interface.
So the AI assisted labelers were unable to learn and take advantage of consistent
label positions because the order of the labels changed for every example. The final hypothesis
that we considered was around variation in the quality of predictions, and how that affects
the accuracy of the human labeler. We expected that a more accurate AI assistance would improve
human labeling accuracy. It turned out that the accuracy difference between the weak AI assistance
group and the strong AI assistance group was not significant. So for this study we reject H3.
And in support of this result we also found that the weak AI assistance group
used the "View more" feature significantly more than the strong AI assistance group, meaning that
the labelers using the weak AI needed to search outside of the recommended label set more often.
So at least at a high level we didn't observe any over reliance on the AI assistance.
Beyond the results and analysis we also wanted to capture some design recommendations related to
this work. So going back to one of our surprising results. We observed that it was faster to label
data with the baseline interface than the ai assisted interface in the case where the "View
more" option was necessary. And we traced this back to the ordering of the labels.
In the baseline case the labels are consistently ordered alphabetically, so this introduces a
spatial dedication. And users can learn and rely on this to index and find labels very quickly.
In the AI assisted case the labels were reordered by probability
or likelihood of correctness for each example. So there was no consistent spatial location of any
particular label. They would change depending on the model's predictions for each example.
And what this means is that human labelers could not rely on spatial positions of labels in the
interface and found it hard to locate particular labels, so they needed to essentially scan the
entire list every time. And we found plenty of evidence of this phenomenon in the survey
data. I have included some quotes here. So the takeaway is that label recommendations
are best presented alongside or in the context of a consistent spatial ordering.
And this way if the labeler can't find the label in the recommended list, they can fall back on a
consistent alphabetical list for example. Another somewhat surprising result we ran into was that
the weak AI assistance group were slightly more accurate than the strong AI assistance group.
And we wanted to understand this a little bit deeper because it seemed unintuitive. Analyzing
the probability distributions of the model predictions we discovered that the weak AI
was more certain in its predictions than the strong AI. And you can see in the chart that
the weak AI was often highly confident when it was both correct and incorrect. Digging a little
deeper we found that labelers agreed more often with the strong AI when it was incorrect as
compared to the weak AI when it was incorrect. And the the theory here is that it's possible that the
the confident but often incorrect weak AI fostered a greater degree of distrust
on behalf of the laborers who were then more likely to scrutinize its predictions.
On the other hand the strong AI being more often correct and exhibiting a more appropriate
confidence range may have caused labelers to more often trust incorrect predictions. So overall we
believe that the calculation and presentation of model confidence should be carefully considered
given that it may affect labeler performance in subtle ways.