Hello, my name is Ana Elisa Méndez and I'll be presenting our work on eliciting confidence
for improving crowdsourced audio annotations.
Audio annotation is unique and should be studied on its own.
It is a slower process because of the temporal nature of audio. Also, audio can be subjective,
perceptual, or ambiguous.
In real-world audio, multiple sounds can happen at the same time, making it hard to disambiguate
one sound from the other, leaving room for annotator interpretation,
increasing the chances of disagreement, and creating the need to quantify uncertainty
in labels
Machine learning and crowdsourcing research have studied the use and collection of "soft"
labels instead of "hard" labels.
Showing soft/continuous labels provide richer training information and higher quality aggregate
labels (leading to needing less data for training).
We study this problem in the context of machine listening of urban sound because it has many
applications where the data is ambiguous and the quality of the annotations highly impacts
results.
Some examples are: 1) Urban noise pollution monitoring, which
can help in developing plans for mitigating noise pollution
2) Urban bioacoustic monitoring, key in understanding bird migration patterns
3) Sound-awareness tools for the hard of hearing, which can increase awareness of critical sound
events for deaf and hard of hearing people 4) Machine perception for autonomous robots/vehicles,
where incorrect predictions about the environment can potentially be fatal
We seek to investigate
Does confidence elicitation for collecting soft labels affect individual annotation quality?
Do the confidence elicitation method and aggregate label type affect aggregate annotation quality
and cost?
Our goal was to collect fine-grained probability estimates from an individual instead of a
population.
To collect labels through crowdsourcing, we created a web-based annotation application,
which had interfaces to support three annotation
tasks, one for each confidence elicitation method.
In all tasks, we ask participants to select yes or no about the presence of a jackhammer
in the audio recording. Here we show the direct confidence elicitation
method, where participants directly state using a slider (from 50 to 100% in increments
of 10) how confident they are in each label provided these are then converted to probabilities
and used as soft labels. If the presence of the jackhammer was yes, the probability estimates
remain the same, if it was no , the probability estimates are calculated by subtracting the
confidence from 1, e.g., if the answer was no, and the confidence was 60%, the probability
estimate is 40%.
We also proposed a betting mechanism where participants indirectly select their confidence
by placing bets either on their hard label or on an X% chance lottery, playing 5 rounds from
50 to 90%.
Choosing their hard label would mean that they are more confident in winning the bonus
with their answer than winning a lottery with X% chances of winning.
Collected labels consisted of a binary list where 0 meant they selected the lottery, and
1 meant they selected their answer to the presence of the jackhammer
The switch from their answer to the lottery would indicate the
confidence they have in their answer.
After we did this process with all responses for
the betting task, we calculated the probability estimates using the same procedure as for
the direct elicitation task.
For both of these methods, we computed z-scores based directly on their confidence responses
because the distribution of the collected labels depends on the personal interpretation
of the scale presented.
We refer to these labels as z-score derived probability labels.
Here we show the interface of the no confidence task where only the binary question is asked
We studied three confidence elicitation methods: simple or direct, betting, and no confidence
We also studied annotation quality of the different annotation
types: binary, probability, and z-scores derived from the probabilities.
For each elicitation method, we averaged: (1) the binary labels per recording
(2) the probability labels per recording, and
(3) the z-score derived probability labels per recording
Finally, the aggregate labels were directly binarized using a threshold of 0.5 to form
what we call "hard" aggregate labels.
The non-binarized aggregates are called "soft" aggregate labels.
We used Scaper to synthesize 10 s urban soundscapes with audio from UrbanSound8K, a dataset divided
into 10 cross-validation folds, containing 10 classes.
Our goal is to annotate the jackhammer.
Each participant annotated 20 recordings. 10 folds, at 10 participants per fold per
task total 300 mechanical turk participants.
Now we show our results.
In this figure we compare the performance of the three tasks in accuracy, precision,
recall, and F-measure.
These results suggest that both the Simple Confidence and the Betting mechanisms have
better per participant performance (in accuracy and precision) than the No Confidence method.
Suggesting that giving participants an additional step to provide their confidence increases
annotator performance, as participants are allowed to change their answers to the binary
question while they annotate their confidence.
Also, these results suggest that eliciting confidence is beneficial even for tasks where
the model is trained with only binary targets, like binary classification with a support
vector machine or decision tree.
To measure the effects of the elicitation method, annotation type, and their interaction
on the quality of the soft labels per participant,
we focused on the AUPRC metric that does not have a set threshold for binarizing the annotations.
We found that the Simple Confidence elicitation method performs borderline statistically significantly
higher than the Betting mechanism.
However, no significant effects were found for the continuous annotation types (probability
versus z-score derived probability).
Similarly to the previous slide, we measured aggregate performance with AUPRC, as it does
not depend on choosing a threshold for binarizing the labels.
Each curve represents the average over 500 iterations when calculating AUPRC, where in
each iteration we randomly ordered the participants per condition.
For averaged binary labels, both confidence elicitation mechanisms produce aggregate labels
of higher quality than the baseline No Confidence method at all levels of participant averaging.
Within each of the two confidence elicitation methods, we see that soft labels have higher
quality than hard labels for all levels of participant averaging.
In addition, much like the per-participant results presented in the previous slide, when
averaged over few participants, there is little difference between the two elicitation mechanisms.
However, as we increase the number of participants, the quality of labels from the Betting mechanism
achieve higher performance than those for the Simple Confidence mechanism.
To measure the quality of the "soft" annotations, we measured annotator agreement, as we increased
the number of annotators and compared it to the aggregated annotations for all 10 participants.
When comparing "soft" and "hard" aggregate label types for each confidence elicitation method
and annotation type, we can see that agreement for "soft" aggregate labels is higher than agreement
for "hard" aggregate labels at any number of participants.
Suggesting that using "soft" aggregate labels, we can collect annotations from fewer participants
than if we use "hard" aggregate labels.
The key takeaways are, that similarly to previous work, we find that when using confidence elicitation
methods, the resulting annotations have higher accuracy and precision when compared to those
of the baseline, giving further evidence that collecting more fine-grained annotations is
beneficial when dealing with ambiguous data.
Also, because some mediums take longer to annotate, asking a bit more from individual
annotators is preferable to having multiple annotators per data item.
In addition, the methods we present offer a solution for collecting "soft" labels for binary and multi-label tasks.
We propose the Betting method as a way of collecting probabilities while avoiding directly
asking participants for their confidence, since it has been shown that people misjudge
their own confidence.
Also, from preliminary data, it showed to be faster than the direct confidence method.
It also resulted in higher quality aggregate annotations, with differences being more pronounced as
more annotators were added.
Finally, we evaluate our methods in an audio annotation task which has applications to
many important real-world problems, e.g., noise pollution monitoring, urban bioacoustic
monitoring, sound-awareness tools for the hard of hearing, and machine perception for
autonomous robots/vehicles.
Moreover, the annotation methods are not limited to an audio use case.
The confidence elicitation methods evaluated could be applied to other annotation scenarios
where data can be ambiguous, binary or multi-label, e.g., tasks such as video annotation for visual
object detection and tracking, or multiple evidence-based diagnosis.