- Hello.
Okay so yes, I'm Mark Cartwright from NYU,
and I'll be talking about crowdsourcing multi-label
audio annotation tasks with citizen scientists.
So first of all context, so I'm a researcher
on the Sounds in New York City project, SONYC,
which is a large project aimed at monitoring, analyzing,
and mitigating urban noise pollution in New York City.
Which has a lot of health and
quality of life effects in the city.
In particular I work in the machine listening
crowdsourcing sides of the project
and our goal is to build multi-label models
that can detect multiple sound events in a scene
to support our data-driven analysis
and build tools for both citizens and city agencies.
And we have deployed 60 sensors which have collected
130 million recordings that are 10 seconds long
which equates to around 41 years worth of audio
if you concatenate it all together.
And this is the audio that we're gonna use
to train our models but of course we need
to label some of this audio.
So in conjunction with the Department of
Environmental Protection in New York City
and through inspecting the noise code
we've developed a set of 23 fine level sound classes
that are grouped into eight coarse level
sound class categories for labeling our audio.
And so this isn't really an exhaustive set of sounds
but it's aimed to be descriptive and actionable
while still being feasible for people to label
and for us to train models on.
So due to the public interest in this topic
we've actually turned to citizen scientists
to help us annotate some of this data.
So we launched a campaign on Zooniverse which is
the largest citizen science platform.
But because of the temporal dimension of audio,
audio annotation is much more time intensive
than the image annotation tasks
that really dominate Zooniverse.
You can't just glance and click to annotate.
So this had us question which annotation norms
we should adopt on this task.
And led us to the following research question
which is how does the type of multi-label annotation task
affect throughput and quality?
Should we adopt the norms of paid crowdsourcing tasks,
paid crowdsourcing audio tasks, rather,
and break annotation into multiple binary annotation tasks,
or should we adopt the norms of image annotation
with citizen scientists like the tasks on Zooniverse
and use multi-label annotation tasks?
Therefore we started collecting our labels,
started collecting labels of our
sonic recordings on Zooniverse while simultaenously
running a small study which
aimed for high ecological validity.
The study wasn't necessarily conducted
as a controlled experiment but rather was
an analysis of a real world data collection,
you know solving real world audio annotation problems.
We wanted authentic behaviors of
volunteering citizen scientists which may differ
from those of paid workers on commercial platforms.
So to measure the quality of the labels
we mixed in the occasional recording from YouTube
for which we have some ground truth into our
set of recordings for annotation.
And we tested three task variations all of which
gave us full multi-label annotations.
We tested multiple paths of binary labeling,
single pass multi-labeling,
and somewhat hybrid multi-pass hierarchical multi-labeling.
We had 339 participants and we asked five annotators
per recording to annotate the multi-label tasks
and for practical purposes we only collected
three annotators per label for the binary
but doing full multi-label annotation
which I'll explain in a minute.
So just to give you an example of what these tasks are like.
So given this sound recording,
(loud construction equipment)
So for the binary labeling task in which it asks
is there a jackhammer present in this recording
let's say, I would say yes, there's a jackhammer present
in this recording and in binary annotation
we would ask a series of these questions
for all of the 23 classes we're interested in.
In the multi-label annotation task you would say
there's a jackhammer and there's a large rotating saw,
and then in the hierarchical task first you would label
the high level classes such as impact tools
and powered sawing tools and then a different set
of workers would actually or volunteers would
go and annotate the fine levels for those class categories.
So for the impact tools you would say there's a jackhammer
and for the powered sawing tools you would say
there's a large rotating saw.
So what do we find?
So in regards to the annotation throughput
we found that binary labeling generated more
overall positive labels per recording.
And the multi-label and hierarchical multi-label tasks
generated about the same number
of positive labels per recording.
And as we'll see in a minute when there's more
labels generated like in the binary labeling tasks
this leads to higher recall in terms
of the annotation quality which we'll see in a minute.
And also in regards to annotation throughput
the binary labeling task, so the individual task,
took half as long as the multi-label task
for an individual annotation.
Now a note about that is that we could do
simply do positive and negative sampling of examples
for binary annotation and not get the full
multi-label annotation which you would have, let's say,
we classifiers go through your data,
find a, propose a list of recordings that it thinks
are positive and then have those verified by annotators.
However this requires good sampling mechanisms
which is the case for some of the classes
but many of the classes that is actually not the case.
So in addition there's also been some studies
that have shown that full multi-label annotations
actually lead to better machine listening models as well.
So with that in mind if you actually think about
doing binary annotation for all 23 classes
of course that's gonna lead to a much longer time
for getting complete, full multi-label annotation.
And multi-label and hierarchical multi-label
are much less in that sense.
So going on to the annotation quality.
So in these plots here we have the f score on the top row,
precision in the middle, and recall at the bottom,
and multi-label annotation, then hierarchical
and binary from left to right.
And on the x-axis you have the minimum votes
for a positive label.
So let's say if only one annotator said that this
class was present, it would get a positive label.
And then the colors are the number of annotators
that we asked to do the task.
So here we find that volunteers tend to
over annotate leading to lower precision in the
binary labeling task and they tend to under annotate
leading to lower recall in the multi-labeling task.
But we can vary the number of annotators
and voting thresholds during aggregation
to balance these tradeoffs.
For example in multi-label annotation we could use
a lower voting threshold with at least three annotators
to increase recall while maintaining precision.
And this result is actually pretty similar to a result
that was found in crowdsourced video annotation.
And the results from the hierarchical multi-labeling task
were not very positive.
The recall was much lower and this is due to
kind of false negatives compounding
at each stage of the annotation process.
And there was a different study on actually
crowdsourced video annotation as well
that found a similar result.
So we were able to get some feedback from participants
but due to the kind of constraints of the platform
we weren't able to really survey all of the people
so in the minimal feedback we were able to get
we found that in general when given the
binary labeling tasks the participants
in the crowdsourcing platform felt underutilized.
They wanted to do more
but they couldn't and were frustrated.
And this is a very different conclusion than
people have come up with when doing similar tasks
in paid crowdsourcing environments when given the
multi-label task people actually in that task
they actually become frustrated rather than the opposite.
And they find it more overwhelming.
So in the conclusions of the study we found
that overall quality of multi-label annotations
from binary and multi-label tasks are pretty comparable,
they have differences but they can be balanced
through the way we aggregate the data.
And multi-label annotation's much more efficient
but only if you really need multi-label annotation.
If you can get by with this kind of
positive negative sampling,
binary annotation can be efficient as well.
In the hierarchical multi-label tends to
propagate error leading to lower recall
and in informal feedback that we received
we indicated that volunteers much preferred the multi-label
which the opposite of the paid crowdsource workers.
And these results really side with the common practice
of citizen science image annotation
rather than that of paid audio crowdsourcing.
So those sort of norms from the image annotation
do go over to the audio annotation side as well.
So we've been continuing with this since we
need to collect our data so we have an ongoing
citizen science annotation campaign
and thus far we've collected data from
about 1,000 annotators, registered annotators rather,
there's actually a couple thousand more
unregistered annotators, and have received around
30,000 full multi-label annotations, audio annotations,
and for around 10,000 completed audio recordings
in which we got three annotators per recording.
And our goal is to get about 1,000 positive examples
for each of the sound classes but the data is,
the way data is distributed in actual natural recordings
is very skewed towards these common classes
so we're still fairly far from our goal.
The green here is when we've gotten
three annotators per label, they're the green in that label,
and orange in two, blue one.
So we still have a bit more work to do.
We've actually released some of this data
as part of a challenge which is part of the
Detection and Classification of Audio, sorry,
Acoustic Scenes and Events which
we're hosting at NYU in October.
So we released about close to 3,000 recordings.
Some of these are training recordings and
over 400 of them we annotated ourselves for validation
and these have three Zooniverse annotators per recording.
And if you look at the statistics for that
and compare it to how the crowdsourced annotators are
comparing to the sonic research team they're doing
pretty well on the coarse level but in the fine level
there's still a lot of confusions.
There's a lot more work that needs to be done
to improve the quality of those.
So we're looking into incorporating more experts
into our loop to improve the quality
of these fine level annotations.
And that's it.
Any questions?
(audience applauds)
- [Jenny] Thank you, Jenny Priess,
thank you for your presentation.
I understand that this is a sort of
methodological experiment but can you give us
some ideas of when it might be useful to be able
to separate out different sounds.
I mean the one thing I can think of is bird song
in woodlands but a lot of the time I think people
want to know the decibel level of the sound.
I live opposite to a place where building is going on
and I actually don't care too much about what's making
the sound, whether it's jackhammers or whatever else it is,
but what I do care a lot about is the level of the sound
that wakes me up early in the morning.
- Sure.
- [Jenny] The decibel level.
- Sure yeah, so our sensors are recording decibel level,
but in order to actually kind of make the decibel level
actionable we need to know what kind of sound it is, right?
So in order for the, there's different,
for instance the Department of Environmental Protection,
they have different, in the noise code,
there's different levels for each of the different
types of sounds of what is tolerable in the city,
what the city has at least defined as tolerable, right?
So if they know that a pile driver for instance,
which is you know, driving big piles down into the ground,
laying the foundation for something,
if that's exceeding the noise level
at a certain time period that's a really big deal.
Whereas if it's, you're complaining,
about your neighbor's dog or something like that
that's making a bunch of noise, you know,
that's kind of, the city treats that differently.
- [Jenny] But that's not normally sorting out
a whole set of different sounds, is it,
as you're doing in your experiment?
- So we need to be able to identify those sounds
from our sensors to make that information
actionable for city agencies.
- [Jenny] Yeah, okay.
I guess what I really question is the realism
of the results that you are getting in real term
applications, but anyway, thank you.
- Okay, so connecting the loop, I guess.
So, oh, okay, whatever, but.
(laughter)
- [Jenny] Thank you.
- [Sandy] Hello, I'm Sandy Gould.
Do you have any insight into the relationship
between so you're using citizen scientists to do this,
do you have any insight into sort of,
the results are nice and clear and lay out really well,
do you have any insight into how these people
are taking on these tasks and how they're
working on them and how that relates to it?
So obviously the binary one doesn't perform all that well
but they're quick and easy for people to do, right?
They're not very taxing relatively speaking.
I thought there was an interesting comment there
about trying to extract more information from it.
Do you get much insight from the way people
are working on these tasks and how that might
influence the choice of which kind
of selection approach you take?
Does that make any sense?
- Um...
- [Sandy] No?
(laughter)
- Can you be a bit more specific?
- [Sandy] So you've got three different ways
of people completing this task.
- Yes.
- [Sandy] And they perform in their own terms
at the different levels but how do they fit with
how people's experience of completing these tasks, right,
the actual people doing it at the other end of the wire?
- Sure, yeah, so I mean, as I said,
due to constraints of the platform we weren't really,
we originally had like a really big
complete survey on there,
we had all this other stuff,
and they actually had us remove all of that
before we launched the task 'cause they really
try to make the tasks really easy for people to complete
and you don't want to ask too much of people
in terms of oh you need to follow this link
and do this survey and whatnot, so.
It's not, Zooniverse really isn't a platform
for experimentation that much, right,
so we weren't really able to collect that data.
So we had some feedback from participants
through like talkboards and direct messages to us
and we tried to message a bunch of people
to try to get more information from them,
but it just really wasn't possible.
So we unfortunately don't know much
about some of those questions at the moment,
just except for this little bits of
informal feedback that we had.
- [Sandy] Yeah, thanks very much,
it's not an issue with the results at all,
just I was just curious, thank you, cheers.