- Thank you for the introduction.
So, this is just a little bit
of background about crowdsourcing.
I don't rely that every one here's,
everyone in this talk knows crowdsourcing,
so just some background.
So, this is a very popular technique for collecting labels
from machine learning training later.
Problem is that the labels can be noisy,
so the common practice is to collect many labels
for each item, and then to aggregate the labels
to get the single labels.
This is a fun example, so for this picture,
let's assume we're three labels from three crowd workers,
and so if we just see a majority voting,
then the aggregated label is just a dog.
So, our crowdsourcing aggregation problem
is a very popular problem in machine learning,
and recently, I know of at least 20 people
and probably more than that on this problem,
but the goal in most of this previous work,
the idea is the following, and this is a very simple idea.
So, they improved from majority voting by waiting
by these estimates of this annotated performance,
so if an annotator has a higher estimate,
performance estimate, then their labels
would be weighted more.
So, and then, also in this previous work,
the aim is to improve aggregation accuracy,
which is important, but in this work,
we take a slightly different direction, so recently,
there have been a lot of interesting
explainable machine learning and explainable models,
so we ask the question, can we make aggregate graphs
of signal aggregation modeling more explainable?
So, by explainable, we are specifically interested
in explaining the errors by these annotators,
so the question is, why the annotators made mistakes,
and yeah, so we would like to use an aggregation models
to detect these patterns of annotation error.
The key ideas in this paper is to combine aggregation
and clustering, and so there are some,
a lot of motivation for why we want to do this.
In general, we want to understand the annotators better.
And so, if we understand why they are making the mistakes,
then we can provide, say, personalized feedbacks
to help the annotators improve.
And then, for the requester, they can also
revise the task instruction to get better datas in future.
So, we consider this task in natural language processing,
so the name entity recognition task is the following.
So, suppose you are given a sentence,
"Ashwin Ram gave a keynote talk at ACM IUI in Los Angeles."
So, that was what happened this morning then.
So, even this sentence, the task is to annotate
who are the persons, the organization, and the location
in this text, and so this is a very useful task
in natural language professing it is useful for downstream,
many downstream application in translation,
question answering, conversations, so,
and so we use this data set.
This is a standard data set in the name,
name entity recognition task,
and we use the crowd label collected by that table.
So, this is just an example of what
an explanation would look like.
So, what's that worker, a crowd worker missed,
so these are basically the missed texts
and notation missed text in this
named entity recognition task, so if we look at
the first cluster, so you can see that this is mostly
acronym, and with some punctuations in the text,
so probably they are being confused by punctuation,
and the second one is, so these are all capitalized acronyms
and the interesting one is the third cluster
where you see a lot of nationalities, so this is,
these nationality are usually in organization words,
and it looks like this worker
has some confusion about nationality.
Okay, so again, this key idea
is to combine aggregation and clustering.
So, we apply a very basic and very classical
clustering algorithm, and in this David and Skene
classic model, this is a, so I just summarized,
this is a directive algorithm that alternates between
estimating the true label and estimating
the worker performance represented by confusing matrixes.
So, that was the aggregation modeling, and for clustering,
we also use a very basic clustering algorithm
based on hidden variable mixture model, so this is,
in this model, they are some certain, if that's a cluster ID
hidden variables and data generated condition
on cluster ID and parameters.
So, in graphical modeling, this is the picture
where this aggregation model on the top right,
and then we have a clustering model in the bottom left,
and then the idea is to combine
these two models together in the graphical modeling.
So, here's some details on learning algorithm.
We use expectation, maximization,
and Gibbs sampling with details on the paper.
So, our first experiment is on, this is a competition
to expand my own, the quality of the clusters that we got.
So, we compare this to a baseline of basically just
aggregating and then to clustering separate, at least.
So just a reminder, our method is aggregate and cluster
in a single model, and this is separately, so,
and we use this metric on evaluating cluster quality.
So, in this data set, so we see that,
so for the bulker, the good news is that
they are some improvement over the baseline.
They are a little bit small, but it's good to see, and,
we also see that with more data,
this improvement is larger, so that, I think, is good, good.
So, that was the competition experiment,
and then for the human experiment, the question is,
can people actually understand the cluster?
So, we have this stance again on Mechanical Turk
because it's easy to do, so, but over 100 participants.
So, people run the either side to two groups.
In the first group, they see a list of correct
and wrong annotation, and because it's a list group,
and the second group is, they see the cluster,
so basically, this, what cluster is at I showed you earlier,
so the hypothesis here is set by looking at cluster.
So, if you are a requester, you are trying to understand
why the workers made some mistakes.
Probably what you do is you look at the list
of correct and wrong annotations,
and here we are trying to see if, instead,
people look at cluster and see if they have
a better understanding of why workers made mistake.
So, the task here is to, if this is an objective task
just to predict future annotation by the same worker.
We have five questions for each Mechanical Turk HIT,
and so they can just do one HIT and quit,
or they can just do as many as they like.
A lot of people quit actually, so.
So, this is the results.
The X-axis is the number answer,
so this is like the number of questions that they answer,
so as I said, there are five questions for each,
and with eight, eight in total,
we have a total of 40 answers.
So far, this is four, five.
This is for people who just did
one HIT and then quit the task.
So, we plot these accuracy
and 95% confident interval
comparing the list group and the cluster group
for predicting, this is the accuracy
for predicting future annotations.
For, we have several observations, so first on is,
it looks like cluster is generally better,
although not so good, the difference is small,
and then we also observe that this is a difficult task,
and in general, we have a quite low accuracy.
Another observation is whether people just did
one HIT and quit or they did everything.
We don't see a big difference in that performance.
Yeah, so for conclusion,
crowdsourcing is a very popular mechanism
for collecting labels, and there have been a lot of work
on aggregation modelings.
So, in this paper, we see this as a step
toward the explainable crowdsourcing aggregation,
and we have the key idea
of combining aggregation and clustering.
There are some promising results, but I also acknowledge
several limitations, so this is a difficult task,
yet small numerical difference, and the cluster
sometimes very difficult to integrate,
so just one idea for future work is
to have more interaction to get better clusters.
So, I also acknowledge the Mechanical Turk's participants
and reviewers, and NSF, and it looks like I finished early,
so we have time for questions.
(audience applauding)
- [Moderator] Questions?
We're a little stuck, oh, got one.
- [Man] Hi, Walter, looks like the University of Michigan.
I was just wondering, so you have this limitation here
that says, "Clusters are sometimes hard to interpret."
I wonder if you thought about adding in
some level of explanation, so the person doesn't just
give an answer, but actually tries to provide
some of what it is that they're thinking about
when they give the answer, does that?
- Yes, we did a little bit of that,
but it can be very difficult to exit in for...
Most of the time, it's just very difficult,
so in the task, we have like a box
that is where they can enter their, say their explanation
for their predictions, so it's like rationale
for their prediction, and I think it was just
a quiet in control mechanism, and we don't find that
actually very, I mean looking at this,
I don't find this very useful, so yeah.
- [Man] Okay, so I just have a quick question.
Looking at your results earlier, you had some,
like 1,000 words and 10,000 words,
and you've shown some improvement,
but then if you're using all the words,
there is some kind of a decline in the--
- Yes, so I mean, these are for different amount of data,
so they are not very comparable, and so basically,
with more data, it's more difficult to get the right,
so the cluster can be more complicated than,
hard to get correct, so.
I mean, the main thing is just
to compare the baseline and the proposed.
- [Moderator] Okay, if there are no more questions,
let us thank the speaker again.
(audience applauding)