- Thanks Michael, so yeah this is work done at
the University of Washington with my collaborators Mausam
and Dan Weld.
So I'm sure you've all heard about crowdsourcing,
has enabled a lot of really exciting discoveries in science
and also new applications in machine learning
and things like that.
So, you've heard all these success stories
suppose you wanna build a classifier to detect cars,
let's say.
So you've written your first instructions,
which say "Is this a car?".
Now maybe just check before you actually launch,
you look through some more data and come across
this image, how many people think this is a car?
Alright, some mixed responses maybe.
So maybe we better update our instructions to specify
that it has to be a real not a cartoon car.
Okay so now we've actually launched our task and
we see there's low inter-annotator agreement on this item
so some workers said that the answer should be yes
some said that it should be no,
so we better update our instructions again,
to specify that it has to be a car you can drive on
a street, not a railroad car.
This process again happens for multiple cars,
maybe for interiors of cars and
you need to see exteriors in order for your
classifier to work.
So, this is a complicated process where you have to
typically interleave a couple of subprocesses
one of which is performing the actual task
and while you're doing this usually
your task definition changes and this is known as
concept evolution like the last speaker referred to.
Another part of the process is actually running the task
to debug it, because initially you don't know what
the models of the workers have in their minds,
this could be viewed as a model mismatch problem.
And you kind of alternate between these two and
revise the instructions accordingly.
And it's also important to revise
the training and testing procedures it's been shown,
because as your instructions grow more complicated
workers might not actually read it and
you might need to reinforce the concepts
that are specified there.
So the dirty secret of crowdsourcing is that
this is a lot of work but unless you don't do this
it doesn't work well, and a lot of people like
to blame the workers or whatever, but we sort of take
the view that a lot of this comes from misspecified tasks.
So you don't have to take my word for it,
other people have said that task design is a major problem,
there's studies showing the difficulties for requesters
as well as the important of task design on work quality,
and also, the risk to workers of
poorly specified tasks and unfair rejections,
is one of the main risks that workers face
in these platforms.
So also crowdsourcing is just a fad, right?
So we don't really care about this problem, maybe.
Well, actually by 2027 there's projected to be
more freelancers than non-freelancers,
so it's really important to help support requesters
in designing good tasks.
Okay so, gonna move on to looking at how can we design tools
that support this range of processes
both performing the task, to define the task
as well as debugging the task.
And coming up with training and testing procedures.
So our system Sprout enables the requester
to explore the task via prioritized bugs essentially,
discovered by the crowd.
So this supports both the goals of of exploration
and task debugging in the sense that,
the crowd is prioritizing exploration based on these bugs.
So the crowd identifies clusters of confusions
and the system displays these to the requester for review,
so here there's multiple cars, partial cars, railroad cars
that again are example.
Afterwards the requester can review these confusions
and edit their instructions and response,
and then the system will rerun the new instructions and
debug the remaining errors, hopefully which are fewer
after this process.
So this loop repeats until debugging is no longer useful,
and as I said hopefully the number of confusions
should decrease over time.
We can visually display some more information
provided by the crowd, here I've shown color
ranging from, that shows the amount of disagreement,
so ranging from all red, red indicates all no answers,
green indicates all yes answers somewhere in-between is
higher disagreement and so the system prioritizes
the largest most confusing clusters to help focus
the requester's effort.
The system also shows requester progress
towards addressing the issues,
as they drill down into the groups and explore the clusters.
So finally the system also provides an area for
the requester to come up with test questions
that can be used to test understanding of the instructions.
So you can just drag, this is a pictorial representation
but you can drag items into this section either,
that should be answered yes or no for instance,
and the instructions also let you imbed examples
as is good practice, here is this red example,
and, as you're typing and you reference an example
the system can also suggest relation test questions
for you to use, to add to the instructions
because it has some sort of semantic embedding
based on the crowd-powered debugging
which I haven't quite explained but
it can find related questions.
So the requester can easily add
one of these suggested questions by dragging it
to one of test question sections.
So, once the instructions have been defined,
Sprout compiles the test questions into a
training and testing workflow,
so this is a very simple process that's been shown
to improve quality, so first there's an interactive tutorial
where workers get feedback after they answer questions,
then there's the test phase where workers are tested on
what they learn in the interactive tutorial,
if they pass-- if they don't pass they're removed from
the task, but if they do pass then they move on
and actually answer task questions.
So, this ensures that only workers who understand
the instructions actually perform your task.
So I don't really have time to go into all
the details of the interface but here's a
view of the actual interface, so A,
part A is the confusions, there's little
summary visualizations of the amount of disagreement
in each confusion.
Group B is the section to drill down into the
crowd-provided explanations,
C there's the instructions editor
and D is the test question area with
the suggested test questions.
So up to now I've sort of just only talked about
the requester facing side, there's also a crowd facing side
of the system that powers the debugging process,
so workers first perform the actual questions
using the instructions in a work phase,
questions that have high disagreement are then move onto
debugging by meta workers which is
a different set of workers, who diagnose whether
this is actually an uncertain or ambiguous question,
and then if it is then they clarify the instructions
to remove the ambiguity, if it's not then they
create a test question with the correct answer.
So this is just a sample of the clarify step of the
worker interface so here there's,
a worker might be shown a picture of a railroad car,
and they're asked to add a clarification bullet
to the instructions, so here the worker could write,
could specify that railroad cars should be labeled no,
for instance, the system just discards these labels of no
because only the requester actually ultimately decide on
what the labels should be, but we found this was more
natural for workers, and has some additional features like
providing autocomplete to try to help cluster results
more than they would otherwise be.
So we connected a user study to see, to look at
the pros and cons of this approach versus a baseline,
and to learn how to better support task design,
so requesters in our user study were given a goal
they were given an underspecified instructions prompt
like is this a car, and they were asked to improve
the instructions to remove ambiguity based on
their own understanding of the concept.
We used two domains from prior work which were both
car images and travel websites,
travel websites were high resolution images of websites
with a mixture of text and images that typically require
more deeper analysis to understand,
and the baseline that we compared to is a,
doesn't use the crowd so instead of the requester
organizes the data themselves instead of the crowd
so this is called Structure Labeling,
and this is a system that was built explicitly
to support concept evolution so the,
the requester can drag items into these
yes maybe or no sections, they can create groups
and they can differ judgements and move around items
between the sections.
So essentially this allows the requester to do
what the crowd was doing with Sprout.
So we had our within-subjects design where each participant
used both interfaces, and we had 11 participants
with varying amounts of crowdsourcing expertise.
So we found that overall Sprout
was preferred by the requesters,
people said things like "Categorizing the inputs,
"showing me the cases where there was confusion,
"etc. made it super easy to identify cases that
"needed clarification", they also noted that
"The similarity metrics seem to be working great
"and the suggested items are great for testing
"the points I emphasized in the instructions".
So this is sort of born out by the fact that
many of the test questions that were actually suggested
came from the suggestions, thanks.
So we also did a preliminary evaluation of
the instruction comprehensiveness.
We found that they were longer,
that requesters using Sprout wrote longer instructions
and used more examples in the instructions,
and also that they resolved more ambiguous categories than
when they weren't using Sprout.
And we evaluate this using experts
to code the number of distinct categories.
So, we weren't able to control for the actual
concept that the requesters had because if you actually
define that upfront and give that to the requester
then this sort of negates the point of the study,
so everyone came up with different, had their own
personal definitions and this was one kind of
confounding factor of the study.
So we also found that like many mixed initiative systems
some requesters liked the support and some didn't
so some wanted to use both tools,
some benefits of Structure Labeling include that
there's no bias introduced by the exploration process
and also let the requester do the task themself
to a larger degree, but of course the trade-off
is that you don't get support,
so people saw value, most people saw value in both tools.
So I wanted to give a kind of an overview of the space of
Sprout in this larger task design tool landscape,
so on the X axis here I've applied
limited task support improvement versus actually the
goal of task support, task improvement support.
And on the Y axis is limited exploration support
versus using the crowd to actually structure
the exploration process.
So in the bottom left we have this baseline
of structured labeling which wasn't actually intended
for task improvement but nicely pairs with task improvement
but it doesn't actually provide a lot of support for that
or for exploration because, the requester has to actually
organize the data set themself.
Revolt is another system that
has the crowd cluster ambiguities, but it does so on the
entire data set rather than Sprout which seeks to
do it on a small portion improve the instructions
to remove the ambiguity and then iterate.
So these are useful for different purposes
Revolt is, produces very reasonable data sets
that you can relabel and repurpose,
and Sprout seeks to remove the ambiguity and lower the cost
of the overall data collection process.
And there's also systems like Daemo and WingIt which
are seeking to provide task improvement
but provide limited exploration support,
and Sprout seeks to do both of these by
prioritized debugging.
So some future work, I'm excited about expanding
the roles further of workers and algorithms and
supporting the requesters so I think you could
try to automate the instructions further and
for instance by inferring requester decisions,
we could try to create even more engaging
and progressive tutorials other than the automated
get instruction that was created with Sprout.
Also I think workers contribute to interface design
or workflow design, previously people have looked
at workflow task decomposition but
these have typically involved a lot of effort from the
requester and monitoring, so I think
it'll be exciting to try to fully automate these more.
I think we also need more studies
and actual field deployments to see how useful
these things work in practice, and I'm also really excited
about moving beyond labeling tasks,
and for instance looking at how can we support and define
open-ended and creative tasks better than we do now.
So I'd like to close and take questions.
(applause)
- [Matt] Hi Matt Lease, University of Texas Austin,
Jonathan, really nice work and a great talk as always,
I was thinking if you look at say annotation tasks
the Linguistic Data Consortium does you know
often you have much higher complexity for
say implicit phenomena, the annotation guidelines get
quite like long, or even if you look at the,
the Google annotation guidelines for relevance
which is a, you think of a simple task as something
relevant or not and yet their annotation guidelines
go through a hundred pages.
So, you know as you start adding more exceptions and rules
and things to specify that, I could imagine over time
at some point it gets unwieldy, and we often think like
maybe workers won't read all that much or
as you start getting longer you have to organize the stuff.
So, how do you deal with this thing about as you
have more complex tasks or you start getting more
and more details for all these rules,
how do you manage the instructions so they're still
manageable or at one point do you have to sort of
figure that even though another rule would
be more specific it would become too long for
the workers to actually make useful use of?
- Yeah, so I think that's a really good question
we sorted of pointed on that issue by
just looking at this first step iteration, but,
I think designing better tools to
support navigating larger sets of instructions
maybe, having the machines suggest portions of
the instructions that are relevant
to certain questions might help,
or identifying which parts of the instructions
are actually important for large sets of questions.
So I think, the requester could define that
but I think there's also roles for workers
or automated approaches in helping with that part.
- [Audience Member] I've also got a question,
in the theme of R2 who's often very confident and wrong,
you have that situation very commonly
with human interpretations right where,
we're gonna have a false negative problem
with this kind of system where the workers
may not disagree at all but the requester would have
an entirely different interpretation and this is gonna be
sort of blind from the requester's perspective,
they're gonna think "Oh great I've covered all the cases"
when in fact if they went and looked at the results
they'd say "Oh my gosh they're getting it all wrong"
or something like this. - Yeah.
- [Audience Member] How do we expand
this kind of tool to support that case?
- Yeah so I think, we don't wanna only look at
the cases that have confusion,
so the system also does display these other cases,
I think that the similarity metrics could try
to help do clustering, on the cases where
workers don't disagree so in those cases workers were.