Hi everyone, today I am going to share a case study about Task Design for Crowdsourcing
Complex Cognitive Skills.
This case study is built on our prior work in which we were trying to ask crowd workers
to generate many different ideas for a topic. For example, we asked workers to generate
ideas for the topic “Ways to use a brick”.
The key issue is that workers often generate redundant ideas. For example, “build a desk”
is duplicate with “build a sofa”. To avoid duplication,
we leverage Dimensions and Values, or simply called, D/V.
For example, one dimension could be “is_destructive” with two values: yes or no. Here, “break
a window” is destructive, while “use as a chair” is not.
Another dimension could be “cost” with three values: “high”, “medium”, “low”.
These two dimensions can partition the idea space into blocks. Any idea on the left can
be mapped to one block on the right. Two ideas that map to the same block are treated as
duplicate.
We asked workers to find D/V that could be used to categorize the given list of ideas.
However, we keep receiving low-quality D/V.
The reason is that we cannot clarify any misunderstandings due to lack of back and forth communication.
To enhance the quality of D/V, we chose to iterate on the design of task.
This figure shows four iterations. Each prior iteration revealed some flaws that inspired
us to design new iteration.
In the first iteration, we assume that workers are familiar with classifying items into categories.
So we simply gave them a list of ideas and ask them to think of categories and sub-categories.
We chose to use the term--category and sub-category, because they may sound more familiar to workers,
compared to the term D/V.
We tested this set-up with 16 workers on Amazon Mechanical Turk (or MTurk), and received 43
dimensions and 156 values. Most dimensions were either invalid or irrelevant to the given
topic. For example. some dimensions are too narrow, such as the dimension “dangerous”
which can only be used to categorize a partial set of ideas in “ways to use a brick”.
On the contrary, some dimensions are overly broad, such as the word “usage”.
In the second iteration, we decomposed the original task into two sub-tasks to reduce
task complexity. Workers were asked to either provide dimensions or values, rather than
both. As shown in the figure, workers are first
given good and bad examples, then asked to provide values for three existing dimensions.
We tested this set-up with 10 workers on MTurk, and received 35 dimensions and 118 values.
For dimensions alone, we see a significant increase in valid dimensions.
However, if we consider D/V together, the “Valid D/V” is merely 28%, meaning that
a large portion of the values were not aligned with the corresponding dimension.
Here is the third iteration. In Part A, workers were given a list of ideas
as input and requested to think of a good question that can be used to ask other workers
about each item in the list. This is because a good question with good
choices is very similar to a valid pair of D/Vs.
In Part B, workers can see the preview of how their questions would be used to categorize
a list item. This will give them more context about how their input will impact other workers.
We tested this set-up with 36 workers on MTurk, and received 84 dimensions and 273 values.
Some good questions were generated, such as "What is the physical effort?" with three
choices {"Little", "Moderate", "Heavy"}. The valid rate of D/V significantly increases
to 44%, but it is still pretty low.
In this final iteration, we provided a structured mini-tutorial to guide workers through the
confusion of generating DV. The tutorial contains four steps. The first
step asked workers to categorize a small list of items by the given dimensions and values.
The second and third step asked them to enter values for given dimensions or enter dimensions
for given values. Since steps 1-3 have gradually demonstrated
all the necessary subskills for the actual D/V generation task.
In step 4, workers were given a list of uncategorized items and asked to think of dimensions, values
and finally categorize the items We tested this set-up with 30 workers on MTurk
and 13 workers passed the tutorial. We only allow workers who passed the tutorial
to proceed, which results in 49 dimensions and 200 values.
We found that the “Valid D/V” is significantly increased to 80%.
To summarize, we’ve tried five strategies in four iterations.
For more details, please check the paper. Thank you.