Today, I'm excited to talk to you
about my work that strike with Brian McInnis on how
we write with crowds.
Crowdsourcing research has commonly
used writing tasks as a way to examine
the potential for systems to accomplish
creative and complex work by coordinating
a crowd of people performing short and simple tasks.
We saw the first example of writing
with the crowd in Turkish back in 2010,
when the crowd collectively wrote paragraphs together.
You had less is known about the range
of designs decisions associated with crowd writing systems.
This paper reflects two years of work
looking to enumerate and analyze the set of design decisions
that researchers make when writing with the crowd.
Specifically, myself and Brian wanted
to understand the breadth of designs
for systems which generate original written
content with the crowd.
To do so, we first performed a literature review
to understand the space.
And after an iterative process arrived at 24 papers,
we felt describe the area of code writing systems
from CHI, UIST, CSCW, and HCOMP.
To better understand the range of designs as well as the area
and more broadly, we also performed 18 interviews
with authors from 21 of the 24 papers.
These interviews shed light on the process of design,
and how the authors view their work within the wider
community.
With this set of data in hand, we needed to make sense of it.
With writing, one of the challenges
can be the variety of genres and specific writing tasks
that systems focus on from explaining
a meme to describing a new urban design to writing a novel.
We chose an analytical lens to facilitate our analysis
in the form of flower and hayes's, a cognitive process
theory of writing from 1981.
This model outlines a general approach
to writing using a number of different components,
such as reviewing, the writer's long-term memory,
and the text produced thus far in the writing process.
Although this model was developed
as a representation of an individual writer,
we find in our work that it functions surprisingly well
as a map for the designs of how we write with the crowd.
In fact, our analysis suggests that there
are design decisions related to every component
of the cognitive process model in the crowd writing
design space.
We direct you to the paper for the specifics
of these findings, but now, I'd like to highlight
some of our overall insights.
Our analysis suggests that there is more work to be done.
A number of components of the cognitive process model
are under explored in comparison to others.
There is a significant amount of existing work
on coordinating workers as part of a crowd writing system.
But significantly, less about actually helping them
construct a specific sentence or paragraph.
Our analysis tells us that designers
build systems to focus on a few key components of writing.
For instance, a system focused on providing design feedback
to students concentrates on the feedback produced thus far,
and the students as the ultimate audience for the writing.
Finally, our analysis sheds light on the design decisions
underlying crowd rating systems, which can not be fully captured
by the cognitive process model.
Workers in a crowd writing system,
do not have the same level of control over their writing,
as someone writing alone or a collaborative group writing
together.
Different systems provide workers
with different amounts of agency,
limiting them in some cases towards efficiency
or letting them have more agency to leverage
their individual skills.
Some systems also restrict communication
between workers and those requesting the writing,
as well as between workers themselves.
Ultimately, workers in crowd writing systems
are distributed collaborative groups with minimal control.
And future work should both explore this further
and take it into account.
There are numerous practical considerations
that our analysis sheds light on.
For instance, there is an overemphasis
in existing work on using Mechanical Turk for recruiting
and on building custom web UIs for presenting tasks.
We need as a community.
We're focused on reproducible reusable artifacts
for crowd writing.
This also extends to our need for better evaluation metrics.
Although the systems we studies show
that we can write with crowds, we
need to collectively develop more general metrics
for comparing writing quality, and setting cross system goals.
To conclude, we present a record of how we write with crowds,
and hope it can serve as a guide for those
in the future crafting their own crowd writing systems.
Thank you.