Hello, I'm Rachel Simons, and I'm presenting our paper
"I Hope This is Helpful--
Understanding Crowdworkers' Challenges and Motivations
for an Image Description Task."
First, some background on our paper.
Current AI researchers are seeking
to provide greater access for people
who are blind to the multitude of visual information sources
that we encounter in our daily life.
These captioning algorithms are trained on large datasets
of captions in order to generate novel captions for images
that they encounter.
However, most of these data sets of captions
are created using crowdsourcing.
And furthermore, many of these captioning tasks
actually use the same basic crowdsourcing task design that
has been used for many years.
And most of the images that are used in these tasks
are actually images that have been
taken by professional and sighted photographers
and taken from prefiltered sources online.
There's little research consensus
on what actually makes a good caption, especially
for people who are blind.
And there have been very few studies on workers' experiences
with captioning tasks, as well as very few studies
on workers' understanding of the captioning needs
for people who are blind.
So for our study, we were interested in better
understanding crowdworkers' motivations and challenges
in performing a captioning task, specifically a task
that we put on Amazon Mechanical Turk in which workers were
asked to describe images that were actually taken by people
who are blind and were told that they would be explicitly
contributing to the development of AI
algorithms to support people who are blind.
At the end of this task, we asked workers
to provide optional comments.
We then thematically analyzed over
1,000 of those comments collected from 577 workers.
For a brief overview of some of the discussion
points in our paper relating to task design,
we found that the common task design
that we did base our tasks on is not optimized for blind users'
images and that workers needed to use more image manipulation
features and have more control of those features
due to the quality issues of those images.
We also found that workers needed
to describe other types of content, such as numbers
using decimal points and trying to decide what was
important text within an image.
We, therefore, argued that the task design
must evolve in order to better support
diverse kinds of image content.
We also found that workers appeared to struggle with
the open-ended nature of the task design--
entering captions into a free-form text box.
So we do propose that other tasks
may want to give specific examples of captions,
to provide even checklists of things
to include in the captions, or potentially
vignettes or stories about blind users' experiences
with captions.
However, we do also argue that more research
is needed on exactly what makes a good caption
and the relationship of a good caption
to the subjectivity of both the captioner
and the user of the caption.
Related to generating captions for blind users specifically,
we found that workers care very much about creating
good captions for people who are blind.
However, they did not always know
or frequently did not know what types of information
to include, such as colors, relational context
between objects, the vantage point of the photographer,
and things like inferred knowledge that they may have.
So we argue that future research should investigate
what blind users would actually like in their captions to help
better guide the workers.
We also found that workers were very highly
motivated by wanting to help blind people.
However, it was very unclear if they fully
understood the exact impacts of their work
or what they were trying to do in terms of helping people.
But we got a lot of comments such as, I
hope this was helpful.
So we do argue that future tasks should
clarify the process and the impacts of this kind of work.
For example, are workers contributing
to research about captions versus providing captions
directly to people?
And then what are the impacts of AI?
We found in terms of engaging crowdworkers
that workers had various levels of experience
with people who are blind, including
acquaintances, friends, family, and personal experiences.
Many workers offered to do follow-up work
and even gave their contact info.
So we do argue that future studies could perhaps
mobilize crowdworking platforms better to recruit allies
and contributors for accessibility research.
And finally, to sum up, we found that the worker comments
helped to improve our task design
both during the actual task run and then afterwards,
when we now analyze the comments.
And we do feel that they offered meaningful insight
into workers' experiences with the task.
So we argue that future tasks should collect and publicly
share comments from workers as a matter of course.
And these are some acknowledgments.
And thank you for listening.