Hello everyone and thank you for attending this talk.
On behalf of my co-authors, Jason, John, and Per Ola, I am Stephen, and I am here to present our
work on investigating the accessibility of crowd work tasks on Amazon Mechanical Turk.
Mechanical Turk, or AMT, is a crowd work platform developed by Amazon, in 2005, that allows
microtasks, also known as Human Intelligence Tasks, or HITs, to be completed by crowd workers.
Key advantages of crowd work include the ability to work from home on relatively
small tasks and at any time and place with an internet connection.
Given these benefits of crowd work, it can be attractive to people who identify
as having a disability, as people can work at their own pace and with
any potential assistive devices that they use on their personal computers.
However few works have investigated accessibility issues in crowd work
and how these may be overcome.
In our study, we had three main objectives: 1. Investigate the demographics of crowd workers in
the U.S who identify as having a disability, and use the AMT platform. 2. Gain insights
into the factors that hinder the effective completion of crowd work tasks. 3. Identify how
accessibility issues may be alleviated in order to encourage effective participation in crowd work.
To achieve these objectives we carried out two surveys and follow-up interviews with AMT workers.
In Survey 1, we surveyed 1 000 participants using a HIT on AMT
to gather basic demographics of users. In Survey 2, we surveyed only respondents from
Survey 1, who identified as having a disability. We then conducted interviews with 5 participants
from survey 2 to gain further insights into accessibility issues in crowd work.
Here are the results on the number of people that identify
as having at least one disability in Survey 1. The highest number
were from people who had problems with dexterity when using a mouse or keyboard.
Here the findings on age, race, gender, and education distribution.
Similar to previous work on AMT, we find a relatively young and educated demographic.
Further investigations in Survey 1 revealed that a majority of people are employed either on a
full-time or part-time basis and that most of the respondents work between five and 35 hours a week.
Although there are a variety of HITs that are completed on AMT,
our findings show that surveys are the most completed type of HIT.
In Survey 2 we found that a high number of respondents use crowd work because it
allows them to work from home, although other key reasons exist, such as: earning a living,
supplemental income. and just enjoying doing these tasks. Regarding time to complete HITs,
this has been identified as an issue in previous research.
In our work, it was a key issue for 43 of the respondents. Even for people without disabilities,
time can be a problem but the problem can be exacerbated by existing disabilities.
We can see the number of people in each disability category that reported having difficulties
completing HITs on AMT, specifically as a result of their disability. In all cases, the number is
close to half, which shows a high unfulfilled need for many participants with disabilities.
We cannot cover all of the issues that we found in this presentation,
but please refer to the paper for information on perceived anxiety and depression, qualifications,
wages and income, goals, UI issues, hits performed, and hits avoided due to disabilities.
We discuss improving accessibility in greater detail in the paper. Here are some key design
implications that follow from Survey 2 and the interview discussion. 1. To mitigate the
risk of overloading the UI, information can be contextualized based on user preferences and
relevance. Metadata can be used to indicate the kinds of abilities that are suitable for HITS.
This could work in the case of popular HITs, for which it is obvious that certain abilities are
essential. 2. AMT workers rely on external tools and scripts to assist them in managing HITs. For
example, finding and tracking reasonable HITs that suit certain abilities from reputable requesters.
This functionality can be implemented in crowd work platforms to assist users, especially those
with disabilities. 3. The participants identified potential barriers to usability such as presenting
too much information and using relatively small interface elements; for instance, buttons and
hyperlinks. Platform holders should ensure revised UIs based on user testing. 4. Communities offer
essential support to AMT workers by highlighting useful information on available tools,
tips, and social benefits. Platform holders can promote and recommend communities,
such as Turker Nation and MTurk Suite, especially to new workers. Furthermore, communities can,
perhaps, make more of an effort to reach out to new workers or workers with disabilities.
That concludes our presentation. We would like to
thank you all for listening and welcome any questions.