Hello Everyone I’m Senjuti Dutta, and today I’m going to talk about our paper “Mobilizing crowdwork: A
systematic assessment of the mobile usability of hits.”
on behalf of myself and my co-authors.
Crowdwork on Mechanical Turk involves HITs or
“human intelligence tasks.
This on-demand information work seems to be built
for desktop environments only, despite a growing ownership of mobile devices.
For example, Pew Research Centre estimates that 85% of adults own a smartphone and 77%
own a laptop or desktop computer in the United States.Therefore
our goal is to support crowdwork on mobile devices in order to help more crowdworkers
join the platform who do not own laptops or desktops and also provide
others a choice to be apart from their workstations.Mobilizing crowdwork remains tedious because of a lack
of understanding about the mobile usability for
HITs.
Our paper develops a taxonomy of the characteristics that make HITs
suitable or unsuitable for mobile devices.
Our paper breaks down the taxonomy in three steps.
Step 1 design an initial taxonomy of mobile usability characteristics which is a Literature
Workshop.
Step 2 we use an Amazon Mechanical Turk survey to see whether the taxonomy aligns
with crowdworker experienced.
Step 3 we demonstrate it by scraping HITs and evaluating
them with the taxonomy, a HIT Corpus.
I’m going to talk more about each step in detail in the next
slides.
Step 1 Literature workshop is done through a targeted literature analysis to understand
characteristics of HITs.
We did literature workshop which has 4 stages.
stage 1 is Idea Generation, stage 2 is Idea Recording, stage 3
Idea Discussion and stage 4 Idea Deliberation.
This provided a way for us to be as comprehensive as scholars.
We found these papers for Off-site Knowledge Demands which are described in detail in the
paper.
Literature workshop came up with a set of seven characteristics.
The characteristics are divided into two types.
Task Characteristics and Interaction Characteristics.
Off-site Knowledge Demands from task characteristics characterizes
the underlying need to navigate away from a HIT’s primary task interface
for example, to find information on another webpage or make use of another web
resource in order to successfully complete it.
Multi-device Demands from interaction characteristics is a HIT’s
underlying need for the use of multiple devices in order to complete it.
You can read about each characteristics in detail
in the paper.
In step 2 we verify the mobile usability requirements of HITs in reality
we draw data collected from an online survey deployed on Amazon Mechanical Turk and recruited
a total of 111 participants for the study.
The survey began by inquiring about participants’ personal demographics and their work experience
on Amazon Mechanical Turk.
Our analysis specifically draws on three questions from two sections of
the survey which are Understanding HIT completion and The Magic Wand.
You can read about the questions in more detail in the paper.
The responses from the survey show the representation of
labels for the HIT types that are currently completed by crowdworkers alongside
the HIT types they believe they need further support.
For example, we find that Survey HITs are the most
prominent type of HITs currently completed on their smartphone according to
31.5% Participants.
Magic Wand responses show smartphone needs for Tools
and Scripts, Design and Compatibility and Task Interaction.
Design and Compatibility issues which was reported by 33% participants that saw HITs
”not made for smartphone use”.
Task interaction discussed by 19% participants saw
HITS which are ”slow on smartphone”.
Tools, Apps, and Scripts reported by 33 % participants ”would like an app to list
only available hits that are smartphone friendly”.
In step 3 we demonstrate the utility of our taxonomy by applying it
to the scraped HITs from Amazon MTurk.
We collected a total of 519 HITs which was used for qualitative coding to assess
the mobile usability characteristics of each HIT in our dataset.
During this coding we randomly selected 100 HITs from the dataset to label each characteristics
and 50.3% were visible and evaluated with the proposed taxonomy.
We have assessed mobile usability of each HIT in the scraped dataset not only with
task and interaction characteristics but also across HIT Types.
For example, Off-site Knowledge Demands from task characteristics 73.9% belong to
Good Usability as they do not require to navigate away from the main HIT interface,
while 26.1% belong to Poor Usability.
96.6% have Good Usability on smartphones as they belong to None,
while 3.4% belong to Poor Usability in case of Multi-device Demands.
In the case of HIT types Text Classification and
Survey will have the most usability if we consider total Good usability among HIT
types.
Our work provides insight on the state of mobile crowdwork.
It has lot of implications in the area of mobilizing crowdwork within and beyond smartphones.
In terms of practicing good mobile crowdwork it works as a mobile design
guide, evaluation tool, pathway to increase mobile-only access.
If this taxonomy works for smartphone will it work
for smartwatches and smart speaker?
There is a demand for mobile support for HITs reported by participants in the survey
that implies that there is a need for mobile tooling.
Thank you for your time.