Hello, everyone.
I'm Daniel Hettiachchi from the University of Melbourne
presenting our work "CrowdCog--
A Cognitive Skill-based System for Heterogeneous Task
Assignment and Recommendation in Crowdsourcing."
Let's start by looking at data quality
in crowdsourcing, which is an important research question.
Researchers have studied data quality issues
from different dimensions such as work attributes, platform
and requesters, and data- and post-processing techniques.
In our work, we explore the promising direction
of using cognitive skills of workers.
In particular, we want to achieve
task assignment or recommendation
for heterogeneous tasks or different types of tasks.
There is limited prior work on this space,
while many methods and frameworks
focus on question assignment.
Our work is built on the concept of executive functions
that describes human cognitive ability.
We focus on three primary executive functions
named inhibition control, cognitive flexibility,
and working memory.
We build on our prior work, which
highlights the relationship between cognitive tests
and crowdsourcing tasks.
To match workers with crowdsourcing tasks
based on cognitive test outcomes and to test
if task matching works in a real-world environment,
we developed CrowdCog.
It is an online system that works
on top of standard crowdsourcing platforms.
Here you can see an overview of the task flow
in CrowdCog deployment.
An important fact here is that our system
aims to identify a suitable task for each worker using
a minimum number of cognitive tests.
Let's examine the flow starting with a worker attempting a task
that is connected to CrowdCog.
Please note that we also use several parameters,
such as number of cognitive tests
to be included in each hit.
And I'm not detailing all of these parameters right now.
First, we start with a simple check
to see if we still need answers for our tasks.
Then we assign new cognitive tests if available
and predict the task performance for candidates using the test
outcomes.
In CrowdCog-Assign, we directly assign a task
based on the predictions.
But in CrowdCog-Recommend, we show the task list
highlighting recommended tasks.
Workers can still select the tasks they prefer.
Based on the requester or platform preference,
we can either use CrowdCog-Assign or
CrowdCog-Recommend.
We use five different cognitive tests
relating to three executive functions and, also, four
different crowdsourcing tasks that
are commonly available in commercial crowdsourcing
platforms.
Our system was seamlessly integrated with Amazon
Mechanical Turk platform.
And we deployed our experiments under five different settings
were 574 workers completed nearly 1,000 hits in total.
We compared our system with a baseline strategy that
is comparable to typical assignment
methods in current commercial crowdsourcing
platforms, a state-of-art question assignment
strategy, and also a standard history-based method
where we filtered workers based on the number of tasks
completed and the approval rate.
In CrowdCog-Recommend, we see that workers
are more likely to work on a recommended task
and also produce high-quality work when they do so.
In CrowdCog-Assign, one key observation
is that 72% of the workers were assigned to at least one
of the tasks when they completed all five cognitive tests.
Then we compare our methods with other experimental conditions.
From Kruskal-Wallis ranks on test,
we report a significant difference in accuracy
among five conditions.
Also, from our post hoc tests, we
show that the accuracies in both CrowdCog-Assign
and CrowdCog-Recommend are significantly higher
than the baseline accuracy.
You can also see the accuracy difference between conditions
across different tasks.
In our paper, we discuss in detail
how CrowdCog can be implemented in a crowdsourcing platform.
We also discuss how we can repeat
tests, how we can find the mapping between new tasks
and the cognitive tests.
We also look at why it is important to use task
assignment as opposed to a question assignment and also
why it is valuable to consider task recommendation.
That's a brief overview of our paper,
and I would like you all to read our paper for more details.
We have also provided the source code for our system
in supplementary material.
I am, finally, a PhD candidate at the University of Melbourne
and an applied scientist intern at Amazon Web Services.
Feel free to contact me through my email address.
And also, you can get in touch by my website.
Thank you.