Hello I'm Sihang. I'm from Delft University of Technology
and now I'm presenting our work Improving Worker Engagement Through Conversational Microtask Crowdsourcing
this is a joint work with Ujwal Gadiraju and Alessandro Bozzon
first of all the background of our study as we all know that traditional user interface
for microtask crowdsourcing is the web so all the questions of the
task are shown on the web page which makes it pretty like an exam then online works can
get tired and bored very easily therefore in previous studies
the conversation agent was introduced to the realm of microtask crowdsourcing
the conversational agent for microtasking can show all the essential elements of
a microtask, but in a dialogue based way so as we can see from the graph of comparison
by conversing with the conversational agent the worker can get
all the essential information such as task title, time limit, and task instructions
and questions and of course workers can provide their answers in a
conversational way as well well and the researchers have shown the evidence that
a conversation or agent has the capability of giving workers high satisfication and
it has similar execution time and similar output quality in comparison
with the traditional web interface however several questions remain
unexplored like how can the conversational interface affect worker engagement or
cognitive task load and what if we apply a conversational style to the
agent and how can conversation style affect the results so our research
questions of these studies are the first one is to what extent can conversational
agents improve the worker engagement in microtask crowdsourcing
and the second question is how do conversational agents with different
conversational styles affect the performance of workers and their
cognitive tasks load while completing tasks
so to address these research questions we first introduced conversational styles into our study
we want to apply different conversational styles on our
conversational agent according to the previous work of linguistic theories the
conversational styles can be classified into two categories namely
high involvement and the high considerateness so high involvement style basically
means the speaker want to keep close with the others and high considerateness style
means that the speaker want to keep distance with the others
and we create criteria to assign these two conversational styles to the agent and
this is how we apply conversational styles on the conversational agents
now let me explain how the conversational microtasking works
so the workflow of conversational microtasking follows four steps the first step is task instructions
the conversational interface in this step is initialized
with opening greetings like hi hello just greet with workers and then the
worker can respond by selecting one out of two options why do we do this I will
explain it later and then the conversational agent well tell the
worker about the basic information of the task such as task title the time limit
and task instructions so the goal of this step is twofold
the first goal is to make sure that users are familiar with the conversational
interface and the second goal is to estimate the
conversational style of the workers why do we do this because previous studies
have shown that style alignment which means aligning
conversational style of the agent with the style of the worker can somehow have
some positive effects only the output so we let workers respond to the agent by
selecting some options and then we use this way to estimate the conversational style
so this is our first step greetings and tasks instructions then
the second step is questions and the answers in this step the agent presents
the actual tasks to the worker and worker will answered to the agent
well the conversational interface supports two modes of input from workers
namely free text and multiple choices so if a task it's a free text tasks such as
CAPTCHA recognition and information finding the worker must type the answer in the
text area of the conversational interface but when a task includes
multiple choices answers such as image classification the worker can either
type the answer as free text or simply click the corresponding UI button and
then it comes to the third step - breaks after a worker has answered
five mandatory questions workers can stop answering and submit answers whenever
they want so if they choose to continue the conversational agent will give
workers optional questions to answer and there are 45 optional questions in
total and we use the number of answered optional question to measure the worker
retention and if worker keeps doing tasks and the
conversational interface will give breaks to the worker to relief them and
during the break the conversational agent sends a meme for amusement that is
something like this... and then it comes to the final step answer review and submission
so when a worker decide to stop task execution or there's no more
pending questions that means worker has answered all the mandatory questions and
optional questions the conversational agent will sent answer review to the workers
then worker can decide to go back to edit their answers or proceed to
submission and after the worker submit the answers a task token will be given
to the worker and the worker can use this to claim for their reward and this
is how the conversational microtasking works then we design experiments to
answer our research questions as for task types we choose two most common
input types which are free text and multiple choices and two most common
data types that text-based tasks and image based tasks which results in
four task types which are information finding CAPTCHA recognition
sentiment analysis and image classification in terms of the user interfaces we use web
interface and the conversational interface note that all functions that
the conversational interface has we mentioned before such as optional
questions breaks and memes the web interface has all of them as well so these two
interfaces are equivalent and it's fair to compare them what's more there are
three conditions of conversational interface which corresponds to three
conversational styles high involvement high considerateness
and style alignment and then it results in four interfaces - web interface
conversational interface with high involvement style
conversational interface with considerateness style and interface with style alignment
finally we have four task types we have four interfaces so we have sixteen
conditions and we recruited 50 unique workers for each condition then we have
800 workers in total and we ask workers to do the microtasks first and then we
give them a post task survey which contains the UES SF form and NASA TLX form
which are used to measure user engagement and the cognitive task load respectively
and then the evaluation in terms of the distribution
of conversational styles we found that workers tend to show different
conversational styles due to the effect of task complexity so for instance when
a task is more difficult workers tend to show considerateness style and when
a task is easier workers tend to show involvement style as for the output
quality we found that the conversational interfaces have no significant impact on
worker accuracies in comparison to the traditional web interface similarly we
did not found any significant difference in worker accuracies across different
conversational style and the results of output quality is aligned with the
previous study and as for the worker engagement we measured in two ways
the first way is worker retention which is measured by the number of answered
optional questions and the second way is using the UES SF form
to get the score in terms of the worker retention we found that
the conversational interfaces lead to significantly higher worker retention in
multiple-choice tasks compared to the web interface and the interesting thing
is an involvement style I mean a conversational interface with high
involvement style has significantly higher worker retention across all task
types compared to the web interface then as for the perceive the user engagement
and the cognitive tasks a load which are you UES SF and NASA TLX we found that the
input types and data types can significantly affect the you UES SF
score while the interface and conversational styles have no
significant impact on that and we found no significant difference in NASA TLX
score for measuring cognitive task load across different interfaces
so to have an overview of our results we found that the conversational interface
can significantly improve the worker engagement as for the output quality and
cognitive task load we found no significant differences then what can we
learn from these results first of all giving a conversational agent
a high involvement style can generally improve the worker retention in conversational
microtask crowdsourcing so if you want to use the conversational agent to
help you execute the task you can use a high involvement style
some people might ask the conversational agent cannot support too many tasks
our proposed conversational interface is purely HTML based so the elements use
the in traditional web interface can be easily ported into our conversational interface
then theoretically the conversational interface
can support all the task types that the traditional web can do
and by adopting conversational interfaces task requesters can improve
worker engagement and allowing workers to complete more work and earn more money
so we believe this can help foster good faith between requesters and workers and
it is good for their long-term relationship so our study has very
important implications in crowdsourcing marketplaces as some of you have might
noticed that the focus of this work is the conversational style of the agent
it's not the conversational style the worker but we still use a very simple
and naive way to estimate the conversational style in this study we found this is
very interesting to study the conversational style the worker so we
did study deeper on that and if you are also interested in that you can see our
CSCW work estimating conversational styles in conversational
microtask crowdsourcing the focus of that work will be the conversational style
of the workers and this is our work thank you for listening