Hello.
I'm Sihang.
I'm very happy that I can present our work at CSCW 2020
on behalf of my co-authors, Ujwal Gadirajul and Alessandro
Bozzon.
The title of our work is "Estimating
Conversational Styles in Conversational Microtask
Crowdsourcing.
So before I introduce the methods we used in this work,
I would like to first explain the background, that is,
what is conversational microtask crowdsourcing?
We know that the tradition no-user
interface for microtask of crowdsourcing is the web page.
So on traditional web interfaces, users--
or we can call them crowdworkers--
they look at the task instructions, questions,
and then provide their answers using the web page
and the web-based elements.
However, recently, we have explored the feasibility
of using conversational interfaces for crowdsourcing.
So on the conversational interfaces,
workers receive task instructions and questions
as text-based messages from a chat bot.
and then we provide the answers by sending messages
to the chat bot.
What's more?
The conversational interface for crowdsourcing
has been shown to be effective for giving workers higher
satisfaction, better user engagement,
and it can help provide a similar quality
related outcomes.
However, since we are using conversational interfaces
to exploit the advantages of conversation,
there is one important point that
needs further exploration, which is the conversational style.
According to previous work in the field of psychology,
the conversational style can be classified
into two categories, namely, involvement style
and considerateness style.
And involvement style present that the person who is talking
wants to be close to the others in chat.
And considerateness style represent
that the person who is talking wants
to keep the distance from the others.
And if we apply this classification
into the text-based conversation in crowdsourcing,
we can have a coding scheme to estimate conversational styles
mechanism.
The coding scheme we designed has
five dimensions and corresponding criteria,
which are derived from [INAUDIBLE] previous work.
And we use this coding scheme to classify
workers' conversational styles into either involvement
or considerateness.
As to the experimental procedure,
[INAUDIBLE] is published our tasks on Amazon's Mechanical
Turk, and each worker would need to have
to pre-task a conversation before the actual task
of execution to get sufficient conversation
data for style-estimation.
Then workers need to complete five mandatory microtasks
and as many of 45 optional tasks as they wish.
And we used the number of answered optional tasks
to measure worker retention.
After that, it is the post-task survey.
Here, we use two questionnaires to measure
perceived the cognitive load and user engagement respectively.
And all the details can be found in our paper.
Here, we don't have time to report
the data and the results, but if you
are interested in the result analysis,
again, you can take a look at our full paper.
But now, let me briefly introduce the main findings.
So first, this coding scheme we proposed
has high inter-rater reliability.
We use [INAUDIBLE] to measure the reliability,
and the results show that it is sufficient to do
robust to account for the subjectivity
of different raters.
Second, we found that workers, with the involvement style,
can produce higher worker accuracy,
perceive higher user engagement, and feel less cognitive task
load while completing tasks, especially for difficult tasks.
Third, the above two findings can tell us
that the coding scheme could be an effective tool
for predicting worker performance
and assisting crowdsourcing task assignment.
For instance, we can assign relatively difficult tasks
to the workers of involvement styles.
And finally, we suggest that the future work
could focus on automatic conversational style
estimation and prediction.
And this is what's going on about our work.
Thank you very much.