everyone my name is Nalan
and today I am going to present this
paper named teachable conversational
agents for crowd work FX and performance
and Trust
this work was completed at the
University of Waterloo with Jocelyn go
and Professor Edith law
so our work is mostly motivated by an
intriguing quest of fostering human AI
collaboration through teachable agents
interestingly this particular area has
been explored across multiple research
disciplines such as computer science
educational research and psychology
for instance for past several years
computer science researchers have worked
on areas like interactive machine
learning and machine teaching that
involves humans in the process of
welding machine learning systems
likewise educational researchers have
extensively leveraged of psychological
phenomena called Protege effect that
suggests that teaching others can be
more effective than self-work
this basically implies that teaching a
task has greater benefits than
individually performing the same task
so the primary question that we aim to
address through our work relates to the
feasibility of combining these two
approaches and examining the efficacy of
teachable agents in the context of
crowdsourcing
our focus on previous work is
essentially pivoted around the research
on teacher learning interaction
techniques to improve crowd workers
performance and studies related to
concepts of trust between human computer
collaboration
teacher learner interaction is a
well-studied area across multiple
disciplines that focuses on the mutual
impact of teachers and Learners on each
other
computer science and machine learning
communities have explored this under
interactive machine learning techniques
such as active learning and machine
teaching
Active Learning aims to improve machine
Learners performance by actively
querying a human Oracle and obtain
labels
and machine teaching seeks to create
optimal training set from humans that
can efficiently guide a learning
algorithm
While most of the computer science
researchers study teacher learning
interactions to optimize the training
outcomes for learners educational
researchers explore this interaction
form
from another perspective which is based
on learning by teaching techniques such
as reciprocal teaching and peer assisted
tutoring
under this theme several studies have
demonstrated the effectiveness of
protege effect or learning by teaching
for cognitive metacognitive and motor
learning skills that are evolved from
the expectation to teach others
now this theme of improving performance
is also very common to Outsourcing
research where researchers have explored
techniques to argument crowd workers
performance in human intelligence tasks
this has been investigated through
monetary techniques such as increasing
the magnitude of reward perceived
fairness in pay as well as non-money
tree techniques such as implicit or
explicit training peer review and use of
intrinsic motivational drivers such as
curiosity and interaction with chatbots
but in any form of interaction one of
the most fundamental component is the
element of trust between the entities
which has been extensively studied in
human AI interaction
Roman and all found that adding visual
presence to an agent on a shopping
website increases participants Trust
likewise wanganol found a transparency
of a system increases cognitive trust
and lemon all found that trust can also
be improved by providing different types
of explanations
while all of this forms the backbone of
our current work it is still different
in the sense that we explored teacher
learner interaction in the context of
crowdsourcing
we validate whether this interaction
technique can be used to improve crowd
workers performance
and finally we substantiate whether this
influences the Dynamics of trust between
human computer interaction
now we achieve all of this through a
modified crowdsourcing workflow where
crowd workers teach human intelligence
tasks to a teachable conversational
agents
through conversational interaction
this is different from traditional
crowdsourcing workflow where crowd
workers perform the tasks themselves in
exchange for monetary compensation from
requesters
so we performed two experiments the
first one was focused on validating the
effect of learning by teaching in the
context of crowdsourcing
for this we recruited 100 participants
for a between subjects crowdsource and
study half of them were asked to perform
a class text classification task
themselves another half were asked to
teach text classification to an agent
all participants performed a
pre-interaction and post interaction
Baseline task before and after their
respective experimental conditions
towards the end they also failed an IMI
activity perception questionnaire
estimated completion time for the entire
experiment was 20 minutes and all
participants were compensated well above
the U.S federal minimum wage
so the results of the experiment 1 were
quite encouraging
it was found that participants who
taught the task to their agent exchanged
more information and performed better in
the post interaction tasks than those
who perform the task individually
further results from IMI activity
perception questionnaire revealed that
participants who identified themselves
as teachers found the task more useful
and valuable than those who identified
themselves as mere annotators
in the second experiment we investigated
the Dynamics of trust within teacher
learner interaction between crowd
workers and the agent
for this we recruited 80 participants
for another between subjects
crowdsourcing study where half of the
participants start their own agent
during the task and other half
interacted with a pre-dot agent
after the respective experimental
condition participants were presented
with a bonus task involving more text
classifications
participants could either skip the bonus
task entirely or select a portion of the
task for the agent as well as for
themselves
in the end all participants failed a
trust measuring questionnaire
estimated completion time for this
experiment was also 20 minutes and all
participants were compensated well above
the U.S federal minimum wage
so the results of second experiments
were also very intriguing we found
that participants who taught their own
agent delegated more tasks to the
teachable agent as compared to the
participants who interacted with a
pre-trained agent
finally after reviewing the trust
measuring questionnaire we found that
participants who taught their own agent
reported more Trust on automated systems
and were more confident in themselves as
their own ability to do the same task
overall this work demonstrates that
teachable agents are useful in
crowdsourcing and improves trust in
human AI collaboration
all the supplementary data related to
this work is available on the project
home page
for additional details and experimental
results
please see the accompanying paper