Hello and welcome to this talk on our work
titled expressive auditory gestures
in a voice based pedagogical agent
this work was conducted by myself Jessy Ceha
and Edith Law at the University of Waterloo in
Canada
conversational agents are systems designed to
communicate with humans through natural language
in education they provide a unique opportunity
to encourage interaction which is considered to be
important for the learning process
over the years conversational agents for
education or pedagogical agents
have been designed for learners of different age groups and
can be distinguished along various dimensions including
subject matter such as science and language
learning form and appearance
text based virtual physical
and also their role either as teacher agents
co-learners or teachable agents
in this work we focus on teachable agents
systems that learn with and are taught
by the learner based on the concept of
learning by teaching in contrast to
a teacher agent teachable agents
or tutee struggle with the material
and make errors which prompts the human
learner or tutor to pay more
attention reflect on misconceptions and
elaborate on explanations
the overarching goal of all pedagogical agents are
to elicit learning outcomes which can be broken
down into cognitive skill based and
affective outcomes
motivation to be moved to do something
is an especially important affective outcome because
it can influence what when and how we
learn teachable agents present a
unique opportunity for exploring
motivation because evidence suggests that
when learners taking on the role of the tutor
have a better relationship with their agent
are more motivated to put in effort to
teach their agent and as a result also
learn more
so we wanted to investigate how to strengthen the
human agent relationship so as to
enhance motivation in learners
in order to build and strengthen
relationships research on
conversational agents suggests the
effectiveness of expressing affect or
feelings moods and emotions
in human to human conversation a form of
vocal gesture known as interjections are
used to communicate a speakers emotions and
attitudes and research suggests
that they can strengthen relational bonds between
humans and computers now in
contrast to humans agents can
express their emotions and attitudes through
interjections but also other types of
sounds
one expressive type of sound is music
and musical expressions have a number of similarities
with the human voice in conveying internal
states some work has
investigated the addition of musical expressions and
other sounds to conversational agent speech
but there is not a lot of research yet about adding
interjections or musical expressions to
pedagogical agents and the impacts on
relationship building and learning outcomes
therefore we designed a study in which we systematically
added interjections and this language
independent counterpart musical
expressions to the speech of a voice
based teachable agent
we built an application which allowed
participants to teach this agent about various
rock types the interface
provided to participants contained
information about rocks on the left hand side
and they could also see a rocks worksheet on the
right which the agent needed to fill
in the agent had some knowledge of the
topic but needed help from participants to
confirm or correct its prior knowledge and
also learn new knowledge
participants used the interface to read through the
information and communicate with the agent by
pressing on the button in the top right corner to
record and send voice messages
the agent responded to the participant and filled in
the worksheet the green text as it was
taught
the agents responses were controlled by a human
operator using the wizard of oz technique
with a set of predefined statements to reduce the
system response time and maintain a similar
conversation across participants
thirty eight participants took part in the study
we used a between subject experimental design with
participants being randomly placed into one of
three conditions interjections
music or control which had no
added auditory gestures for the
voice of the agent as well as interjections we
used a company called CereProc and
musical sounds were taken from the musical
emotional bursts dataset
throughout the interaction the agent made both correct and
incorrect statements about the topic being
taught and the statements led to
moments where the agent replied with an interjection
or musical expression of either positive or
negative valence depending on the context and
condition
we collected data through a number of questionnaires as
well as the conversations between participant and agent
we were focused on three main things the human
agent relationship in terms of rapport
and perceived quality of the interaction
learning outcomes specifically motivation
and recall of material and cognitive
workload
so on to the results first we found that
the participants expressed perceiving both the
interjections and the musical expressions as
intended to express moments of positive and
negative valence and in terms of the
perceived relationship there was a
statistically significant difference in rating of
emotional rapport with the interjections leading
to feeling more of a sense of emotional
connection with that agent than in the
control
in terms of both being intrinsically and
extrinsically motivated to make the effort to teach
the agent participants in the interjections
condition reported on average significantly
more motivation than those in the control
condition
in terms of learning we found that condition had no
significant effect on change in quiz score from pre
to post interaction and lastly we found
that overall workload was not
significantly affected by interjections or
music and neither were any individual
workload dimensions
in conclusion where previous studies have
focused largely on expressions through
facial gestural and utterance level
cues this work explored the lesser
researched modality of auditory gestures
and looked at both language dependent
interjections and language independent
musical expressions as
interjections are highly prevalent in human to human
dialogue and are becoming more widely
available in popular systems for example
speech cons in the Amazon Alexa
the results provide evidence for the influence of
interjections on human agent relationship
building we found that expressive
interjections and musical expressions can be
used to convey moments of positive and negative
affective state in a voice based
agent and we found evidence for
interjections leading to the feeling of a
significantly stronger emotional connection with
the agent and finally
interjections appear to be particularly effective
at positively influencing both intrinsic
and extrinsic motivation in learners
without having detrimental effects to cognitive
workload
the results provide insights for voice system
designers in education as well as across other
domains including healthcare
entertainment and customer service for
example in which building rapport and
enhancing motivation can be of similar value
thank you for your attention and please don't hesitate
to get in contact if you have any
questions