Hello everyone, I am Deepali Aneja and today I am going to present our work on Understanding
conversational and expressive style in a multimodal embodied conversational agent.
Embodied Conversational Agents are interactive virtual characters that combine facial
expressions, speech, and gestures to enable face-to-face communication with the users.
Most of these agents have relied upon scripted dialogue or prescribed intents that had to
be pre-trained, limiting the opportunities for less constrained interactions.
While the state-of-the-art embodied conversational agents have proven useful in
limited domains, none of them have been developed with an eye towards open-ended conversations across a wide variety of
domains or for general productivity tasks. Style matching has numerous benefits that
help interpersonal interactions proceed more smoothly and efficiently. For understanding
human conversational style, Deborah Tannen categorizes
conversational style into high consideration (HC) and high involvement (HI).
The high consideration style emphasizes consideration, independence, and is characterized
by longer pauses whereas high involvement style
emphasizes interpersonal involvement, understanding, and is
characterized by faster with shorter pauses between conversational turns.
This brings us to our first research question - Can we design an embodied agent that is capable of relatively open-ended conversations
and automatically aligns its response to conversational and expressive style of the user?
We present SIVA (Socially Intelligent Virtual Agent), an expressive, embodied
conversational agent that can recognize human behavior during open-ended conversations and
automatically align its response to the style of the user.
In our work, we combine both - the conversational and the expressive style
of the user to generate agent's style aligned behavior.
The design of SIVA is truly multimodal, allowing for the capture of audio-visual features
and synthesis of engaging, face-to-face conversations with a human.
Here we show an overview of SIVA's architecture. We built components to sense and analyze the
audio signal from the user for conversational style matching and generating the lip sync,
performing text sentiment analysis for extracting the contextual information,
and sensing and analyzing the user's video feed for expressive style matching.
Finally, the outputs from audio and video pipelines are composited together
to synthesize SIVA’s conversational and expressive style aligned behavior.
SIVA performs style matching based on the user’s interaction style
during the conversation using the listed style variables.
Our second research question is -
Do people connect better with a style aligned agent
that uses style matching, as opposed to one that does not?
To investigate the effect of conversational and expressive style matching on users' perceptions of
the system, we conducted a user study with 30 participants comparing two interaction
conditions - participants interacting with SIVA style matching (experimental) and others interacting with SIVA with no style matching (control).
Our results show that the participants rated the style matching condition to be statistically significantly
more reflective of their emotions compared to the control condition. The results also showed that
the participants perceived SIVA to exhibit more convincing non-verbal behavior for the
style matching condition compared to the control condition in a statistically significant manner.
Our next research question is - How style matching influences the perception of the agent?
For evaluating agent’s impressions, we found significant interaction for perception of Animacy.
high involved conversational style group found the style matching more lively and natural
whereas high considerate conversational style participants paid more attention to the verbal
communication and perceived the control condition to be more natural
Now we show an interaction demo from our user
study where the participant is performing one of the tasks.
We presented an embodied conversational agents that is capable of longer conversations with a user,
examined how multimodal conversational and expressive style matching
influences a users’ perception of the emotional behavior and connection with the agent,
and demonstrated how style matching influences the perception of the agent.
We found that the agent was considered more animate by people who preferred
interpersonal involvement and less animate by people who preferred independence.
Thank you.