Hello Everyone.
My name is Qingxiao Zheng, a doctoral student at the University of Illinois at Urbana-Champaign.
In this presentation, I will share our literature review of ACM papers that addressed user experience
research on conversational human-AI interaction.
There is extensive literature on conversational agents or chatbots, which are referred to
as CAs in the rest of my talk.
Early CA focused on dyadic human-AI interaction, which involves only one-to-one interactions
between humans and their CAs.
In recent years, polyadic human-AI interaction is gaining popularity.
Unlike dyadic interaction, polyadic interactions aim to support or mediate human-human interactions.
So, what motivates this study?
First, polyadic CAs are unique to design, they encompass a wide variety of complexities
that dyadic interaction may not encounter, for example, multi-party interactions, social
roles taking, group hierarchy, or social tension.
Second, most prior empirical studies and reviews focus on dyadic CAs, without addressing the
unique design challenges of polyadic CAs.
Therefore, in this literature review, we focused on examining how polyadic CAs are designed
and evaluated to fill the gap.
In particular, our research questions focused on understanding the fundamental challenges
of human-human interaction that are addressed by polyadic CAs, research interests between
dyadic and polyadic CAs, the practices of designing, the effects of using the polyadic
CAs, and the evaluation metrics.
Findings can help researchers and practitioners identify common practices, research gaps,
lessons learned, and promising areas for future advancements.
Our literature review has four stages, definition, search, selection, and analysis.
So first, in the definition stage, we developed a series of criteria.
We use them to include UX research on conversational human-AI interaction.
Such as, the articles are research papers with user studies; the major design feature
is conversation-based, and the CA interaction in the article is bidirectional.
Also, we identified data sources by piloting 200 papers from 5 different digital libraries.
We choose ACM DL because it features a wide selection of reliable HCI works.
We search the queries through ACM using CA synonyms, web-crawl the metadata and convert
PDFs of papers to build our database.
After removing duplicates and going through the paper screening, we identified 36 papers
that designed polyadic CAs and 135 that designed dyadic CAs.
We then used mixed-method and conducted thematic analysis and topic modeling to analyzed the
data.
Moving to the result.
We found that polyadic CAs were designed to resolve inefficient communication.
For example, because online teams are often less organized, reaching consensus can be
time-consuming, and discussions are hard to organize.
Also, inactive engagement was identified as a common issue.
Besides, building and maintaining social connections, and identifying common grounds can be challenging.
Polyadic CAs were proposed to address these challenges.
Through analyzing proven effects, we found that
First, polyadic CAs can help with consensus-reaching, aid communication comprehension, enhance task
management, and save the time and energy of human collaborators from tedious work.
Second, they benefit group dynamics by encouraging engagement and balancing uneven participation.
Third, they help in regulating team emotions by encouraging users’ positive attitudes.
Lastly, they can build balance and solidarity in the group by showing agreement, disagreement,
or bias toward one side(s).
We also look at polyadic CA's application domain, modality, characteristics of embodiments,
the originality of the design, etc.
Some aspects involve only multi-user interaction, which are unique to polyadic CAs.
We created a new aspect of “relationship type”, which was not found in prior work.
It contains sub-categories of co-learner, co-workers, co-players, first-Mets and etc.
We also synthesize the metrics researchers used to evaluate both polyadic and dyadic
CAs, and categorize them into:
chat log analysis or observation, survey scale, and interview.
Chatlog analysis use a conversation log or code user behaviors through watching recorded
videos.
These metrics reflect language patterns, appear in connected speech, and are related to conversation
structure and flow, task fulfillment and model training, user perceptions, and behaviors.
For survey scale metrics,
They evaluate user perceptions on CAs's dialogue management, adherence to social protocols,
system usability, and UX.
For interview analysis,
Users were interviewed regarding their general impressions on the CAs, perceived CA capability
in handling requests, CA personality; and their behaviors and efforts used.
Also, some overlooked issues we found can be inspiring for future design guidelines.
They relate to the appropriateness, privacy, and ethics in the designs of CAs, which warrant
deeper discussions about the role of polyadic CAs, their social influence, and their relationship
with human users in different group settings.
Overall, we find a very small fraction of CAs, published in ACM venues, designed for
polyadic human-AI interaction.
This literature review map the progress in this field for future researchers.
We also bring up the social boundary issues.
Boundaries in social sciences can be understood as a set of rules followed by most people
in a particular society, which are vital in society because they can guide human behaviors
and assist in managing what is and what is not acceptable.
This issue is closely related to privacy and disclosure, as the presence of polyadic CAs
as social actors changes the social dynamics and users’ information management strategies.
We also discuss a research agenda for future communicative AI technologies.
For example, to advance relational dynamics, which reflects the ways people interact with
technologies, with themselves, and with other people or groups of people.
We suggest that polyadic CAs can be further investigated in their effectiveness to foster
new relational dynamics.
Finally, I would like to thank my fabulous collaborators Yiliu, Yiren, Weizi, and Yun.
On behalf of my team, I would also like to thank our funding agencies, the National Science
Foundation and the IBM-ILLINOIS Center for Cognitive Computing Systems Research, a research
collaboration as part of the IBM AI Horizons Network.
Thank you!