Hello my name is Alexandru Placinta. I am a master's student at EPFL, in the Data
Science program, and in this presentation, I will talk about User Expectations of Conversational
Chatbots Based on Online Reviews. Throughout this presentation I will start the discussion with
the motivation, then I will briefly explain the structure of the study and the techniques we used.
Finally, I will conclude with the results and implications.
To begin, let's define what a chatbot is: by a chatbot we understand a machine agent that
interacts with the user via natural language. They are widely classified in two types: task-oriented
that help users achieve a specific goal and open domain that do not serve a specific functional
purpose, but try to become a virtual companion to the user by engaging in social conversations.
While task oriented is currently the most represented type of chatbot in industry,
open domain chatbots have started to emerge for mass-market platforms such as Google Play
or App Store just recently. It opens the opportunities to study user experience with
real chatbots instead of conducting in-the-lab experiments with prototypes. With this approach,
we aim to provide insights into how well the user expectations align with the current chatbots'
social capabilities and propose implications for designing more engaging experiences.
For the aim of our study, we decided to analyze user experience and expectations based on online
reviews of open domain chatbots taken from Google Play. First, we created a list of 16 sufficiently
diverse conversational chatbots for the analysis. Second, for the previously identified chatbots,
we collected the available user reviews, including textual feedback and the numerical
rating that the user assigned to it. Next, we applied some filtering procedures to
retain only the reviews that meaningfully reflected user thoughts about chatbots.
At this step, we split our analysis in two independent parts: experience and expectations.
Each part entailed its own steps which I will describe in more details in the upcoming slides.
Finally, the findings of these two steps allowed us to synthesize the design implications.
While studying user experience we focused on chatbot's conversational skills and social
qualities. Our main interest was to establish how these aspects influence user satisfaction.
To compromise resource constraints, we conducted the analysis using a representative sample of
collected reviews. To pull useful concepts out of the data we employed open coding approach.
Overall we produced four categories of codes, as follows: chatbot's social skills describe both
positive and negative aspects of the conversation experience, affective satisfaction describes the
sentiment expressed by the user in the review and serves as a complementary descriptor for the
numerical rating, personification specifies the way user refers to the chatbot in terms
of its name and pronoun usage, social role encompasses how the user perceives the chatbot.
After, we took two different directions: we performed a statistical test and a
regression analysis in order to study the effect of these social dimensions on user satisfaction.
Considering chatbots' social skills we perform the regression analysis for user sentiment and
star rating. In these plots, every bar is associated with the weight of a specific
issue or assets identifying its effect on user satisfaction. The models demonstrate
consistency in the effect of different features, validating the reliability of the results.
We can notice that issues describing chatbot's social skills have a negative
influence on user satisfaction while assets influence it positively. Intrusion into personal
space is deemed as the most detrimental property of the existing chatbots, while,
on the contrary their ability to keep company of the user is the most appreciated quality.
Coming to personification and social role, our findings indicate that these two factors
also play a dominant part in defining user satisfaction. Chatbots with a name and a
human-like role receive higher ratings compared to their counterparts without a specific identity.
Our next step was to study user expectations in order to evaluate how they align with user
experience. The analysis consisted of three steps: lexical filtering, coding, and thematic analysis.
Since not all reviews express expectations, we devised a lexical filtering pipeline in order
to retain only the appropriate reviews for further analysis. The process included identification of
keyword phrases that indicate expectations, topic modelling and heuristic n-gram filtering. Then,
we proceeded with the coding. During the process, a lot of reviews were discarded as
they expressed user wishes about the technical aspect rather than a social one. As a result,
260 user reviews remained. Finally, we conducted thematic analysis and in total we identified 26
codes that belong to eight larger themes. We present the emerged themes in the next slides.
One of the largest themes that emerged is that users expect more social involvement from their
chatbots. First of all, users expect chatbots to memorize information that they share, as people
typically do while developing a friendship. They also want chatbots to acquire and demonstrate
new knowledge to extend the range of topics for discussions and foster shared interests.
Furthermore, users wish their chatbots to advance their proactivity skills and get more involved
into the conversation. The next pronounced theme in our analysis is empathy. It encompasses both
the abilities to recognize the emotions of a person and express appropriate emotions
in return. Users expressed a need for chatbots to better understand their emotions given that
they frequently reach out to chatbots to release their negative thoughts and receive non-judgmental
support. It is especially critical for chatbots to accurately identify users' negative sentiments and
treat them carefully. Additionally, many reviews explicitly called for chatbots' ability to express
more emotions to diversify their unnaturally supportive behavior and appear more natural.
The final set of emerged themes describes the user desire for existing chatbots abilities to evolve
further. Specifically, many reviews in this set ask for their more distinctive personalities with
a specific persona. The next theme concerns the topic of politeness and social norms.
While at the beginning of user interaction with the chatbot, its rude and provocative behavior
would mostly probably hurt the user satisfaction, after establishing the social connection some
users prefer their chatbots to get more cheeky. Finally, last theme relates to the entertainment
aspect of the user experience: users want chatbots to keep developing their sense of humor
and engage in more advanced entertaining activities such as playing board games.
The presented analysis demonstrates that initial promising steps have been taken towards creating
natural open domain conversational agents. However, existing chatbots seem still incapable
to adhere to more advanced social protocols. To assess the discrepancy between user experience and
expectations we leveraged the PEACE model which defines four key qualities of conversational
chatbots: politeness, entertainment, attentive curiosity and empathy. We separately mapped the
codes describing the current user experience and expectations to the dimension of the PEACE
model and compared their distributions. The plots demonstrate that politeness and entertainment are
the only two social qualities broadly integrated into available chatbots, while attentive curiosity
and empathy comprise the most significant user expectations, which illustrates the existing
gap. Building on top of these findings, we synthesized the implications for the design
of future chatbots. The five guidelines read as follows: design politeness for trust building;
enhance entertainment to sustain engagement; add empathy to ensure trust maintenance;
learn to personalize; and assign chatbots' persona for greater acceptance.
To learn more about these implications we invite you to check the respective section in our paper.
To conclude, our work presented an overview of social skills of existing open domain chatbots
and their influence on user satisfaction, as well as elicited user expectations of future chatbots.
These findings allowed us to evaluate the gap between experience and expectations, and
produce the guidelines about how to endow future chatbots with greater social intelligence. Even
though in our study theoretical saturation was reached, we suggest that future works practicing
similar research approach should consider more robust coding process to cover a larger sample
of collected data. Furthermore, more attention should be paid to the fact that online reviews
data is subject to various biases and developing appropriate control mechanism would be beneficial.
Finally, we propose similar studies to be conducted for other platforms and users'
cultural background. Thank you for our attention and now we would be happy to take your questions.