hi and welcome to my talk my name is Sarah Theres Völkel and I'm going to present
our CHI 2020 paper on developing a personality model for speech-based
conversational agents using the psycho- lexical approach. Of course I didn't work
on this project alone but this was a collaborative effort by this great team
that you can see over here. Speech-based conversational agents have become
increasingly popular often presented as helpful assistants in everyday tasks.
Due to their intended use and conversational nature this type of user
interface seems much more likely to be seen as a "being with a personality". For
example Siri sometimes gives funny answers or Google assistant uses filler
sounds such as "mhm". Hence users may unconsciously attribute personality
traits to voice assistance like friendly, helpful or funny. According to previous
research, acceptance and credibility of voice assistance are determined by their
abilities to be perceived as having a consistent and coherent personality.
However, systematically designing agent personalities remains a major challenge.
As of today no dedicated personality model for speech-based conversational
agents exists. Thus, most researchers have turned to the Big Five personality
taxonomy. The Big Five taxonomy is the most predominant model for describing
human personality in psychology. This model comprises five dimensions:
Openness is related to seeking new experiences, creativity, and novel stimuli.
Conscientiousness describes a tendency to be thorough, organised, and careful.
Extraversion is characterised by being outgoing, active, and social. Individuals
who score high and agreeableness tend to be cooperative, socially, harmonic, and
modest. Emotionally unstable or neurotic individuals easily feel anxious,
depressed, and stressed. Since this model was developed for humans it remains
unclear if it is actually suited to describe agent personality as well. For
example, the Big Five dimension of openness might be less applicable or
important for agents. Moreover, further dimensions beyond those in human
models might be necessary to sufficiently describe agents personality.
For example, in previous research people use adjectives like logical,
robotic, or technical to describe a voice assistant personality. We addressed this
gap with the first systematic analysis of personality dimensions dedicated to
speech-based conversational agents. We see this as an initial step to
developing a personality model for such agents. To do so we follow the psycho-
lexical approach which was also used to derive the established Big Five
personality model in psychology. The lexical hypothesis assumes that since
people notice and talk about individual differences, these individual differences
will eventually become encoded into the language. The psychological approach
consists of two steps: first the item pool generation in which unique phrases
for describing personality are collected. This is then followed by an exploratory
factor analysis, which explores the structure and relationship between the
items. Determining the optimal procedure for compiling a representative set of
descriptors from the entire lingual lexicon represents a major challenge of
the psycho-lexical approach. For example Norman identified 27,000 personality
terms in the dictionary. Researchers have then reduced this number of descriptors
by successively excluding unfamiliar, slangy, or redundant terms based on
expert and empirical ratings. We developed a new approach to generate the
item pool for our model. Inspired by traditional test construction theory we
use multiple methods to collect potential descriptors. In an online
survey we asked 135 participants to name descriptors for a chosen voice assistant
in a free description task. This resulted in 228 unique descriptors. In a lab
experiment, 30 people interacted with three popular voice assistants, namely
Google Assistant, Siri, and Alexa and afterwards described their personality in
semi-structured interviews. This resulted in 176 unique descriptors. In
order to also include implicit depictions of personality in everyday
language use we crawled the latest Google Playstore reviews for
Cortana, Alexa, and Google Assistant. We analysed these 30,000 reviews by
combining natural language text analytics with manual analysis.This then
yielded 383 unique descriptors. Finally we merged the three descriptor sets
together. The descriptors collected in the three different methods show only small
overlaps. While 28 descriptors were named in all three methods 493 descriptors
were only found in one of the three approaches. We also accounted for the
possibility that traditional human personality descriptors may be suitable
to describe speech-based conversational agents. Hence we merged our set with
Goldberg's established a list of 339 adjectives for
human personality. The resulting list comprised 870 descriptors. Of course
870 descriptors are way too many to ask participants to rate them in a survey.
Therefore we refined this list in several iterations. First we applied establish
exclusion criteria, such as eliminating words which are slang, difficult,
ambiguous, over-evaluating or linked to gender or physical characteristics. In
case of lexical opposites, for example unfunny or unkind,
we only include the positive form since negations have been shown to be easily
misunderstood. We sorted the resulting list by word frequency to
favour frequently used and thus well-known descriptors over unfamiliar
expressions. We then clustered all synonyms together using definitions by
the Merriam Webster dictionary. For each cluster we finally selected the most
frequent and less ambiguous word as a representative. In this way we arrived at our final set of 349
adjectives. We conducted an exploratory factor analysis to reveal latent
personality dimensions. You can see here an except from our online survey which
744 workers of Mechanical Turk answered. They rated one of the 3 most
popular voice assistance namely Alexa, Google Assistant, or Siri on all of the
resulting 349 adjectives. We investigated the descriptor set's underlying
structure with an exploratory factor analysis based on the correlation matrix of all
349 descriptors. To determine the appropriate number of factors we use
the Empirical Kaiser criterion. We use an obliquely oblimin rotated
principle axis analysis for factor extraction. The resulting ten factors
accounted for 49% of the variance. The empirical Kaiser criterion proposed a
ten factorial solution. Most notably the resulting dimensions to not match the
Big Five structure. Comparing instead our ten dimensions to the Big Five model, we
find that particularly adjectives from the dimension agreeableness can be found
in several of our dimensions, such as Approachable, Social-inclined, and Social-
assisting. Since we focus our work on voice assistance agreeableness seems to
play a key role here distributed over several dimensions. I will now present
you three examples of our dimensions a little bit more closely. The dimension
Serviceable is described by positive terms which mostly relate to cognitive
functioning and role fulfilling as assistant, such as informative, functional,
thorough, or knowledgeable. For example an assistant scoring high here is likely to
react promptly, provides adequate feedback, and performs tasks in an helpful
and reliable way. It is also interesting to note that this dimension encompasses
human characteristics, for example knowledgeable or thorough which correspond
with the Big Five dimension Conscientiousness.
It also includes technical
terms such as useful or responsive. This contrast of functional and social
descriptors appears to be a pattern which can be found within the majority
of the ten dimensions.
Two dimensions emerged that appear as independent of an assistant role, such as
the dimension self-conscious. This dimension encompasses terms that render
a voice assistant as an entity capable of independent thought. While
actual artificial self-consciousness might still be in the realm of science
fiction for a long time, a voice assistant might create such an
illusion in some contexts. Hence designing a speech-based
conversational agent to score high here likely require us to implement
convincing responses in conversations about opinions, impressions, or feelings
or also requiring creative answers. The dimension artificial seems to describe a
speech-based conversational agent's similarity to humans for example
synthetic, robotic, or mechanic. Other terms here hint at the technological
implications of bringing such a "thing" into human social contexts, such as
intrusive, odd, or abrupt. A conversation agent might score high on this
dimension if it clearly presents itself as an object, for example by a
synthesised voice or poor vocal quality. So what are our takeaways here? The factor
analysis reveals that the commonly used Big Five model for human personality
does not adequately describe agent personality. Instead our descriptors also
include terms which are not associated with human personality. Yet and we want
to stress this we can only provide a first step here to developing a
personality model for conversational agents. tThe Big Five model was not
derived from one study either but was developed by multiple researchers over
decades. Thus, we encourage you, other researchers, to validate our findings. An
important next step for HCI practice is also to to derive solution principles
for effectively implementing agent personality. Practitioners usually have
specific characteristics in mind when designing agent personality. Our
descriptors can be used as a communication tool to make these
characteristics explicit and to discuss the desired personality of a new
agent in a systematic way. If you are interested in further developing our
model, please feel free to contact us at any time.You may also find additional
material for replicating studies on our website. Thank you very much.