Welcome to the presentation of the paper we present is entitled “Optimizing the Use of the Sentence Completion
Survey Technique in User Research: A Case Study on the Experience of E-Reading”.
It is co-authored by Carine Lallemand and Emeline Mercier from the University of Luxembourg
and the Eindhoven University of Technology.
Before we begin, let’s try a warm-up exercise.
We will display the beginning of sentences on the screen and ask you to complete them
with the first idea that comes to your mind.
Ready?
According to me, an online presentation should be ____
I’d be positively surprised by an online presentation that _____
I’d feel bored from attending an online presentation that _____
The most original format for an online presentation I could think of would be _____
The most memorable online presentation I ever attended was _____
What you have just experienced is part of a method called the sentence completion technique.
It basically consists of incomplete sentences, which are given to respondents who are invited
to fill in the missing words spontaneously.
These incomplete sentences are called sentence stems.
The method was originally used in psychology, more than a century ago, for intelligence
tests (by famous psychologist Herman Ebbinghaus).
It later became popular in the context of personality assessment.
What is interesting for us is that the sentence completion technique happens to also be a
powerful and cost-effective way to explore users’ needs or to evaluate systems, products,
or services.
This has been evidenced through a dozen of published studies in HCI and some reports
from UX practitioners.
While it appears as a valuable and promising asset to extend the methodological toolkit
of both scientists and practitioners, very few publications in HCI inform on the use
of the sentence completion technique.
How to design these sentences?
Are some sentence types more efficient than others, in a specific context?
Previous literature, mostly from psychology, focused on several experimental variations
in sentence stems.
For instance, in the stems subject, the affective tone used, the length of sentence stems, the
position of the blank in the sentence, the number of stems within a test, or the effect
of the instructions on participants’ replies.
Our objective is to consolidate and deepen previous work on the sentence completion technique,
by analyzing the replies given to sentence stems on a large sample through a real-life
case study.
Our research question is the following: How does variation in sentence stems affect collected
user data?
We collected data on sentence completion outputs through an online case study on the experience
of digital books.
We created a total of thirteen sentence stems related to the experience of (e)reading, covering
different dimensions of the experience.
1880 people participated in our online survey.
68% declared reading e-books and were assigned to the readers’ path, the remaining 32%
declared reading print books only and were therefore assigned to the non-readers path.
The survey was broadcast online on social networks for 2 weeks.
To address our research question, we investigated four variations in sentence stems and their
influence on participants’ responses.
Three major metrics were used to assess the richness of users’ replies to our sentence
stems: quantity, variety, and originality.
Quantity is commonly defined as the number of generated insights or ideas
Variety is usually a measure of how much of the solution space is explored.
We used the number of answer categories as an indicator of the variety of the outputs
collected.
Finally, we measured originality by comparing each idea to the frequency of its occurrence
across participants.
An idea mentioned by only 5% of respondents is considered ‘unusual’, it is deemed
‘unique’ if only mentioned by 1% of respondents.
The paper presents a detailed account of the findings, both regarding the experience of
e-reading and the formal analysis of the stems.
Based on the insights gathered in the study, we provide recommendations for the design
of effective sentence completion surveys.
As a general rule, first, define what you want to learn about and brainstorm several
sentence stems on each dimension under investigation.
Use cognitive interviewing to pilot test sentence stems on a diversity of target users in order
to find the right wording and to select the best stems.
Prompting generic dimensions such as the overall experience is good as an onboarding question.
Our item “The reading experience on an e-book is” did not lead to a wide variety of answer
categories but had the highest response rate.
Specific items can trigger a more diverse set of answers.
Comparative items collected a high response rate and a high number of ideas.
They seem interesting when there is an equivalent to the experience you seek to explore.
Print books vs. e-books, e-voting vs. paper voting, a competitor’s application vs. yours,
and so on.
Don’t be afraid to use redundant items to dig deeper into the dimensions that are of
most interest to your project.
Our item “The problem with e-books is” was complemented by “What frustrates me
the most with e-books is”, both eliciting issues and frustrations.
We collected new ideas in 63% of the cases using a redundant item and also covered more
hedonic aspects.
Items that seem suited for redundancy are the ones that produce less stereotypical data.
Finally, using “extreme or fictional” items is a double-edged sword.
The item “I dream of an e-book that” led to a lower item response rate and a smaller
number of ideas.
Yet, the ideas produced were both more original and novel.
To know more about how to design an optimal survey using the sentence completion technique,
we invite you to read our open-access paper at CHI 2022.
Thank you for your attention.