hi everyone my name is ming ming fan and
i'm assistant professor at rochester
institute of technology
today i'm going to present our work on
older adults think aloud verbalizations
and speech features
for identifying user experience and
problems this work was done with my
students achievement and veneta
think aloud protocols have been widely
used industry to detect usability
problems
however manually analyzing recordings
and cloud sessions is often obvious
what's more ux practitioners often
analyze test sessions along
the main miss starting ux problems and
suffer from the evaluator effect
to address these challenges recent
research began to explore ai
assisted ux data analysis methods in a
recent work
we found that subtle patterns in what
and how users verbalize their thoughts
indicate ux problems however these
subtle patterns were identified among
young adults
it remains an open question of whether
similar patterns exist among older
adults
in this work we seek to answer this
question
first we recruited older adults to
participate in think allowance usability
test
sessions during the sessions
participants used the three physical and
digital products
which included a programmable coffee
machine a pad adoption website
and a food delivery mobile app these
products represented
common products that older that was used
in their lives
when working on the tasks participants
were asked to think aloud
to extract verbalization and speech
features we first divided each session
into small segments based on the
policies in the recording
for each segment extracted audio
features such as loudness and a pitch
would transcribe the audio and extract
the textual features such as speech rate
sentiment and verbal faders what's more
we
adopted widely used the categorization
strategy to provide a high level
description of users verbalizations
for each segment two ux researchers
independently identified the ux problems
and then the third researcher joined to
discuss in the consolidated final
problem labels
with the verbalization features as well
as the problem levels we could now
identify common patterns that indicate
ux problems
to measure how features indicate ux
problems we computed precision recall
and f
measure for each feature in general the
higher the measures are
the better the features indicate ux
problems
here i highlight some of the key
findings when users encounter problems
their verbalizations tend to include
observations next i'm going to play a
short video clip
to show how observations could indicate
problems
okay we'll do that
nothing's happening
when users encounter problems their
verbalizations tend to include active
sentiments as well
i'm going to play a short audio clip to
demonstrate this
notice that the user used a series of
words with negative sentiments
just press the auto button this is this
is goofy
oh i i can't
[Music]
too complicated when users encounter
problems their voices tended to include
abnormal speech features
abnormal speech features were defined as
two standard deviations away
from the average of these features next
i'm going to play a short
video to demonstrate this
oh this is horrible
okay
we also conducted a word frequency
analysis to identify the most
frequently verbalized words when users
encountered and
not encounter problems we found that
when users encountered problems
they tended to use negations question
words
words with negative sentiments and a fit
of words
when users did not encounter problems
they tended to use
task related nouns and verbs
we compared the patterns with those
identified with young adults and
found that the subtle patterns are
similar for both older adults and young
adults in general
the implication for ux practitioners is
to pay attention to these subtle
patterns
in what users say and how they say it
the first implication for hci and ai
researchers is that it is
possible to design ais that leverage
subtle patterns to detect potential ux
problems
automatically the second implication
is to leverage these subtle patterns to
design human ai
collaborative systems to increase the
analysis completeness
and reliability if you are interested in
this line of research feel free to check
out our recent papers
thank you so much for attention and i'm
happy to take any questions