Hello.
My name is Caluã de Lacerda Pataca.
I’m a student affiliated with the Center for Accessibility and Inclusion Research at
the Rochester Institute of Technology.
I’m going to present the paper “Visualization of Speech Prosody and Emotion in Captions:
Accessibility for Deaf and Hard-of-Hearing Users.”
Automatic speech recognition (ASR) systems have come a long way over the past few years,
thanks to advancements in machine learning and natural language processing.
As a result, and despite remaining technical shortcomings, automatic captions have become
a commonplace technology, helping to make video content more accessible to a wider audience.
Despite these technological improvements, however, the visual design of captions has
seen little change since their introduction in the late 1970s.
Captions are still predominantly displayed as white text on a black background, frequently
still in uppercase letters.
Speech, though, is a dynamic and expressive medium.
Its subtle inflections and nuances help convey many layers of meaning beyond just its words:
highlights, emotions, moods, sarcasm, etc.
None of this is reflected by how captions represent words, flattening how a speaker's
voice can make their utterances expressive.
But does this matter?
While issues like latency, word error, and caption occlusion have been shown to influence
the experience of Deaf and Hard-of-Hearing users of videoconferencing systems, little
has been said about how missing paralinguistic elements can affect these users.
Our first study, then, aimed at investigating whether this distance between spoken speech's
expressiveness and captions' flat visual style is (or isn't) a significant issue, particularly
from the point of view of Deaf and Hard-of-Hearing individuals, who may depend on captions for
communication.
To do so, we conducted in-depth semi-structured interviews with eight Deaf and Hard-of-Hearing
individuals about their experiences with online meeting settings with hearing colleagues from
work or school.
These interviews were conducted by a native ASL speaker and lasted up to 70 minutes.
One key finding, in line with the literature, is that videoconferencing is, in general,
a challenging setting for Deaf and Hard-of-Hearing individuals.
Captions are often delayed, and errors in their transcription are not uncommon.
This is enough to make communication challenging.
However, even if captions work well and are able to correctly transcribe and show what
people are saying, our participants mention that they can make words seem dull or hard
to follow.
This can make it difficult for people to know what the speaker really means.
For example, many interviewees mentioned that they often fail to catch jokes because they
don’t realize that the speaker is making a humorous remark, not to be taken seriously.
Irony and sarcasm aren't shown in captions, so it is only when they notice others laughing
that they realize that what was said shouldn't have been taken literally.
Subjects talked about ways they use to deal with the unclear nature of captions, such
as closely watching the speaker's body language and facial expressions.
These methods are not perfect, though, and, in failing, they often make the interviewees
feel excluded or unaware of what's happening.
During the interview, our participants were open to the idea that captions could have
some sort of visual cue depicting subjective dimensions of speech.
However, different participants had different ideas about what exactly these cues should
reflect.
Some people thought they could show how words sound, with dimensions like the tone of voice.
Others thought captions could mimic how a speaker's emotions and feelings
change throughout a conversation.
Focusing on this issue, we ran a second study aimed at investigating which aspects of speech
should be represented in captions.
We wanted to see if it would be easier to understand a speaker's emotions, moods, and
emphasized words when the visual design of captions depicted that speaker’s prosody,
emotions, or both at the same time.
We presented four caption styles: a baseline condition with a conventional caption design;
a prosodic condition, which depicted loudness, pitch, and duration extracted from a speaker’s voice;
an affective style representing emotions, extracted with a machine learning model from speech;
and a fourth condition that had both prosody and emotions.
The baseline condition had captions styled as they typically are in everyday applications.
One detail to note, however, is that in this condition, as in the other three, we had scrolling
captions, i.e., one word showing up at a time, as one would expect to see if the captions
were generated by an Automatic Speech Recognition system.
We used different approaches to extract prosody and emotion from speech.
To obtain prosody, which includes features such as loudness, pitch, and duration,
we used acoustic measurements taken from the raw sound signal.
These acoustic features were obtained at a per-word level.
Each measured value was mapped to a specific typographic modulation, which was then combined
to form the whole modulated word.
The exact mapping between sounds and typography, along with the normalization algorithm, was
inspired by previous research on speech-modulated typography.
The second proposed model depicted emotions extracted from speech using a transformer-based
neural network.
The emotions were defined through the features of valence and arousal, as per the circumplex model.
In it, valence is related to the polarity of an emotion, and can take positive, neutral,
or negative values.
Arousal is related to the intensity of that emotion, and can have
calm, neutral, or excited values.
To the best of our knowledge, there isn’t prior work that specifically explores the
representation of emotion, as per the circumplex model, through typographic modulations.
We thus extrapolated the use of color and size from studies in tangential areas, such
as data visualization.
A fourth condition consisted of combining the modulations of both the prosodic and affective
caption styles.
The experiment involved 16 Deaf and Hard-of-Hearing individuals who participated
in a within-subjects study.
Each participant judged eight videos, two for each condition.
The videos were of individuals looking at the camera and retelling personal stories,
not unlike what could be experienced in a typical video conferencing setting.
Regarding the effectiveness of conventional captions, participants had mixed responses
when asked if they were helpful in understanding the speaker's emotions, moods,
and emphasized words.
Some found them helpful, while others did not, with no clear pattern emerging.
When considering prosodic captions there was no clear indication that they were helpful
in representing emotions.
They scored better than convention captions at representing emphases, but this result
was not significant.
The conditions with affective captions, either with emotions only or emotions alongside prosody,
significantly outperformed conventional captions in helping participants understand the speaker's
emotions and highlighted words.
However, the greatest effect was observed with captions that only had affective cues.
These captions were effective in representing both emotions and highlighted words, outperforming
conventional and prosodic captions.
One possible explanation for prosodic caption’s low performance may be found in how participants
responded to a question about how easy to read each caption style was.
Participants struggled with the styles that had prosodic cues, regardless of the presence
of affective cues.
Participants' open-ended comments, made at the end of the video portion of the experiment,
also shed light on their experiences with the new caption styles.
Some found them harder to read than conventional captions, with prosodic captions, in particular,
singled out as too 'wild' and busy.
However, despite this, the new caption styles, particularly the ones with affective cues,
were thought to make the speakers' emotions and speech clearer.
One participant, for example, suggested that this could reduce misunderstandings when they
chatted with their hearing friends.
Additionally, some participants noted that the new styles made the stories told in the
videos more engaging than those with traditional captions.
Overall, the results suggest that while prosody-based speech-modulated typography may be promising
when used with hearing individuals, as has been shown by prior work, the same may not
apply to its use with Deaf and Hard-of-Hearing people.
The limited benefit it provides in terms of communicating paralinguistic dimensions of
speech seems to be overshadowed by its negative impact on legibility.
On the other hand, we found that captions with affective cues effectively communicated
both the speaker's emotions and emphasized words.
These captions showed little perceived loss of legibility, making them a promising candidate
for future studies on speech accessibility through captions.
In conclusion, our study explored how Deaf and Hard-of-Hearing individuals experience
captioned speech, specifically in videoconferencing settings.
Captions are sometimes ambiguous or dull, perhaps due to the fact that they do not include
prosodic and affective dimensions of speech.
We found that using typographic modulations to express these dimensions is a promising
approach to bridging the gap between spoken speech and written text.
Affective cues, in particular, can represent both highlighted words and the emotional state
of the speaker with little loss of legibility.
Please reach out if you have any comments or questions.
I can be found at the links below.
Thank you for watching!