Hello, everyone. My name is Yiwen. I am a PhD student at the University of Maryland. I would
like to present a haptic-captioning system using audio-haptic interfaces to enhance
speaker indication in Real-time Captions for Deaf and hard-of-hearing viewers. This
is the work I did with my colleagues at the Rochester Institute of Technology.
Captions are widely used in different media sources to support DHH viewers in understanding
the spoken content and following the conversation. However, in multi-speaker scenarios, such as panel
discussion, live stream, caption alone can be challenging for DHH viewers to identify the
active speaker in real time due to the often off-screen speakers, the conversation rapidly
switching between multiple speakers and technical delay generated by Automatic Speech Recognition.
Previous work on speaker indication has been focused on visual methods, including
caption positioning, visual cue indication, and textual cue embedded with captions.
For example, using avatars with character images and names or placing a lightbulb
to indicate the current speaker. Researchers also have investigated DHH viewers' preference
for visual-based speaker-indication methods, like color caption and speaker names caption. However,
many challenges have been found using visual methods for non-real-time caption,
such as visual overload, difficulty to follow, and distraction caused in multiple-speaker scenarios.
Inspired by some audio-haptic methods to enhance the sound awareness of DHH people.
We proposed a haptic-captioning system, which produces real-time vibrations through a voice
coil actuator and a power amplifier using Techtile Toolkit. To evaluate the haptic-captioning system,
we conducted a three-phased experiment to answer three research questions. First, how do DHH
viewers perceive speaker information in the media content through haptic feedback alone? Second,
what are the user preferences and efficiency of speaker indication of Haptic-Captioning compared
to existing real-time and non-real-time captioning methods? Third, how does the Haptic-Captioning
system affect DHH's user experience, and what factors should be considered in future designs?
For RQ1, the goal of the preliminary study is to qualitatively explore the DHH users'
ability to perceive speaker characteristics.
We removed captions and visuals from the videos as we wanted the participants to
only focus on the haptic feedback. After each clip, we asked participants to answer
two questions regarding the perceived number of speakers and perceived demographic information.
Overall, the average accuracy for the perceived number of speakers is 72.34%.
The result is still promising, considering the system evaluated only through vibration.
Participant's feedback reveals that DHH people can infer the speaker's demographic information
through the vibration intensity. However, some participants were still concerned when speakers
had similar voice patterns or speakers' tones changed. Next, we conducted a within-subjects
study in Non-real-time Caption methods, Avatar Caption, Color Caption, Position Caption, Pointer
Caption, Speaker-name Caption, and 2 Real-time Caption modalities, Traditional Real-time Caption.
The same style is used with Haptic-captioning. The order of caption modality is randomized.
In study 1, we first ask participants to report perceived speaker transition moments
through an experimental interface. After each condition, Participants filled out
a self-reported questionnaire. We conducted a post-study interview to understand their
subjective experience in depth. We recruited 16 DHH participants aged from 18 to 35.
Participants were recruited from locate groups on Facebook and the institute's mailing lists.
The overall results of the Perceived Speaker Transition accuracy showed no significant
difference between caption modalities, including real-time and non-real-time caption methods.
In the self-reported questionnaire, we compare haptic caption with traditional real-time
caption and non-real-time caption methods. The results show that although it is not significant,
Haptic-Captioning had better ratings compared with Traditional Real-time Caption in all dimensions,
more engagement, more confidence in speaker indication, more comfortableness,
better understanding. And less difficulty in following distraction, and fatigue.
To compare with non-real-time caption methods, the analysis shows the Position Caption performs
significantly better than haptic-caption in the engagement, confidence, and comfortableness.
The post-study interview helps further understand user preferences, as well as
the characteristics of the Haptic-Captioning system. One Participant commented on the
benefits of the haptic-captioning system as "it helps me hear the vibration and read the
caption in which it will switch the speaker so it looks like paused voice by the speaker."
Some concerns remain, as P6 mentioned, "Distracting (is) like, I was trying to
focus on the caption content, but then I feel the buzz and shift my attention
to my watch and realized I forgot it mean switching speakers…" Participants provided
suggestions that Using Haptic-Captioning in combination with other visual captioning
methods as Haptic-Captioning adds another level of feeling connected to the content being shown.
Informed by the previous two studies, we first asked participants their preferred vibrating
positions. Then we played three 4-minute videos podcast, sports, live stream and movie in three
settings. We conducted a semi-structured contextual interview to elicit DHH users'
feedback in different application settings and explore future design opportunities.
Through the thematic analysis, we identified three emerged themes: attention, perception,
and customization. Participants reflected on several positive experiences as the system
improved the caption's readability and maintained media engagement.
The Haptic-Captioning system enhanced the user perception of emotions and then assisted speaker
indication, especially when the speaker is off-screen. We also identified several
situations in which the system design needs to be improved to fit different usage contexts.
Due to the time constraint, we recommend you read the full paper for more details. Some design
implications for future haptic-captioning systems. We want to provide adjustable
vibration based on the usage context and generate distinguishable haptic patterns for the caption.
We want to support immersive experiences through spatial audio-haptic feedback. One of our ongoing
works is the conversational tone to present more rich speech and non-speech information.
Thank you for listening. If you have any questions, please contact us!