Wearable Subtitles: Augmenting Spoken Communication with Lightweight Eyewear for All-day Captioning
Galina: Dima, when you are done washing dishes can you please put a salad for me into the
plate? Dimitri: OK, I put salad.
In this work, we explore the potential of a proof-of-concept eyewear display system
to augment the communication for deaf and hard of hearing individuals. In contrast to
mobile phones, eyewear can enable hands-free access, mobile use and be more socially acceptable
and convenient across many everyday activities.
Eyewear might be particularly beneficial for activities where the user's hands are occupied,
where users want to focus on a primary task with maintained awareness of speech or instructions,
or in general want to avoid split attention between a handheld display and social interaction.
We developed a proof-of-concept thin-client architecture that is lightweight and low power
to enable the private display of transcribed speech for a full workday.
We worked closely with Dimitri Kanevsky, a hearing accessibility researcher, who relies
on transcription and lip reading. These videos were filmed through our display with the live
system, to show what the wearer experiences.
Alex: So what I am hearing from you is that group conversations are particularly challenging
when you have multiple speakers and you need to follow who is speaking. With the phone
you say it's difficult. Can you tell me a little more why?
Dimitri: Yes, with phone I don't know who is speaking. But if I'm using glasses...
I am looking at one person. When person finishes to speak, and another person starts to speak,
immediately eyes of this person looks at another person.
Alex: So, even though the current implementation does not separate the speakers, you can associate
by following and being more aware of the dynamics in the room and across different speakers.
Dimitri: Yes, yes... yes. So, when I use glasses, I really know who said what.
To evaluate our proof-of-concept eyewear, we ran a number of studies with various activities,
such as group conversations in a controlled environment. Our findings suggest that eyewear
have the potential to increase awareness during multi-speaker conversations. Our participants
expressed that group conversations were particularly challenging and that eyewear would be a welcome
technology to facilitate participation.
Galina: Please be careful, this fish has a lot of bones.
Dimitri: Oh, really? Thank you for telling me this.
Galina: Sara, would you like some water? Sara: Yeah, I would love some water, please.
Dimitri: Oh, I give you water! Sara: Thank you.
Sara: Thanks so much.
Dimitri: You are welcome!
Our results also suggest that eyewear have a unique potential to improve conversations
in mobile contexts.
Dimitri: When you are walking on street, it's difficult to watch phone and talk to
person who walks near you. But with glasses you can see everywhere, cross road, cross
street and at the same time you can understand what talk... what other person talk to you.
For all-day comfort, our revised prototypes emphasize a balanced design and personalization
through customizable nose bridges. The nose bridges help with individualized fit, alignment
of the eye to the virtual image, and enable adjustment of the image location.
In our technical evaluation, we characterize power both from a system and communication
scenario perspective. At around 260 mW, we can support continuous streaming of transcribed
speech for 5-15 hours, depending on the desired display brightness.
Wearable Subtitles shows that eyewear have the potential to cimprove hearing accessibility
through privately transcribed text, which can be used hands-free, in mobile contexts,
and in socially acceptable interactions.