I'm pleased to introduce our work: Wearable Subtitles Augmenting Spoken Communication
with Lightweight Eyewear for all-day captioning. My name is Alex Olwal and this is work that
was conducted in Google research with my colleagues Kevin Balke, Dmitrii Vontintcev, Thad Starner,
Paula Conn, Bonnie Chinh, and Benoit Coda. Mobile apps, which transcribe spoken conversation
to text in real time, can help deaf and hard of hearing individuals follow spoken conversations
more easily than when relying only on context and lip reading. While, we acknowledge that
not all deaf or hard of hearing individuals want to use sound or captioning technologies,
prior work has demonstrated The many people would find captioning technologies desirable
and useful in everyday activities. However, transcription apps require users to look down
at the phone, which can make it challenging to maintain eye contact or situational awareness.
Similarly the need to regularly reference a handheld device can make many common social
interactions difficult such as conversing while eating or walking. Researchers have
explored ways to address such challenges through head-worn displays . For example, in 2013,
Google supported Georgia Tech’s “Captioning on Glass” effort, which used speech-to-text
to create a head-worn captioning system. The system was, however, focused on short conversations
given the display, which was centered high in the visual field, and a battery life that
was designed for brief, glanceable interactions. In “Wearable Subtitles”, we discuss proof-of-concept
eyewear that transcribes speech and ambient sound into visual representations that people
who are deaf and hard-of-hearing can use during daily social interactions. We developed a
series of research prototypes designed for all-day use that are hands-free and allow
the user to read what is being spoken while still visually engaging with their conversational
partners. The architecture is designed to support both cloud-based and stand alone offline
transcription with all day battery life. "Be careful. This fish has a lot of Bones. Oh,
really? Thank you for telling me this . Sara would you like some water? Yeah I would love
some water . Oh, I'll give you water. Thank you". Our user studies with deaf and hard-of-hearing
participants indicate that head-worn displays can help resolve a number of challenges with
current phone transcription solutions and can facilitate communication through privately
transcribed text, hands-free use, improved mobility, and more socially acceptable interactions.
To learn more about challenges for heads-up, real-time transcription in mobile contexts,
our team conducted a brief, large-scale online survey in the US with 501 participants who
are regular users of hearing-related assistive technologies.Participants were asked about
scenarios that they experience on a daily basis in which they might experience difficulty
hearing while using their current assistive technology. The range of problematic scenarios
provides a motivation for more subtle, comfortable communication devices with all-day power to
mitigate difficulties in everyday scenarios. To enable all-day wearable subtitle experience
we have to carefully consider not only transcribing latency, but also make correct architectural
decisions about communication and computation distribution between eyewear, mobile device,
and the cloud services. With the eyewear functioning as a thin client, receiving only user commands
and transcribed text from the phone. The payload from the glasses to the phone at this time
is also minimal, as all the audio is captured using phone microphones, or remote microphones
connected to the phone. Now let’s zoom into the eyewear electronics architecture. At the
heart of the system you will find an all-in-one low-cost SOC from MediaTek - MT2523D. Originally
designed for wrist-worn devices, it features an integrated Cortex-M4 processor running
at 208MHz with the possibility of dynamic frequency scaling to further power savings.
Power management unit is also built in and features a triage of switching and linear
regulators powering the numerous power rails with the battery terminals as the single power
input. Lastly, the SOC has a built-in BLE transceiver with sufficient sensitivity, transmit
power, and efficiency to have a reliable yet low power communication channel to the phone.
On top of the SOC we had to add all the user interaction elements - microdisplay with the
appropriate power regulators, user buttons, and an LED. LED is only used when the user
is not wearing the device for indications such as “glasses are powering off”, “glasses
are on”, or “battery is too low”. The user buttons are comprised of a power button
tight directly into the power management unit, and two buttons that are handled by our logic,
and can be used for display brightness adjustments, or modes of transcription. All the features
of system are fitted on a single board that is residing in the right temple of the glasses.
Here is the zoomed in version. The PCB features 8 layer stack with tightly placed components
to fit into a non-obtrusive and lightweight temple. Interestingly enough, one of the most
challenging components to design into the system turned out to be the mechanical buttons.
The tactile switches had to be sufficiently small to fit onto the board, yet providing
large enough surface for depression and good enough tactile response. That took quite a
few electro-mechanical iterations to get this to an acceptable state. Proper fixation of
the board in combination with guided caps that go over the buttons’ domes helped with
the good overall design. After the first design iteration, we had conducted a number of user
interviews that showed that there are areas of electro-mechanical as well as firmware
and software stacks improvements. I will focus on the hardware aspects in the next few slides.
One fairly straightforward technical challenge was resolved after the participants started
complaining about the device disconnects. As it turned out, the antenna layout was missing
critical radiator traces which resulted in poor TX/RX performance. Replacing the antenna
with a chip antenna of the appropriate characteristics fixed the problem. A much more fundamental
challenge was to come up with the design that would be able to accommodate wide variety
of users with their unique facial features and geometries. As seen on the graph, most
challenges were associated with the display not being fully visible, not optimally located
in the field-of-view, or disappearing over time as the glasses would move on the person’s
head. It is an especially challenging problem given the research nature of the project,
as there is no resources to design and build unique mechanical arrangements fitted to every
user. To mitigate the challenge with the display placement and visibility we designed a modular
aspect in the mechanical arrangement of the glasses - adjustable in V1 or interchangeable
in V2 of the nose bridges. The adjustable, bendable, or sliding bridge, made of cut steel
plates, did a decent job in providing independent adjustments for inter-pupillary distance or
wrap. The challenge with this method is that it was not an easy fitting procedure, and
also wasn’t stiff enough and would get out of fitted position over time, significantly
affecting display visibility. As the next step, we’ve introduced an interchangeable
nose bridge, like shown on the figure to the right. The users would go over a fairly quick
fitting process. Several facial measurements such as apex and IPD would help us narrow
down the bridge selection to only a few, and we would try fitting a user with an optimum
nose bridge through several iterations. We could clearly see the improvements in display
visibility, position, and fit thanks to this methodology. Another aspect that was addressed
as part of making the glasses comfortable is weight distribution. The V1 design placed
all system components in the temples, and as a result, the weight of the device rests
primarily on the nose, up to 78% of the overall weight. Changing the power architecture to
use two smaller batteries located behind the user's ears, moves to the center of mass backwards,
reducing the nose weight to 56%. As a result person’s nose is less stressed, and the
glasses are much less prone to sliding down the nose. Individually-adjustable nose pads
and per-user fitted swappable nose bridges enable the device to conform to the user's
face and improve both comfort and display legibility. The two largest challenges to
device usability in user studies were Bluetooth connectivity and display legibility. We addressed
these factors by building a highly fault-tolerant Bluetooth firmware implementation that could
recover from failures in the Mediatek and Android Bluetooth stacks and by implementing
rendering improvements for text display to make the transcript smoother and easier to
read. We also improved battery life by characterizing the power consumption of the major system
components and mitigating each drain with a software solution. Batched transmission,
low power modes, display brightness control and auto-off increased battery life significantly.
Here we show the relationship between transmission efficiency and the Bluetooth transmission
interval and data block size. By packing more bytes into individual transmission events
and sending them more infrequently, we can improve battery life. Our Bluetooth implementation
leverages this strategy for non-latency critical data, such as events logging. For latency-critical
data, such as the transcript updates, we transmit smaller events more frequently, trading off
power consumption in favor of user experience. The display backlight is the single largest
consumer of battery energy. Here, we show the relationship between backlight level and
battery life. By adjusting the display brightness to match ambient light levels and by turning
off the display while not in use, we can approach 15 hours of battery life. To iterate quickly,
w e developed tools to enable controlling our captioning devices. Our JavaScript binding
in our Android service application enables building web apps that can be loaded like
any other webpage, but which are capable of communicating with the captioning service
to control the paired device. Our Python API binding provides for rapid prototyping natively,
communicating with the captioning device using the built-in Bluetooth radios on any Mac or
Linux computer. We built applications in Python that enabled us to quickly explore new user
interface designs together with our UX design collaborators, as well as to easily test the
display quality of each device without needing to pair to a phone. We also developed bench
test applications that measure connection latency and throughput and exercise other
system functionality for hardware validation. These test applications were used to generate
the efficiency graph shown previously. Here we show two test patterns used to validate
the display and graphics engine performance. The Python API gives full control over the
graphics API, enabling quickly prototyping user interfaces without having to either rebuild
the device firmware or deploy an app to an Android device. One hour in-person sessions
were conducted with 6 participants to compare the social and physical comfort of captioning
on a mobile device versus on eyewear. Two activity conditions were considered during
these sessions: engaging in a group conversation while being stationary, and while being in-motion.
Upon experiencing both of these conditions and activities, Participants shared that the
eyewear was easier to use while walking and that it was more discreet than using a phone.
The eyewear also resulted in higher level of awareness of the environment and who was
speaking. In comparison to using a mobile phone for captions, a participant explained,
“The capacity to have other people talking and have me actually know what’s going on,
instead of me going off other peoples body movements, that feels freeing.” The prototype
also showed more favorable ratings than the first version of the prototype, namely across
measures like, fit, caption visibility, clarity of field of view, and UI placement. In summary,
this work introduces a lightweight, all day wearable display with real-time speech recognition,
translation, and sound annotation all within the wearer’s field of view. We add to emerging
research, namely the work by Jain et al., that discusses the opportunities for head-worn
captioning displays while engaging in on-the-go activities. Future work seeks to continue
refining and evaluating the prototype across different usage contexts and with a larger
number of participants. This includes a greater understanding of multiple speaker context
and a public environments. Future work will also extend to the contexts that conversational
partners who did not self-identify as deaf of hard of hearing, expressed interest in.
These were contexts where all conversational partners imagined themselves turning on a
head-worn captioning display. We’d like to thank our collaborators outlined in the
paper. This work was completed in collaboration with Alex Olwal, Kevin Balke, Dmitrii Vontintcev,
Thad Starner, Paula Conn, Bonnie Chinh, and Benoit Corda.