Hello! My name is Matthew Seita and I’ll be presenting our paper “Remotely Co-Designing Features
for Communication Applications using Automatic Captioning with Deaf and Hearing Pairs.”
Our research involves technologies that incorporate automatic captioning. The motivation for our work
stems from challenges faced by the Deaf and Hard of Hearing (DHH) community. Many DHH people experience forms of
workplace inaccessibility, in large part due to the frequency of impromptu conversations.
Access services, such as sign language interpreters or professional captioning services
usually need to be scheduled well in advance and are often unavailable.
One potential solution to this challenge is the use of automatic speech recognition (ASR)
which automatically translates speech to text in real-time and can be used on a mobile phone.
ASR, however, has its own challenges. Compared to the rapid growth of ASR apps, how design factors affect usability,
especially for users with hearing disabilities, has been relatively under-explored in HCI. To investigate this more in-depth,
we conducted a co-design study with DHH and hearing participants to develop new features and designs for ASR
technologies to improve usability.
Due to the onset of the COVID-19 pandemic, we could no longer meet in-person, so we had to shift completely online.
Moving to a remote modality for co-design was a challenge, and it was unknown how effective having online co-design sessions
with DHH and hearing participants actively collaborating would be, given the potential for communication
or visual-attention barriers for DHH users while drawing and communicating with a hearing partner.
Our primary contribution is methodological. We present a novel virtual co-design methodology with DHH and hearing
participants, in which participants collaborate and create prototype designs.
Our secondary contribution is empirical where we use our virtual co-design method to investigate the design space of features
for ASR-supported communication applications in both in-person mobile and, motivated by COVID-19, remote
video-conferencing contexts alongside two design dimensions: (1) Error correction and
(2) Notification systems for influencing speaking behaviors.
For our study, we recruited 18 DHH and 18 hearing participants who were paired up during our sessions. Half investigated
and the other half remote contexts. Two researchers were present as moderators and we had two-hour sessions,
including ASL interpreters, and met through Zoom and utilized Miro as the collaborative tool.
Our sessions included four separate activities: brainstorming, ice-breaker, sketching, and prototyping.
During brainstorming, participants independently brainstormed ideas for each design activity. Next participants had an
had an ice-breaker where they were introduced to each other and to Miro. Next, they quickly and individually
sketched some of their brainstormed ideas. Finally they came together and collaboratively worked
during the prototyping phase. For full information about our methodology, please refer to the paper.
We present a picture here that shows a view of Miro during some part of one session, and you can see the design space
on the right side while participants are prototyping, and on the left they are conversing using integrated text-chat.
The design prompt we gave participants emphasized our two design dimensions, guiding participants to create features
based on error correction and notification systems. The next slide shows some examples of prototypes for notifications.
One idea on the left included a small circle in a corner that expands when an action is needed, like speaking more slowly,
On the right, participants prototyped a pop-up that appears on the screen with instructions,
in this case, saying that the voice is unclear and to record the speech again.
After our sessions were completed, we performed qualitative analysis and inductive coding of video recordings we took of
each session. A final list of themes emerged, including communication modalities,
use of the online collaborative workspace, conversation flow and communication strategies,
negotiation and reconciliation of ideas, and participant feedback. Due to time constraints, I will discuss only
numbers 3 and 5, communication strategies and participant feedback. For information on all themes please see the paper.
Recall that previously we discussed that the DHH person has to divide attention between multiple sources while working
simultaneously, and our groups addressed this issue differently. Our analysis identified
various successful strategies that occurred, and I will discuss one here: Hearing draw and DHH reinforce.
One successful strategy was when the hearing person spoke while drawing, to describe in detail what they were drawing and
what each feature does. Meanwhile, the DHH participant set-up the ASL interpreter and Miro side by side on their screen,
in order to rapidly switch attention between them, which better enabled them to contribute, e.g. giving feedback
as the hearing participant drew. This strategy took advantage of the hearing person’s ability to speak and draw simultaneously
There were a couple other successful strategies, for more information please see the paper.
Exit interviews revealed that our participants had a satisfying co-design experience. The integrated chat within the drawing
on Miro, with the picture again shown on the right for reference, was very helpful and our methodology helped
include both DHH and hearing perspectives. However, participants did suggest that they would like to
have more time to get to know each other and to get more familiar with Miro features.
For future researchers designing studies with this methodology, we recommend the same 4 phases: Individual brainstorming,
ice-breaker with drawing tool, quick individual sketching, and collaborative prototyping.
These four phases allowed participants to begin individually with their own ideas then compromise later as needed.
Setting up the Workspace: To keep sessions organized, moderators should pre-assign and label locations within the
drawing canvas of the online collaborative drawing workspace to organize the data and ensure participants are clear on
where to prototype within. This is especially critical with mixed-ability groups as more people are involved, e.g. interpreters,
and could increase distractions for DHH users. Moderators should encourage participants
to add text boxes to their drawings of each design, to add explanations of features to aid access to information.
That’s all for our presentation! Thank you for watching.