Hello, my name is Abraham Glasser. I'm a PhD candidate at the Rochester Institute of Technology,
and I'm here to present "Analyzing Deaf and Hard-of-Hearing Users’
Behavior, Usage, and Interaction with a Personal Assistant Device that Understands Sign-Language Input."
My co-authors for this work are Matthew Watkins, Kira Hart, Sooyeon Lee, and Matt Huenerfauth.
The motivation behind this work is that personal assistant devices are traditionally voice-based and pose accessibility
barriers for Deaf and Hard-of-hearing users. Personal assistant devices are gaining popularity,
and many people own or plan to buy these devices for their homes. Voice-control is also becoming an
ubiquitous interface to technology.
There has been prior work on phases of interacting with a personal assistant device, but little with DHH users.
For device activation, prior work used prototypes and discussed with DHH users their preferences and concerns
about waking up a device. Prior work has asked DHH users to imagine and prioritize commands they
would issue to a device. Prior work has asked DHH users about imagined device output modalities, such as videos
or ASL avatars or English captions for the device response. There has been a call to action for researchers to collect
more data, the key bottleneck for sign language technologies. 15 0:02:00,321 --> 00:02:15:570 Due to COVID, we used a Wizard-of-Oz setup via Zoom videoconference. The "wizard" ASL interpreter has the Alexa 16 00:02:15:570 --> 0:02:26,402 device in the room with them, and they are on a Zoom call by audio only. An Alexa device that has a display screen
to show its output, is viewable in the camera view of the zoom call. The Deaf, ASL-signing participant joins
the Zoom call from their home. There is also a research moderator on the Zoom call, guiding the participant
through the protocol. The interpreter has their video turned off on Zoom, and Alexa's Zoom instance has the
microphone turned off. When the Deaf participant signs to Alexa, they do not know that the interpreter is 21 0:03:01,151 --> 00:03:14:152 voicing the commands in spoken English to Alexa. A team of annotators watched recordings of the Zoom call 22 00:03:14:152 --> 0:03:21,902 and identified segments of the videos, and they transcribed each command into English and ASL gloss.
They noted Alexa's responses, the "wake-up" commands used by the DHH user, the category of the command,
whether the device made an error, and how people responded.
Research question 1 asked how users instinctively "wake-up" the device or initiate a command. Essentially, we discovered
some new ways that were not captured in prior work. This picture shows six screenshots of different wake signs,
and there are more details in the paper.
Research question 2a asked "What categories of commands/ requests do users produce?" and 2b asked "What do these
commands look like in ASL?" "30
For 2a, this pie chart summarizes our command categories and their frequencies. The most popular was "command and
control," where users adjusted device settings, navigated through the results, and answered Yes/No questions.
Next was entertainment-related questions, followed by lifestyle and shopping. Our paper has a detailed table of
all categories with descriptions and samples. For 2b, some key findings were that, despite signing into
a device, participants made sophisticated use of the space around their bodies, for example to represent and refer
to people or things that were the topic of their questions. We also observed unusual use of a question-mark sign at the
beginning of Yes-or-No questions, perhaps to call the attention of the device; this sign is typically used at
end of such questions. Our paper discusses more findings, including specific ASL vocabulary participants used for
command-and-control or for responding to device prompts.
Research question 3 asked how users recover or respond when there is an error or breakdown in dialogue. Our paper
describes various errors that occurred, such as Alexa not giving a result that the user was looking for, and we go
into detail how users followed up after these errors. The most popular approach by users was to simply ignore the
error, and move on to a different command. A close second method was to repeat the command with the exact same
wording and signing style, followed by rewording the command. For instance, some reworded their questions to be
more English-like, or fingerspelled words for emphasis upon re-attempts. Please refer to our paper for more details.
In summary, we describe the first observational study of the behavior and interaction of DHH individuals engaging with a
personal assistant that appears to understand ASL. We provide guidance for future designers of this technology,
and also for creators of sign-language recognition technology
We also share our video-recordings and accompanying annotations. This will be useful for future HCI
research, and serves as potential data for sign-language recognition researchers.
Again, our paper has more detailed results, and we also touch upon limitations of this study, and avenues
for future research.
Thank you!