My name is Steven Goodman and I will be discussing our paper, “Toward User-Driven Sound Recognizer
Personalization with People Who Are d/Deaf or Hard of Hearing”.
Automatic sound recognition can support communication and environmental awareness for deaf and hard
of hearing, or DHH, people.
Prior implementations use pre-trained models with a set of generic sound classes, but these
systems do not meet personalization requests from DHH users:
They do not allow custom sounds—such as name calls from different family members.
Nor can they recognize edge cases in real-world sound events—such as separating my dog’s
barks from a population of many.
Enabling personalization would benefit DHH users, but end-user ML systems that augment
sensory abilities present challenges for users with sensory disabilities.
Two prior studies have explored personalizable sound recognition tools with DHH participants,
But how DHH users record and engage with audio data is mostly absent—despite this data
predicating the effectiveness of a personalized sound recognizer.
We identified three research questions:
First, how can a DHH user, who has difficulty hearing a sound themselves, effectively record
samples for a system to recognize that sound?
Second, what considerations do DHH users make when recording in environments with real-world
acoustic variation?
And third, what kinds of features can aid DHH users in assessing their recorded samples
as training data?
We conducted a three-part study with 14 DHH participants:
first, we held a remote interview session to introduce audio recording for sound recognition.
We demonstrated spectrogram and waveform visualizations
And introduced a simple ML workflow using Google’s Teachable Machine.
Participants recorded hand claps, paper crumpling, and background noise, then trained and tested
a model.
Throughout, we discussed factors impacting the quality of audio data for training.
During the second stage, we ran a week-long field study of their real-world recording
practice.
Each day, participants used Rev Recorder—an off-the-shelf smartphone application—to
record at least three non-speech sounds in their life, and completed a questionnaire
about the experience.
At the end of the week, we met again for a semi-structured remote interview and used
a design probe activity to discuss how to improve the experience.
Let’s move on to the Findings.
All 14 participants were enthusiastic about recording sounds to personalize a sound recognition
system, describing the experience as as “easy”, “interesting”, and “fun”.
They recorded a total of 243 sounds at an average of 2.8 samples for each sound, and
commonly chose sounds pertaining to indoor mechanical events and social presence.
Participants reported the most success for sounds that were continuous like P12’s “floor
fan”, prominent like P14’s “microwave beeps”, and controllable like P13’s “flushing
the toilet”.
Likewise, uncontrollable sounds were a source of frustration, such as unreliable “meows”
from P3’s cat.
Because DHH people rely heavily on visual awareness, hidden sounds were especially difficult.
Emergency sirens were a commonly missed sound; as P7 said, “I had no idea when to start
the recording. If I wasn’t able to see the vehicle, then I couldn’t do it.”
Several participants said Rev’s waveform was crucial for interpreting the contents
of their samples.
However, breakdowns occurred when participants’ intuition of the sound did not align with
the displayed visualization.
For example, P6 expected to see prominent waveform peaks during a thunderstorm, but
instead found a confusing “jumble of noise” and “blob of information”.
This caused her to disregard the waveform during the rest of the week.
When considering the sounds in terms of training a recognition model, participants’ limited
frame of auditory reference led to uncertainty over how closely their samples replicated
the sound’s real-world population.
Those with residual hearing tried audio playback to determine whether their samples reflected
the real-world version, but this was unreliable.
Others, like P12, did not have this ability: “As a deaf person, I’m just relying on
my vision and my other senses there are visual indicators, but it’s hard to emulate realistically.”
When recording multiple samples of the same sound class, limited perception of audible
differences caused many participants to report further uncertainty about how to capture realistic
variation in their samples.
For example, although P2 recognized the benefit of diversity within samples of the same sound,
incorporating it into her data was left to guesswork, such as, “I suspect the doors
and blinds sound differently when they are pulled or pushed in different speeds.”
Participants’ limited ability to hear audible differences between sounds also contributed
to uncertainty toward possible decision boundaries within the model.
For instance, although P9 desired two separate sound classes for the faucets in his home,
he was unsure whether “a stainless steel rectangular sink” and “a rounded porcelain
sink” would produce an audible difference to enable this.
To summarize:
Participants reported a positive subjective experience, but their limited auditory expertise
led to unique challenges with: Assessing a sample’s contents using playback
or waveform visualization.
Replicating a sound’s real-world occurrence, including its range of acoustic variation
Estimating potential decision boundaries based on audible difference among sounds.
Thank you very much for listening, and I invite you to refer to our full paper for the complete
details about this work.