Hi I'm Jordan Wirfs-Brock, presenting "Examining narrative sonification using
first-person retrospection methods to translate radio production to interaction design."
This is work done with Alli Fam, Laura Devendorf and Brian Keegan. Sonification is the practice of
communicating data through sounds -- think of it as charts and graphs for your ears.
There are many ways to do this, for example you might map quantity on
to pitch so a normal distribution bell curve might sound something like this
Just like visualization communicates data through visual marks, sonification communicates
data through auditory sounds. And typically sonification designers focus on non-speech
sounds -- the bleeps and the bloops -- and not the context in which sonifications are delivered.
This is a little bit strange. Imagine if visualization designers did the same thing
graphics might look like this. It's pretty but what does it even mean?
and if i were to slowly reveal some of these scaffolding annotations
it gets much easier for you to understand what this visualization is all about. And yet this
covered up version without annotation that's how sonifications are often presented to listeners.
So visualization isn't just the marks it's also the text scaffolds like labels and
annotations. Similarly we wanted to explore how sonification isn't just non-speech sounds but
also how it includes speech-based scaffolds like spoken narration. We call this combination of
sonification and speech-based scaffolds narrative sonification, and to explore the role of narrative
contexts we turn to a medium with a long and rich tradition of spoken narrative: radio. We created
sonifications for the radio show "Marketplace" and analyze them to understand how they work.
the goal of this study is to examine our practices as sound designers and radio producers so that we
can translate that tacit and implicit knowledge into extensible interaction design principles
for emerging technologies. we explored three key research questions: one, how can narrative help
people learn to listen to data; two, what can interaction designers developing emerging tech
like voice interfaces learn from radio; and three how can we use retrospection to support
interaction design. in this video i'll be focusing on findings addressing the first two questions.
Here's an overview of our research process: first we made the radio sonifications,
then we retrospectively analyzed these radio pieces and our process of creating them,
and finally we speculated how we might reimagine those radio pieces as voice interactions. This
study deals with two radio sonification pieces from the early days of the covid19 pandemic,
one on the volatility of the stock market the other on unemployment claims.
Today I'll be focusing on this second radio piece.
In the spring of 2020 as we first saw the economic impacts of covid19, unemployment
claims in the us were staggering. In this New York Times visualization they used the entire
front page gutter to extend the y-axis. With such a mind-boggling jump in scale we've grappled with
how to convey that data through sound without blowing out your eardrums. Ultimately we've
represented new unemployment claims with a harp sound with a higher pitch for more claims and
we've represented continuing claims -- that's the total number of people on unemployment benefits
at any given time -- with a series of layered drum sounds. We then embedded these sonifications into
a radio interview for Marketplace where we taught listeners how to understand and interpret the
sounds through a dialogue. Here's a diagram of the narrative structure of that radio piece. I'm going
to play a short excerpt where we first hear the whole sonification to produce an auditory gestalt
and then the host, Kai Ryssdal, tees up the training session we're about to have
(Ryssdal) We really like the idea of economic data that you could hear so we called her back
to ask her about unemployment those incredible first-time claims numbers we've been reporting on
since march and the related continuing claims for unemployment benefits and here's what she sent us
all right so that is what unemployment in this economy sounded like from February to
the beginning of this month, but since most of us aren't really used to listening to data
we have gotten Jordan back on the phone to help sort it all out for us. Jordan thanks for coming
back on. (Jordan) thanks happy to be here. (Ryssdal) okay so as we did last time we're
gonna break this down because uh it was really interesting to me in our previous conversation how
we you you had to teach me how to listen and thus we taught everybody else together how to listen so
that's right talk to me first of all yeah talk to me first of all about the new first time claims
for unemployment what is what are those sounds. (Jordan) yeah so to break this down... Then
from there we went on to decompose the sonification then we put it back together
so listeners could hear it again with new ears and finally apply those new listening skills
to new data from the great recession of 2009. Now we came up with this narrative structure
somewhat on the fly using our intuition as sound designers and radio storytellers. So to better
understand the radio sonification pieces and our process of making them, we conducted a rigorous
multifaceted first person retrospection process outlined here. It included reflective writing,
close listening, analytic sketching, and applying a theoretical framework. You can see our paper for
further details. Here are some of the themes that emerge through this process and i'm going
to focus on three of them: one, our relationship to time; two, narrative multiplicity' and finally
three, data literacy as an act of interpreting and creating. Time is fundamental to the nature
of sound which is linear and ephemeral. With radio you cannot pause and you cannot go back.
As a sound designer i was trying to compress time to make the sonifications faster so that
they would take up less of it. Whereas the radio producer, Alli, was quite focused on slowing
things down on stretching time. But as we were realized we were working toward the same goal
which was manipulating time to make room for the narrative -- narrative that could help the
listener learn to understand the sonifications. In these radio sonifications narrative served
as a bridge between the past and the future it could hearken back reminding the listener of
what they just heard or foreshadow the future and by labeling the sounds with words we could help
listeners remember and compare them. There were many different kinds of narratives at play here:
the story of the impacts of covid19, the story of the data and how to interpret it, and the story
of my journey as a sound designer creating these sonifications. And this last one is intriguing
because it's something i would have edited out but the radio producer, Alli, thought it was
important for the audience to hear it this story of how i interpreted and created the sonifications
illustrates an important concept of data literacy, which is that literacy combines both theoretical
knowledge -- say how a chart functions -- with procedural knowledge -- how to actually make
a chart. To be truly data literate you have to be able to both read and write visualizations.
And so we speculated how we might apply this to sonification literacy using voice interfaces.
An interactive version of the unemployment certification might first play a sound clip then
the voice assistant would ask you to interpret it: were the unemployment claims going up or down?
Then it might ask you to creatively extend this idea to make your own sonification:
what do you think unemployment claims sounded like when the dot-com bubble burst back in 2001?
These design insights lay the groundwork for future interactive sonifications that can
support data literacy through acts of creation -- and these kinds of voice technologies
we're planning to continue prototyping and deploying in the future. Thank you so much.