I’m Jordan Wirfs-Brock, presenting work I did at Spotify last summer with Sarah Mennicken
and Jenn Thom.
Music streaming services, like Spotify, collect data about their listeners – things like
which songs we play the most, or what time of day we prefer acid rock over pop.
This data is necessary to personalize streaming services and help listeners discover music.
Yet this data, and how it’s being used, is often not apparent to end users.
The user data is there, behind the scenes, but it’s silent.
While it’s hard to access this data in visual interfaces, in voice interactions there’s
even less context for it.
In this project, we explore how to make this silent data meaningful to music listeners.
We focus on voice assistants because, people listen to a lot of music on smart speakers.
But, they tend to use simple functionalities, basically treating them as glorified voice-activated
remote controls -- so there’s a huge opportunity to experiment with new ways to design experiences
in this space.
Taking a research through design approach, we conducted five user workshops focused on
designing future interactions with a voice assistant around music.
Each 90-min workshop had a pair of Spotify listeners who’d never met before.
We recruited 10 highly active Spotify listeners who regularly used voice technologies and
self-tracked personal data in some way, like keeping a journal or wearing a smartwatch.
We focused on these extreme users in order to push beyond current listening and voice
paradigms.
As a prompt, we gave each participant a data profile, generated from their Spotify listening
history, that included top most-played songs, temporal patterns, and genre preferences.
We conducted the workshops in pairs so that participants could interpret their personal
data in a social context and bounce ideas off each other.
The workshops had a range of design activities that encouraged participants to reflect on
their data in diverse ways, including...
A guided visualization exercise imagining a future interaction with one of their most-played
songs and a voice assistant.
Mapping out conversational voice interactions with structured dialog cards.
Exchanging data profiles and discussing them.
Building a timeline of voice interactions unfolding over the course of someone’s life.
And interacting with Wizard of Oz prototypes of their new voice interactions, that incorporated
text-to-speech, music, and sound effects.
We analyzed transcripts from the design workshops using an affinity diagramming method to let
themes emerge inductively.
I’m going to share some of our findings in two areas, first about how personal data
can be a design resource; and second, how calling attention to personal data can inspire
new kinds of voice interactions.
Through viewing their personal listening history data, participants reflected on their identities
as music listeners.
Sometimes, the data validated pre-existing views participants held – like how they’re
a person who has eclectic taste, or is a superfan of an artist.
But a metric can hold many meanings.
Take play counts: A song with a lot of plays could indeed be someone’s favorite… but
it could also be the music they listen to while they sleep.
Or their child’s favorite song.
Take the act of listening itself – one participant associated listening with being busy at work.
Upon seeing how many hours a week he listens to music, he said that he needs to work less,
and “get outside more.”
Whereas another participant saw that same metric and commented that he needs to listen
to more music – something he associates with downtime.
These nuances around data and identity emerged as participants interpreted, reflected on,
and designed with their data.
As they did, participants were able ground their futuristic thinking in their current
experiences.
For example, one participant was fascinated by the discrepancy between her top songs of
the past month and her top songs of all time.
She reflected on how she goes through phases where she becomes obsessed with an artist
or song and listens nonstop, until she gets burnt out and can’t listen anymore... so
she imagined a voice assistant that could help her manage this behavior and extend the
pre-burnout phase.
Here’s how she described it:
‘I wish it was like when I hit five listens in one day, it was like, “You’ve listened
five times.
Here’s another artist that opened for her.”
Or if it gave me other context where I could get distracted from her a little bit.’
In addition to behavioral data, we shared meta-data that participants might not have
been aware of, such as genres of their favorite artists, or their geographic origins.
Exposure to this meta-data provided a new lens on participants’ existing behaviors.
Some imagined a voice assistant prompting them with these metadata dimensions, like:
You listen to more artists from Detroit than any other city – would you like to hear
a Detroit playlist?
Which leads to our next finding: That data served as a trigger, not a focus, of the voice
interactions our participants designed.
Going into this study, as a data nerd, I thought people might want to explore their data using
sound and voice as a standalone activity.
Instead, they focused on data milestones – the 100th song added to a playlist, the anniversary
of the first time they heard a song – as events that might trigger a voice assistant
to suggest actions, like sharing the playlist, or exploring adjacent music.
These interactions explicitly reference data points, but the data isn’t the point.
The point was an enhanced way to meaningfully explore music.
Participants wanted to ditch the voice-activated remote control and instead imagined something
more like a music coach, who could proactively support them as they expanded their musical
personalities and evolve their musical tastes....A voice assistant that could leverage deep knowledge
about music to do things like jumping straight to the chorus, reciting back lyrics, or explaining
the meaning of a song.
Some participants imagined that this musical intelligence might even be combined with intelligence
about them, the listener.
But people acknowledged that this wouldn’t “just work” – but that they’d have
to collaborate and teach the voice assistant.
How?
Well, they didn’t expect a voice assistant to always get things right, but they imagined
that it might ask for feedback so it could course-correct when it gets things wrong.
Or, some participants imagined ‘tagging’ the music with their own language.
As one participant put it:
“I feel like it’d be really easy to be like, ‘I like that...that song makes me
feel really happy’ or ‘I’m feeling uplifted”’ - Brianna.
Through this kind of specific, on-the-fly labeling of their music experiences, participants
described building a personalized, shared language with the voice assistant.
They also imagined dialogs to decide--together--what to listen to… collaborating with the voice
assistant, as a music coach, to figure out what to play, rather than simply commanding
it.
These findings are helpful for creating voice interactions around music, but they also have
implications for the broader HCI community in voice and system design.
Thus we leave you with two ways you might consider applying this approach to your work:
First, consider how previously silent data might become a new vocabulary for exploration.
We found that presenting people with silent data provided a new lens on their existing
behaviors – and a new vocabulary for understanding them.
A vocabulary that can also scaffold people as they discover features or content.
Exposing silent data may be a way to tackle a common problem in voice interaction design
– how do users figure out what they can do with voice assistants?
Imagine this interaction:
A user asks a voice assistant to, “Play more music like this.”
A voice assistant might respond, “Ok, how would you describe this song in your own words?”
Or, after playing an algorithmically selected song, a VA could ask users why they think
it chose that song, then respond with the reasoning
behind the recommendation.
These types of interactions expose and build the system’s contextual intelligence, while
also teasing out a user’s preferences.
Second, we encourage a more user-centered approach to systems that log and apply user
data.
Recall the play count example, how number of plays could mean all kinds of things.
Users have valuable knowledge about what data metrics mean that are often unavailable to
designers.
In systems that log user data, there is often a disconnect between how data is collected
and how it is ultimately used.
Involving end users in the design of data logging capabilities could help address this.
And data-inspired design activities, with end users, can spark forward-focused, generative
discussions about what people can do with their data.
For more on this work, please check out our paper, “Giving Voice to Silent Data: Designing
with Personal Music Listening History.”
Thank you.