(ethereal music)
- Who here has an Amazon Alexa or a Google Home at home?
There's a few hands in the air, maybe not as many,
but that's what I want to talk about today.
So, I want to talk about these,
I like to call them screenless devices,
'cause they're quite different from the kinds
of devices and other kinds of voice interaction
that you might be used to, such as Siri on your phone.
So, these are the kind of devices you might have heard
about as voice first or voice-only interfaces as well.
So, they're products without a screen
that are pitched for the home, as well, which is important.
So, that's how they're sold to us,
but what we are interested in
in our research is the reality.
And so, that's why I'd now like
to introduce you to the Kent family.
So, the Kent family is a British family of four,
Susan and Carl and their two children,
Emma, who's about 12 years old,
and Liam, around 7 years old.
And we were interested in seeing
how they would use the Amazon Echo.
So, we recorded them for a period of a month,
alongside four other households.
And then, we examined the recordings,
and that's what we built our research around.
And that's what I also want to do today,
is to share some of those recordings with you.
So, in the first recording I want to show,
we're actually joining the Kents for dinner.
So, the Kents, they're having a dinner here.
They're having a takeaway meal.
It's Mother's Day on this day,
which is relevant as you will see,
and also the takeaway is Susan's choice.
The Echo is on a sideboard near the table
where they're eating and they've, at this point,
had the Echo for about a week, so bear that in mind.
And they often use the Echo for
listening to music, and also using skills
like the Quiz Master and the Beat the Intro.
So, Quiz Master's a trivia quiz,
and Beat the Intro plays a few seconds
of a song that you then have to guess.
So, before I play the clip, just a word on transcripts.
We use these kinds of transcriptions to analyze recordings.
We do this because we care precisely
about what is being said, and this is a made-up conversation
here, but I just wanted to introduce you
to some of the things that we use to analyze it.
So, for example, we use these,
these numbers here are pauses,
so this is a 1.5 second long pause.
This is a micro-pause, so something
around 0.2 seconds or so long.
This here indicates that talk is overlapping.
So, actually this hi there and how are you
are actually said at the same time.
And this means that something is being said,
but we can't quite make it out or we can't quite hear it.
And then we can also transcribe
relevant nonverbal actions like so.
So let's play this recording now.
- [Susan] Alexa, beat the intro.
(family chattering)
- [Man] Alexa, Alexa, play beat the intro.
- [Alexa] Resuming the music.
(crowd laughing)
(upbeat music)
- [Carl] Alexa, stop.
- Okay, and it continues.
- [Carl] Alexa, Alexa, play beat the intro.
(crowd laughing)
- [Emma] No she didn't like that.
Alexa, Alexa, play beat the intro.
- [Alexa] You want to hear station for B-B intro, right?
(crowd laughing)
- [Emma] No, I don't Alexa, no!
- [Alexa] Alright.
- [Carl] We played it the other night,
the game we played the other night.
(crowd laughing)
- [Susan] Skills beat intro.
(crowd laughing)
- [Emma] She didn't like that.
(crowd laughing)
- [Susan] Alexa!
(crowd laughing)
- [Susan] Alexa, beat the intro.
- Right, so now you know what using an Alexa is like.
(crowd laughing)
I think, it's fair to say what stands out about this data
that I've given you an idea of,
is that, you know, it really doesn't work very well.
It's about failure, really, here.
And how the Kents are struggling to get this thing to work.
Actually now, in a study,
we find that about 30 to 50 percent of the time,
people are struggling and are failing to basically
use the Alexa in a way that they intend to.
So, this means failure is really a normal feature.
And actually, it's important to understand how designers
can help people for example, recover
from these errors and actually move on.
And so what I want to do now is take a closer look
at what happens when something goes wrong.
So let's look at the segment again that I've played to you.
- [Susan] I'd like to play beat the intro in a minute.
Alexa, beat the intro.
- [Liam] No!
- [Susan] Alexa, Alexa, play beat the intro.
- Okay so, what's happening here then.
Let's step through this.
So Susan is trying to instruct Alexa to play Beat the Intro
and there's a bunch of other stuff
that's going on at the same time.
What she does at the beginning
is she announces her intent that
she wants to play Beat the Intro,
and this occasions Liam's initial 'oh no'
and then Susan presses on regardless
with her instructions, saying 'Alexa beat the intro'
and in turn, we get Liam's making
his disapproval very clear, don't we,
about the prospect of playing beat the intro.
- [Liam] No!
(crowd laughs)
- And then about 0.6 seconds into Liam's 'no,'
we get Carl saying, 'It's Mother's Day.'
And that's interesting, because what Carl is attempting
to do here of course is to get Liam to comply
with what Susan is doing with the Echo.
So after this idea of after all it's Mother's Day,
it's a recourse to social norms
and norms come with rights and obligations
and in this case, because it's Mother's Day,
it really is Susan's right to decide
to play Beat the Intro, and it's Liam's obligation
to play along with that, whether he likes it or not.
So, my point is that there's a lot going on here
at the same time and all of this
is embedded in the social goings-on
and everyday life of the Kent family.
And then, Susan also then kind of seamlessly
switches from addressing Alexa to
then telling him to eat up, to say to Liam,
you need to keep on eating your orange stuff.
So Susan is, you know, very easily switching here
between using Alexa and parenting,
if you want, doing parenting activities.
And then what follows next is this joke
that is coming up here between Carl and Emma
and Liam as well about Susan's turn of phrase
when she said keep on eating your orange stuff,
you know there's 'and your green stuff,
and your brown stuff, and the yellow stuff,
and the meat stuff, yes', so this is all happening
as part of this interacting with Alexa.
So the point that I'm trying to make is
that voice interfaces are designed
with a single user in mind, but actually
in the real world, things are different
and there's a lot going on at the same time.
And we call this kind of multiactivity settings.
And these are really the norm,
and these settings really create design challenges
for voice interface designers.
Okay, so moving on from that, what I'd like to do now
is take a look at two kinds of failures that go differently.
So this is from a different moment later on
where they're trying to start the quiz,
and let's especially have a look at how Alexa responds,
and how the family deals with the trouble in that response.
So, again, let's listen to this.
- [Emma] Will you ask for a normal quiz?
- [Susan] Alexa, set us a family quiz.
- [Alexa] Sorry, I can't find an answer
to the question I heard.
- [Emma] Alexa, set a family quiz.
- [Alexa] Sorry, I don't have the answer to that question.
- [Liam] Alexa, please set family quiz.
- [Alexa] I wasn't able to understand the question I heard.
- [Carl] Alexa, family quiz.
- Okay, so they do eventually get
the quizmaster skill opened at some point later
but for now, let's just have a look again
at how this plays out.
So we've got Susan starting this here
with the instructions 'set us a family quiz'
and then if you look at the response,
the response is 'I can't find the answer
to the question I heard.'
Well, so the response categorizes
what Susan said as a question,
really, not as an instruction.
And this is important because talk is sequentially organized
and one utterance follows another
and a present utterance allows a person
to make a future one, right, but in this case,
there's an error message here,
but it's really not very clear
how to proceed from that error and to recover from it.
And so what happens next is Emma really doesn't have
many places to go after this then.
She sort of repeats the instruction
with a slight variation, something we see very often
when people are trying to deal with trouble.
Again Alexa responds with this miscategorization,
'I don't have the answer to that question.'
And you can see where this is going.
Although when Liam actually turns this
into something that's amusing when he says
'Please set us a family quiz.'
But again the response is not very helpful here from Alexa
and finally, an even more pared down attempt by Carl
just to sort of narrow it down to the sheer essence,
when he says 'Alexa, family quiz.'
So the point is that if you look at these turns here,
and what you see is family members
working together to try and get this device to work
but the response being unhelpful,
and it effectively not helping the family
to recover from the problem that's occurring
and moving on and sort of being stuck here.
So we could say that responses from Alexa
are treated as resources by users for further action, right?
And this is quite important.
So when Susan says, 'Alexa, set us a family quiz'
and Alexa responds, 'I can't find the answer
to the question I heard,' then that's not really helpful
and that doesn't really give Susan
resources to move on with her request.
And she might have even thought something like, hm,
that wasn't even a question, not sure what to do next.
So, okay, let's look at a slightly different and final
piece of data here, another way how this can turn out.
So this is something we've heard already
but I'll play it again just to remind you.
- [Emma] Alexa, play beat the intro.
- [Alexa] You want to hear a station for B-B intro, right?
- [Emma] No, I don't Alexa, no.
- [Alexa] Alright.
- Okay so, how is this different?
Well, Emma instructs Alexa to play beat the intro
but again let's have a look at Alexa's response
and what it lets Emma do next.
The response actually incorporates
a transcription of what was heard,
which is B-B Intro, which is of course, a mistranscription
so it's an error that has occurred,
but it's being made visible and it's being made
hearable to Emma, and as a result,
building this into the transcription
and offering also this kind of next action
in the form of a question, you know,
the action being to 'hear a station'
and asked as a question, 'right?'
Actually just allows Emma here to vocally
and very quickly say no, 'no I don't Alexa, no.'
So it allows Emma to move on and make the next move
and then this sequence is sort of drawn to a close
when Alexa just responds 'alright.'
So although it's not successful, they've managed to move on
and close the interaction, if you want.
So the point here is that the response design
we've seen in that last example
is quite different from the previous one,
in that it gives users the interactional resources
to move on and sequentially to do the next action.
And, actually to progress with
what they are trying to get done.
So, it seems important to consider then that
or how to explicitly design these resources
and embedding them in responses.
And so the responses enable certain kinds
of possible next moves in sequence
and as designers, you know, they need to think about
how responses enable this progressivity
to move on towards getting something done with the device.
Okay so concluding, what have we learned?
Well we learned that for voice interface
to become truly embedded, they need to be able
to better cope with these sort of
social multiactivity settings such as a family.
We know that talk is sequentially organized,
and what I tried to show in this talk
is that responses are resources for next actions
or next moves.
Thus, you can start to think about responses as,
or responses should be carefully designed
to enable people to move on towards
getting something done, towards the goal they have.
So overall, my talk should have shown you
that voice interaction design really has
quite a long way to go, so don't let the marketing fool you.
And what I presented is based on this
CHI paper that's available for download
if you're interested, or email or tweet me as well.
Thank you very much.
(Audience applauds)
(ethereal music)