[MUSIC PLAYING]
LUKE DAVIS: Welcome to "Assistant on Air,"
where we have conversations with the people that
build for the Google Assistant.
I'm Luke Davis, a developer product
engineer for Actions on Google.
Today we'll be chatting with Cathy Pearl.
Hi, Cathy.
Why don't you tell everybody about yourself?
CATHY PEARL: I'm the head of Conversation Design
Outreach at Google.
And what that means is I go out in the world,
and I talk to people about what is conversation design, why
it's important, and best practices for how
to do it well.
LUKE DAVIS: I remember watching last year's I/O,
and seeing the Duplex demo.
And after the fact, I was thinking about Duplex
and how interesting it was, how compelling it was,
and that it wasn't actually the speech.
It was the pauses.
And even now, like a year on, having had this realization,
I don't think that I've incorporated
that idea into the voice user interfaces that I've built.
I don't see that being done very broadly.
But when I see it done, it has such an interesting impact.
CATHY PEARL: Yeah.
Related to that, sometimes we'll get
feedback where someone says, oh, a user said it's too fast.
We have to slow down the TTS.
And we say, hold on.
Often the issue is really not the speed,
which was created specifically at the correct rate
of normal human speech patterns.
It's often, like you said, the breath patterns.
A TTS computer doesn't have to take breaths,
so you can just keep having it talk, and talk, and talk,
and talk.
But that's hard for people to listen to,
and so you do have to break it up in these more natural ways.
I'm thinking about pausing.
We carry a lot of information in silence.
When we're having a conversation,
the time between our turns is like blink of an eye.
It's 200 milliseconds.
LUKE DAVIS: Wow.
CATHY PEARL: But if you wait longer than that,
that is information.
So if say to you, hey, can I have a ride on Friday,
and you pause for more than one second, guess what?
You probably don't want to give me a ride.
So even though it's your turn, I'll
probably take my turn back and say, oh, OK.
You're probably busy.
And you didn't say anything.
You just paused for one second.
And that's something that is very hard for computers
to do right now.
And so, again, as a conversation designer,
you're taking what you know about the way humans speak,
but you're cramming it into a very low signal.
All you get are the words.
You don't get eye gaze.
You don't get body language.
You don't get pauses.
You don't get very good intonation.
So you're having to cram all this meaning
into just the words.
And that is where it takes some skill to really think about
how can we take how humans talk but actually implement it
with the real technology that exists today.
LUKE DAVIS: Wow.
That is super interesting, that the messaging of the silence
is something that I have totally not thought about when
building these things before.
I guess part of it is that I'm very visually oriented,
despite the fact that I have to wear glasses.
That assumption that, oh, all the important information
in an interaction is going to be visual, so very little of it's
going to be auditory, is a thing that I actually feel
is handicapping my ability to approach these things.
You happen to know a lot more than I
do about this world of accessibility.
Do you happen to have any numbers on this
that you can share with us?
CATHY PEARL: So there are about 285 million visually impaired
folks in the world.
And, as you say, one of the things
when I think about designing for accessibility
is this idea of inclusive design,
which is that at some point or another
we all have impairments.
Whether it's oh, I don't have my reading glasses
and I can't read the text message on my phone,
or I'm carrying in some groceries
and I temporarily can't use my hands,
or I fell and broke my arm.
There's all these parts and things happening in our lives
where we ourselves are impaired.
And so I think of it more as we're
designing really for everybody.
We might think of it with someone
who maybe has motor or mobility impairments,
or visual or hearing impairments,
or cognitive impairments.
But we all have ranges of, say, memory loss.
And so to me, it's really building
for everyone in the world.
And voice has really started to take off
in this arena of really being helpful to people.
So you can imagine, of course, things
for visually impaired folks, whether it's
being able to choose their music more easily,
be able to call the pharmacy without having to go
through more laborious steps.
Even I was watching this video about someone talking about--
he's visually impaired and how he uses the Google Assistant.
He was saying he can use his phone, which
has accessibility options, to do things like set a timer,
but what's really hard is to find out how much time is left
on the timer, something we very much take for granted
but is harder if you're visually impaired.
But now he can just ask, how much time is left on the timer.
And these kind of benefits, I think, are just amazing.
LUKE DAVIS: That's really cool.
I had no idea that it was over a quarter of a billion people.
CATHY PEARL: Yeah, I know.
It's really a lot more than we might think of.
LUKE DAVIS: Since I have you here,
there's something that I wanted to ask you about with regards
to some of the things you did before you came to Google.
So you worked at a couple of companies,
doing voice user interface things before,
but this was in the era before we used really heavy NLU-sorts
of things with deep neural nets and all
the fancy new buzzwords.
And my suspicion, though I need you to confirm or falsify this,
my suspicion is that the abstractions
that you used in those contexts, when
you're dealing with some kind of phone system--
you worked on a medical system as well, right?
The abstractions, the key ideas, the things
that you talked about a lot, those
were different than the things that we talk about now,
where we have a much more flexible way
to interpret user utterances.
Can you speak to that at all?
CATHY PEARL: I think people might
be surprised about how many things--
so it's almost 20 years to the day
that I started working at a company called Nuance
Communications on automated phone systems, which was where
speech recognition was first mainstream, first
made available to the public.
Calling your bank, or your airline, or things like that.
And I spent a long time there really
learning the ins and outs of how to craft
these conversational interfaces.
And it's true the technology was not as good.
The speech recognition was not as good.
The natural language understanding was not as good.
We had to handcraft our grammar.
So, for example, if we asked the user a yes/no question
like do you want to book this flight, I had to hand write
uh, um, please, yes, yeah, all of the things
that could possibly come out of the caller's speech.
But honestly, so many of the best practices we learned
back then still apply to our voice user
interfaces and conversational systems today.
And I think sometimes people feel
like, oh, it's all brand new.
It's so different.
We don't have to think about those ancient, automated phone
systems.
But in fact, we learned all sorts
of things about how to present information,
how to confirm information, how to make the user comfortable,
all these different things that, in many ways,
really still apply.
LUKE DAVIS: Can you talk about that confirmation bit
a little bit more?
CATHY PEARL: So one of the things
about when we're talking to somebody else
we have a habit of knowing when and how to confirm information.
So if you're at the bank, and you're
like, I want to withdraw $500, the bank teller
is going to say $500, is that right?
Something like that, and make sure that they really
have it correct.
So that kind of thing, where you stop and force
the person to either say yes or no, that's
called an explicit confirmation.
I'm saying oh, you want to go to Austin?
Is that right?
I'm waiting.
I'm not going to do anything until you've either
told me, yep, that's right, or no, it's actually Boston.
Now if it's something that is maybe not quite as important,
like you just said what's the weather in San Francisco,
if we get it wrong it's not the end of the world.
I might just say, OK, San Francisco.
So I am repeating what I heard from you,
but I'm not stopping the conversation to say,
did I get that right.
I'm just saying, OK, San Francisco.
So you heard implicitly.
I heard what you said.
And then you could say no, no, I really
meant Santa Clara or something like that.
LUKE DAVIS: Interesting.
CATHY PEARL: So those are the implicit versus explicit models
of confirmation.
To have a really good conversational interface,
you want to apply them in the right place at the right time.
So you don't always want to do explicit.
You don't always want to do implicit.
It depends on the situation at hand,
which one you want to use.
LUKE DAVIS: This feels like a kind of cousin
to error handling to me.
Is there a similar sort of abstraction
in conversation design to error handling in programming?
CATHY PEARL: Yeah, for sure.
I think what's similar about it is that what you're getting
is that it's this idea of making sure the person feels
understood.
Just like when we're having a conversation,
if you're saying something to me and I've kind of got a confused
look on my face, even if I don't speak you'll probably
repeat yourself or say, oh, did you know
what I meant by that thing.
It's really important, throughout a conversation,
to make sure the other person is on the same page.
You don't wait till the end.
We don't wait til we're into this whole thing
and I say, you know, I didn't understand
that second thing you said.
No.
I'm going to give you some feedback that I didn't
understand something earlier.
And we call that repair.
So, repair, error handling, kind of similar.
But repair is a little nicer, because if I was saying,
oh, I was talking to Luke last night
and we had three errors in our conversation--
I wouldn't say that.
Because we're so good at repair and recovery
even when we don't understand each other.
And so to build a really great action,
if you have good repair, similarly, you
can get things back on track.
So something about voice user interfaces,
and natural language in particular, that's different
than, say, a mobile app or something visual
is that when someone taps a button or swipes,
you know what they did.
With voice and natural language, it's
a little more sort of an educated guess.
We're pretty sure what you said, but--
because of this, and because of the huge variety in the way
that people ask for things, we have
to do a lot more of error handling or repair,
where people are going to say things we did not expect.
Even if you're an expert in this space,
you'll design your system, you will still
be surprised by the ways that people say things.
I used to work on a health app, and it was a symptom checker.
And you can only imagine how many ways
someone can say abdominal pain.
I mean, so many different ways to say
that that I couldn't anticipate, so I
had to do more data gathering.
But the point is when things go off track-- because they will--
you got to help the user get back.
There's something called the cooperative principle, which
basically says that when we're having a conversation
we are really trying to succeed.
I'm not trying to talk too much or too little.
I'm not trying to use words that maybe you
wouldn't be familiar with.
Your user really wants to succeed with your action.
Help them.
So if they say something you didn't expect,
don't just say I didn't understand.
What are they going to do with that?
They're going to have no idea how to recover, how to move on.
So you might say, oh, did you want dark chocolate or milk
chocolate?
You get another error.
You might say, I need to know what kind of chocolate
you want.
You give them more information.
So help them.
Don't just left them out in the dark, because they're trying
and you want to help them get back on track.
LUKE DAVIS: So in a normal application
I might build, when I throw an error that bubbles up
to something that maybe--
usually it makes my program crash,
but we want to do a different thing here.
It sounds like what you're saying is maybe the default
posture that we should have when building these things is--
correct me if I'm wrong about this--
is that when we face some ambiguity
on the side of the action in terms
of interpreting the user, what we should do
is maybe fall back to exposing the capabilities of the action
at that point in the conversation?
Is that approximately right?
CATHY PEARL: Yeah, exactly.
And I think one reason we try to use the word repair rather
than error handling is because, like you said,
in programming error handling has
a very specific connotation.
And here we're just saying you had an unexpected user
response.
You're going to repair the conversation by giving them
context sensitive information.
Sometimes I see people say, oh, I've got a fallback intent.
We're good to go.
And they have one for the entire action.
LUKE DAVIS: You mean that's not good enough?
I need to go rewrite some things real fast.
[LAUGHS]
CATHY PEARL: And what we really need
is the person needs help at each turn in the conversation.
So if you're asking about x, then your repair,
your fallback needs to address that particular thing
because over here they might be thinking
about a totally separate thing.
And one other thing I'll say that's really important,
that a lot of people overlook, is this idea
of adjacent responses.
So let's say you're booking a table at a restaurant,
and I say, how many people in your party?
Now, you've built your responses to include between 1
and 20 people, whatever.
But I might say, well, do you have outdoor seating?
So I didn't say a random thing.
I didn't suddenly say, what's the average temperature
of Antarctica, because people don't talk that way.
They don't just randomly throw intents out all over the place.
But I'll say something similar, like I want
to know more about the seating.
And so you should be able to handle those types of things
in your conversation.
LUKE DAVIS: To go back to the repair bit a little bit,
would you recommend actually having
a fallback intent for each specific context
inside the conversation?
Is that generally the best practice that you see?
CATHY PEARL: Yeah, for sure.
And I know that's some extra work,
but really, it's this really crucial part of design
and of building these things because,
again, no matter how careful you are,
people are just going to be people.
They're going to speak the way they speak.
They might not hear the question.
They might be busy talking to someone else.
They might not understand the question.
They might say it in a way you didn't expect.
It's going to happen.
And again, if you can swiftly get them back on track--
sometimes I've seen usability testing
where I'll say, well, when that error happened,
what did you think?
And they're like, what error?
Because you guided them so nicely,
it was just part of the conversation.
It wasn't like error, error, error.
So it's worth spending some time on.
LUKE DAVIS: Thank you so much for coming to join us
today, Cathy.
We hope you liked today's show.
If you have suggestions for future episodes,
let us know in the comments below.
Follow us on Twitter for updates and new episodes.
I've been Luke Davis.
Chat with you next time.
[MUSIC PLAYING]