- My name is Erin Beneteau,
and I am a Speech-Language Pathologist.
I am also a doctoral student at The Information School
at University of Washington, like Matt said.
And I, and my co-authors, were all part of a research team
funded my Mozilla,
to investigate how families, who have children in the home,
use home digital assistance,
or smart speakers.
however you wanna refer to these,
for the first time.
So, we recruited 10 diverse families
who had never before owned or had a smart speaker,
digital home assistant, in their home.
And, we gave each of them an Echo Dot to use for one month.
Now, our families, we were so fortunate,
with our participant families.
They were incredibly diverse, not only in ethnicity,
but in family composition.
So, we had families of two person families.
And, we had families, up to five person families.
And, you can see that the children ages
were also quite diverse as well.
We had two families
that were fully bilingual, Spanish and English,
in which all family members spoke both languages.
And those are the ones marked with asterisks there.
We used Anchored Audio Sampling as our methodology
and there's a talk on that on Thursday.
So, we audio recorded interactions
between family members and their Echo Dot.
After one month's time,
we captured 14.5 hours of conversations,
and as the research team started to transcribe
these conversations, and started to discuss them,
we realized that all 10 of our families
had experienced communication breakdowns.
Now, that's not really a huge shock
if you've been reading the HCI literature
on conversational agents.
There is discussion about how these truly are not
conversational in nature.
However, this was such an exciting time for me
to bring in my Speech-Language Pathology background
and to analyze how these communication breakdowns happen.
So, just to make sure, we're all on the same page
for terminology.
A communication breakdown for this study
is defined as when a human talks to the Echo Dot,
or Alexa,
and Alexa does not respond back appropriately.
That's what we code as a communication breakdown.
And also, just because I'm a Speech-Language Pathologist,
and I wanna make sure we're all on the same page
for terminology, I'm gonna go through just a few basic terms
I'll be using for the rest of the talk.
In the paper, I go into these way more in-depth.
And, I'm always happy
to talk about speech-language stuff, too.
So, first of all, when I refer to speech,
I'm referring to how you produce sounds, so articulation,
and also the volume of your voice.
So, how loud you're speaking or how quietly you're speaking.
When I talk about language, I'm referring to,
kind of those words that you put in speech bubbles
in cartoons.
So, that includes syntax,
which is how you put your words together,
like sentence structure.
As well as semantics, which is the meaning,
or the vocabulary of the words.
And, within that larger domain of language,
there's a sub-domain called pragmatics.
And, I know that this word means something else
in different fields.
But, in speech language pathology,
pragmatics refers to the social use of language.
And, that involves non-verbals
like eye contact and body language,
but it also involves things like turn taking.
And, a concept that I gonna discuss a little bit more deeply
called code switching.
Now, we all usually engage in code switching.
Especially when we're at a conference.
We might be using very specific,
certain types of communication skills,
when we're meeting a colleague or a peer for the first time.
Versus, if you were talking to a one or two year old child.
You'd probably be using different communication skills.
Different speech and language abilities.
So, that's an example of code switching.
Okay, so now we are all going
to go through a communication breakdown together.
And, we're going to analyze this communication breakdown
from Family B.
And, Family B, is a family of four.
They have one child, aged 12.
And, in this particular communication breakdown,
three family members are present in the home.
So, to start off the mother says "hey daughter, hey father,
South Korea, what part of Asia.
Southeast or east Asia?"
And then you can here in the background,
it sounds like the family's in some other room,
they're talking and you can't quite make out what they say.
And the mother says "from America. What you...".
And then again, like off in the distance,
you can hear the family talking.
And the mother says "yeah".
And father says "Google is South Korea East Asia or South"?
And, in this case, the father is using Google as a verb
and he's providing a direct instruction
to the mother on what she should do next.
She should involve technology to answer her question.
The mother, modifies that direct instruction slightly,
and she says "Alexa!
South Korea is East Asian or Southeast Asian"?
And Alexa's response?
"Sorry, I'm not sure".
That is a neutral response.
And a neutral response is a Speech-Language Pathologist's
bread and butter
when we conduct evaluations and assessments.
Because a neutral response does not provide
any contextual clues
as to why the communication breakdown occurred.
So, when I am assessing someone's speech and language skills
I don't want to influence their natural
speech and language skills.
So, I provide neutral responses.
And in fact, in the speech language pathology literature,
back in the 1980s,
there was a really fun study
in which researchers were trying to determine
if there were communication repair strategies
that were different based on children's ages.
And to conduct this study,
the researchers used neutral responses.
"Huh?"
"What?"
"I don't know."
Again for the same purpose
that I use when I'm conducting an evaluation.
I don't want to influence someone's
natural speech and language.
However, when you're in a conversation with someone
a neutral response isn't particularly helpful.
So, after Alexa provides this neutral response
of "sorry, I'm not sure",
the daughter says "Alexa, are Koreans Southeast Asian?"
So, now we have two family members who've jumped in.
And, remember in the beginning,
we couldn't really hear the other family members.
So, now they've kind of moved in a little closer
to the Echo Dot.
And Alexa provides a beep as a response.
Now, this is like the ultimate in neutral responses.
What does a beep mean in this context?
So, the daughter and the mother,
both have a complete stop to their conversation.
Like, in this whole transcript,
you're listening and you hear all this talking going on
and then it's just silence, after Alexa beeps.
So, then the family collaborates.
The mother says "try again, daughter".
So, then the daughter does a great,
typical human communication repair strategy.
She increases her volume and she repeats.
"Alexa, are Koreans Southeast Asian?"
And Alexa responds, "I'm not quite sure
how to help you with that".
So, this is again, another neutral response.
The daughter says "hmm".
And then, the mother jumps in again.
And she says "Alexa, who is Eastern East Asia?
What country is East Asia?"
And Alexa says, "there are no UN recognized countries
in Eastern Europe".
And, this is great, for me,
as a Speech-Language Pathologist.
'Cause I'm like "Wow!
Something really interesting
has just happened here linguistically!"
The family's maybe not as excited about this response.
But, I think it's really cool.
'Cause this shows a couple of different things.
This shows how the family is scaffolding
on each other's language strategies
to try and repair this communication breakdown.
So, to start with, at the very bottom,
mother's original request is a two-part question.
Sort of like a multiple-choice question.
Then when that doesn't work,
the daughter uses a really good language strategy.
She contracts that question, and simplifies it
to a one part question.
A yes, no question.
It's a really great idea, but it still doesn't work.
So then, the mother and daughter collaborate
and try and figure out what to do next.
And then, ultimately, the mother refers back
to her very original two-part question.
But, she follows the daughter's model
and she breaks it up into two one-part questions.
But, in order to do that,
and to have them syntactically accurate,
or correct,
she adds in the vocabulary word country.
And that is really helpful for Alexa
to have some additional context clues
as to what the family is talking about.
So you can see here,
that Alexa's response picks up
on two key vocabulary terms used in the mother's request.
And as a result, Alexa acts on an understanding,
misunderstanding.
Not understanding.
She acts on a misunderstanding.
So, she provides a response based on misunderstanding
what the request was.
However, this is wonderful,
because it gives the family
at least some contextual clues,
as to why there is a communication breakdown happening.
However, the daughter's response is "Ugh"!
And the father increases his volume,
and he says "Alexa.
Please define East Asian countries".
So, even though the father hasn't been participating
this whole time,
you know he's been listening, because of the language
that he uses.
He continues that scaffolding process
and he uses the keywords
that are needed to get the correct answer.
So, then Alexa responds correctly.
And then she does a nice job
of some turn taking, for a clarification response.
"Did that answer your question?"
Mother says "yes".
And Alexa says "thanks for your feedback".
And then the topic has ended.
So, when I think about all these communication breakdowns
that we were analyzing,
and I think about my background
as a Speech-Language Pathologist,
it really hits home
that communication is hard.
It's a really difficult thing.
And, humans have communication breakdowns
between each other, all the time.
So, of course we are going to have communication breakdowns
with a machine.
So, I think that when you're designing
voice interfaces, there's one track where you're designing
to improve the accuracy of the voice interface.
But, there's this other track where you're designing
to improve the communication repair process,
by assuming that communication breakdowns,
are going to happen.
So, one of the ways you can do this,
is you can increase specific clarification responses.
You can increase acting on a misunderstanding.
I almost hate to say that,
because I know some folks are gonna be like
"oh, but that's so annoying"!
But, at least then, you're getting some context clues
as to where the communication breakdown is happening.
In addition, decreasing neutral responses
can be very helpful.
Because neutral responses don't give you context clues.
You're just fishing around in your toolkit
for human communication repair strategies.
And, they may or may not work.
Another thing that came to my mind
when I was reviewing all these communication breakdowns,
is that, remember these old time photos,
of people huddled around the radio,
listening in the family environment ?
Well, a form of that,
is happening with voice interfaces
in the home environment.
We see all three family members,
that were present at the home at this time,
coming in and collaborating
to repair this communication breakdown
with this voice interface.
And this also relates to something our keynote speaker
talked about today.
Were any of here, to hear about
the death dance of Jibo, the robot?
Some people, yeah.
It was, like, a heart breaking tale
she told she told of this robot,
that all of a sudden came online,
and said "you know, I'm gonna be leaving you now",
and did a little dance, and then it shut down.
And died.
And her comment was, "think about who else is in the room".
They had a five year old child.
And, to witness a robot, kind of doing this dance
and saying well I'm going to leave you, and dying,
could be a little alarming, or upsetting,
to a young child.
It sounded like it was alarming to the parents, as well.
And the same is true with a voice interface.
When you're designing a voice interface,
specifically for the home environment,
I think it's important to consider
that there's going to be joint engagement
between all of the family members
who are within listening distance.
And that's shown with scaffolding that we saw.
But, in addition, I wanna just suggest
that voice interfaces can help
families with this scaffolding process.
And , in fact, in our analysis
of communication breakdowns,
we did see, a few, positive instances,
of where Alexa helped support families
repair communication breakdowns.
But, in this case, there were communication breakdowns
on the part of Alexa not understanding.
And, she gave a very specific response.
For example, if I say "Alexa, set an alarm for 10:45".
Alexa would say "is that 10:45 in the morning,
or 10:45 in the evening"?
It's a really nice way that she can give a specific response
to clarify, so that she can understand
her human communication partner, clearly.
And then, finally,
I wanna return to code switching and pragmatics.
Pragmatics, the SLP defined way.
Of social use of language.
And, when I think about multiple generations
in a household, interacting with this voice interface,
I think it would be really interesting.
And I think possible, at a basic level,
for voice interfaces to be able to understand
if it is a small, young child interacting,
or an adult interacting with the voice interface.
There are some speech and language patterns
that are indicative of a young child speaking
versus an adult.
And, with code switching, if a voice interface
is able to identify those patterns,
and respond more appropriately,
to the age of the person who is speaking to them,
that also might help aid
in resolving communication breakdowns.
For example, we did see a lot with jokes.
A lot of children would asks jokes
and the jokes would not be age-appropriate
for that child to understand.
So, code switching, is a possibility.
All right.
I want to wrap up by thanking Mozilla.
I also really wanna thank
the two Speech and Language Pathologists
that helped me define and discuss
these different constructs
from speech and language pathology,
and kind of bring it into the human-computer
interaction space.
Which was really nice.
And, especially Teresa Fleck,
who talked to me a lot about,
bilingual communication, 'cause she speaks Spanish,
which is very handy as well.
And then thanks to the entire research team.
This was a pretty intensive research study,
with all 10 families, and they did so much work.
And then, thanks again to our participants.
And, if any of you are interested
in talking more about speech and language pathology (laughs)
it's such a great topic, I think.
So, I'd be really happy to talk to you about it further.
Thanks!
(audience applauds)
- Thank you Erin, we have time for a question.
And we've got one just down here.
- [Jen] Hi, Jen Tom from--
- Hi!
(static interference)
- [Jen] Hello?
Oh. - Hi, now I can hear you.
Uh-oh.
(static interference)
You know I used to always teach these courses,
and my mantra was
"technology will always fail - Okay.
- At some point". - Okay so.
- [Jen] Okay I think this works.
So, one of the reasons why, these systems
give neutral responses, is because their, the system itself
is not confident of the result, right?
- Mm hmm.
- [Jen] And so, you have a design recommendation
to, I guess, anchor on maybe some specific part
of utterance.
So there's, like, a little bit of this, like, tension there.
So, how do you think you would kind of address
that tension?
I guess, a little more specifically.
- Mm hmm.
Well, I'm not sure.
This is where I really need a computer scientist
to help me.
I'm not exactly sure what Alexa was encoding
in that example, in the first three times,
when she gave neutral responses.
But, if she had just given a little bit of a clue,
about anything, that would have given the family
at least some context, as to what she was interpreting.
So, that they could then adjust their speech and language.
Does that sorta answer?
Okay.
- Thank you very much, Erin. - Sure.
- So we're on to our final talk for this session. And--