Welcome to this presentation, which is part of the proceedings of HRI 2020.
My name is Hannah Pelikan and I work together with professor Mathias Broth and professor Leelo Keevallik at Linköping University in Sweden.
We studied how people make sense of a Cozmo robot's displayed emotions.
So what comes to mind when you think of emotions? When have you smiled for the last time?
Robot emotions are usually studied under controlled circumstances.
Study participants are asked to categorize particular robot behavior or to rate how well it is recognizable as a particular emotion.
But what about robot emotions in the real world? How do people actually react when a robot shows emotion behavior?
To answer this question, we collected and analyzed video data of human-robot interaction in people's homes.
We worked with four German adult pairs who interacted with a robot for a short time while the researcher was in control of the robot.
In the second study, we worked with four Swedish families who kept the robot for several days and videotaped themselves.
We only recorded them on drop-off and pick up of the robot.
All participants signed informed consent before any recordings were made.
To get more insights into the designers' perspective we interviewed the robot's sound designer.
When analyzing the data we took a multimodal conversation analysis perspective,
which focuses on participants' demonstrated understandings of actions on a moment-by-moment basis.
We worked with a Cozmo robot – a commercial toy robot, which is developed by Anki.
It has a variety of different animations involving sounds, animated eyes and movements.
It can recognize faces and its toy cubes, but it does not actually have speech recognition.
Cozmo is controlled through a smartphone app, which allows for different modes of play.
Before we dive into the videos from family homes, I want to address one question. Why should robots display emotions at all?
There's been different approaches
but the most important one for this paper is the quite recent argument in HRI
that emotions are not just nice extras or random outbursts, but they are crucial in calibrating mutual understanding.
Conversation analysis has problematized the relationship between emotion as a public display and what you actually feel inside.
It is only the displayed, the public emotions that others have access to and can react upon.
So we don't know that somebody is sad until they show it.
You might feel sad inside, but you can still laugh about a joke.
We focus on how people achieve emotions socially and interactionally and how emotion is co-constructed by several people.
Take the example of the joke: people laugh together.
Emotion in human-human interaction is not a random outburst
but usually a response to a previous event. It is also interpreted in the context of such previous events.
So for example laughter ABOUT the joke that just came before.
So this is what we know from human interaction. Let's see how this plays out in human-robot interaction.
We picked one happy and one sad animation and analyzed them in real-world settings.
The happy animation that I'm going to show you occurs towards the end of a name learning activity where Cozmo learns a person's face.
While this is happening, people have to stay idle. As the sound designer explained
at the end of this long activity they tried to indicate that Cozmo is happy to meet you.
((happy animation))
Let's see how this animation looks like in the real world.
Try to focus on how participants react.
Warning: There's a bleep at the beginning of the video, participants' names are bleeped out.
So this adult pair played with the robot for about 20 minutes and then they called their son
to come and see the robot as well
and they had Cozmo learn his name. So the bleeps were where Cozmo said his name.
I'm gonna walk you through the transcript of this video very quickly.
Cozmo says the name twice. This is where the bleeps were. I called the participant Jonas.
They smile, the mother laughs and then Cozmo plays the happy animation.
In response, the son is laughing (you can hear his laughter on the video).
But what I want to stress here is that we get a nonverbal response. We don't get any talk.
He doesn't say anything back to the robot.
We just get this "ha-ha" and then the interaction moves on. The dad changes topic and asks about another robot.
We have 26 instances of this happy animation and in the majority we get nonverbal responses like laughter
and the interaction just continues smoothly.
Let's have a look at the sad animation, which occurs in the context of a fist-bump activity.
Cozmo will lift its arms trying to get you to bump your fists against them.
If you're not quick enough or no fist bump is detected, Cozmo will play this animation.
((sad animation))
Again let's check how this plays out in the real world and try to focus on how participants react after the animation.
So again, I'm gonna walk you through a quick transcript.
The mother is trying to explain to her little son that he should do a fist bump.
The son is too slow
He tries to actually fist bump his mother and then Cozmo initiates the sad animation. We get this "wao" sound and
already, then the daughter offers her explanation. She says he is sad.
We get a verbal response. Compare this to the happy animation where we just get laughter or another embodied response.
We don't actually get a verbal response in the happy animations
whereas here she verbalizes her explanation of the robot's behavior.
As the interaction continues, Cozmo plays the rest of this animation and
the mother responds "m-yaa, are you sad?" and starts petting the robot.
I'm not so interested in the comforting and petting
but what's interesting is that she too demonstrates how she makes sense of the emotion display in a verbal manner.
They both suggest that Cozmo is sad.
Interestingly enough it doesn't end here.
They don't just move on but the mother actually jumps back and offers to do a fist bump with Cozmo
reconsidering what has happened before and trying to provide the missing action.
We have 11 fist bumps that involve this sad animation
When people knew that the robot was trying to get them to do something
they reacted in surprisingly similar ways.
We have another instance with a Swedish family with a mother who does it almost exactly in the same way:
comforting the robot, trying to fix the trouble by doing the fist bump.
So, what do we learn from all this?
Happiness seems to imply that interaction can move on so we can use it as a go-ahead signal.
Sadness in contrast makes people reconsider their preceding actions. So it may function as an interactional rewind button.
People interpret emotion displays in the context of immediately preceding actions,
You saw it with the fist bump: if there's a sad animation just after a fist bump
people will try to fix it by doing the fist bump.
Make use of this context.
Take the sequential context into account when using emotions in your design.
For example when you learn a person's face and you want to indicate that it was successful
play the emotion animation right after to indicate that the action was successfully completed.
And then you can move on saying the name and all other things.
Final observation for this presentation.
Emotion displays have no rigid meaning. They leave room for people's interpretation
and their interpretation may differ from the designer's intentions. (see paper for examples)
We can use emotions to communicate quickly in a semantically underspecified way how the interaction may evolve in general terms.
We suggest that emotions can actually be used to communicate quickly and vaguely.
For human-robot collaboration that could mean
using an emotion display can help to quickly communicate that there's trouble.
Of course it's vague and you don't know immediately what it is about
but if necessary you can add a slower and more specific verbal explanation afterwards.
We contribute a study of robot emotion displays in everyday interaction in people's homes.
We demonstrate that emotion displays can have a range of different interpretations and they might not always just fit one category.
But that's not a problem.
Emotion can be used as a resource for communicating quickly and vaguely and in that
Happiness might work to move the interaction forward whereas sadness can work as an interactional rewind button that indicates trouble.
Thank you for watching and don't hesitate to contact us if you have any questions or comments.