(00:19): Hi, thanks for watching our talk on
embodied geometric reasoning with a social robot. We're really gonna highlight the ideas of embodied
cognition in this geometric reasoning and how that relates to our interactions with a social robot.
because our cognition and problem solving is embodied. some key findings that inform
this work are that we construct and manipulate situation models and problems we are working on.
For example, this student is reasoning about a geometry conjecture that has to do with angles
of a triangle. Her situation model is constructed that includes the triangle and those angles as she
sort of understands them, and we know that gestures are produced as simulated actions
that are related to this cognition. In this way our, we can use our body to represent parts of
the situation model, uh, in the real world. And because of that, our gestures can aid in reasoning
about problems that aid in communicating. Finally, we also find that we conduct mental simulation
with others thinking, in our understanding of what it is that they are trying to communicate with us.
creating a very social dynamic to the way that we understand the other people. That social dynamic
is important to the work we're doing here. (01:23):
because our interactions with robots, our social from this body of research,
we take some key findings to inform this work, particularly that the physically embodied nature
of robots will enhance the social presence of that robot and that our interactions with
robots are interpreted socially and that they follow social rules. We can summarize some of
this complex interaction through the cognitive human-robot interaction model, from Bilge Mutlu,
where things like the shape of the robot will impact our mental model of what that robot is
about, how it reasons, how it does things and that in turn will impact our actions.
Other things like a robot's actions themselves can impact our mental model that will have an outcome
on our actions. And that means that we posit this idea that a robot's gesture that could impact the
social presence and mental model. To explore this idea, we look at two different types of gestures
a robot might make. (02:15):
The first is a beat gesture that follows along with the rhythm of speech or emphasizes parts
of speech that, that do a lot of work to help us comprehend people's speech. They are dynamic
movements, but they are important movements that happen when people talk. We also look at what
we're calling dynamic representational gesture, where a robot might represent an object or an idea
and dynamically change that through its gesture. The idea here is that the type of gesture and the
use of gesture are gonna have effects on our, the social presence and mental model of our
robots that we interact with. Um, in particular, the math reasoning that we might do with robot,
the mental model will be influenced how we perceive the robot's, internal or mental. The
dynamic gesture from a robot may imply more genuine reasoning, activity and capacity for
reasoning than other gestures. And a dynamic gesture may actually provide more socially and
cognitively appropriate ways of communicating. (03:10):
Therefore, we would have a higher concept of a robot's ability to reason if it uses dynamic
gestures. And we might have a higher concept of robot's perception of our own internal states
that leads to the following hypotheses. That a robot using dynamic gesture is gonna lead
to higher visual attention, higher dynamic gesture, more accurate reasoning about proofs,
and a higher rating of it as a partner and the quality of its gestures. To test these hypotheses,
we designed a random random control experiment where 30 participants work with either a beat
gesturing robot or a dynamic gesturing robot, where in both conditions the robot would read and
reason about four conjectures. The participant would then follow up by reading and reasoning
about four conjectures. And then do a post study survey. The conjectures that we use
were all statements that describe geometrical relationships. For example, the area of a
parallelogram is the same as the area of rectangle with the same length and height. Participants were
asked to state whether that was true or false. (04:11):
And the reason about it in the last four and in the first four robot did that. For example,
the robot might reason with speech acts by saying, Okay this is tricky. Let me think about it. If I
have a rectangle, I can tilt the vertical sides of the rectangle to make a parallelogram. Okay,
I think the area does not change, because it is like a shifting the sides to make a parallelogram
and pushing it to the other side to make a rectangle. Note that these speech acts were
the exact same in both conditions and they were verbal utterances about the robot's reasoning
This reasoning was accompanied in the beat condition with beat gestures where the robot would
retract and extended its arms with multiple one or two handed pairs that would align with the robot's
speech to sync with words that it was going to emphasize, or the rhythm of the robot speech.
In the dynamic condition, the robot's speech was accompanied by dynamic gestures, where we
based these gestures on beneficial reasoning, uh, gestures from prior work. And we created in each
of these, a representation of the initial state. (05:12):
You can see on the left hand side of the pictures that were then dynamically changed to represent
test conditions. Each of the first four had its own custom set of dynamic gestures. You can see
align here and they accompanied the speech about the reasoning robot does. So what did we find?
Well, we found that those who worked with the dynamic, the gesturing robot looked more at
the robot, we believe this has to do with there being more information and better support for
following the robot's reasoning when it used dynamic gestures. And so people
visually attended to the robot more to get that (05:48):
We also found that the, use of gesture by the participants themselves was affected by
the condition where those who were in the dynamic condition had overall more gestures, uh, that were
almost entirely representational gestures, which is why these two bar graphs are very similar here
and of those representational gestures when they were then dynamicly manipulate,
this is again, the participant changing their gestures be dynamic. Uh, we found that those,
the dynamic condition, more dynamic gestures themselves, and finally under survey items, people
describe the robot as a better study partner. They prefer study with it in the future more
in the dynamic condition and that the dynamically gesturing robot had higher quality gestures.
Let's look a little bit more at the participants' gestures. Again, they used more representational
gestures if they're in the dynamic condition. And to give you an idea what that looks like
we can demonstrate here where one participant started by repeating the conjecture.
(06:41): The first part of it,
by saying, reflecting a point over the X axis here, she first represents the point with her hand
and then rotates that point around an X axis. Again, you can see how this relates to what
might be a simulation model. And then she takes a moment to consider this new point says, the second
part of the conjecture is the same as rotating at ninety degrees clockwise, and then takes her hand
to rotate ninety degrees to a final position to consider whether or not this is a true statement.
These types of gestures really were an important relationship
to the condition that they were in so that when people were in the dynamic condition,
they did more of this dynamic gesturing themselves. We think this is a really
important finding in that the participant may perceive the dynamic gesturing from a robot
is communicating at a level of sophistication. (07:34):
That's capable of creating and manipulating a mental simulation. In other words, the robot
is demonstrating. It is making its own mental simulation by having these dynamic gestures.
And therefore we actually use our gestures to help contribute to that robot simulation.
Where compared with the beat gesturing the robot may only be perceived as describing
facts or reciting the answer. So, we think these findings suggest that our mental model, of the
robot activity could be influenced by your perception of how robot can communicate and
that robots with better gesture communication, get more attention and are seen as better partners. We
encourage you to look at our, for more details about this, to out to us with any questions.