Hey, thanks for watching! My name is Jan de Wit. I'm also here presenting on behalf
of Arold Brandse, Emiel Krahmer and Paul Vogt. We're all from Tilburg University in the Netherlands.
I'm here to talk about human-like gestures for social robots
in the context of second language learning. But before we go into the current study
I would like to take you back to 2018 HRI in Chicago where we presented
kind of a first study that led to the current study in which we had children
aged 4-6 years old, native Dutch children, interact with a humanoid robot
in the context of a game of "I spy with my little eye"
so in Dutch, the native language, the robot would call out
"I spy with my little eye" followed by an English word that we wanted children to learn
and in this case there were six English animal names included.
So then the child would have to find the matching image to this word on the tablet screen
and tap it to answer this round of "I spy with my little eye" correctly.
And we played thirty of these rounds, and children in this study were split in two groups.
One received iconic gestures from the robot and these iconic gestures were related to
the concepts they had to answer on the tablet and the other group did not
receive these gestures. The gestures looked a bit like this one for "horse".
So, we saw that if the robot would use these gestures to support its teaching
activities we saw that children learn more and that they also resulted in a
higher level of engagement with this interaction while learning these words
with the robot. So we followed up on this study in 2019 with a larger scale
study in which we didn't use the game of "I spy with my little eye" but we had kind
of an overarching storyline where children would have to perform all kinds
of tasks on these virtual 3D scenarios like in this case the zoo where the
animals had escaped from their cages and they would have to be brought back to
their cage and we use these kinds of scenarios to teach children words like
'add', 'take away', 'one', 'two', 'three', 'four'.
The gestures in this case looked a bit like this, this is a gesture for 'add'.
And we noticed in this case that there was no effect of these iconic gestures
on learning gains so I created a small overview of comparing these two studies.
We see that the second one, the 2019 one, is of larger scale.
The target group was a little bit more narrow so we only
included five-year-old children. It was a long-term study so we had seven sessions
rather than the one interaction with the robot, and this interaction was also more
complex so you would have to drag and drop objects on the tablet screen,
repeat after the robot, and of course since there were more sessions we also had
more words included but moreover these words were a bit more abstract so these
were things like 'next to', 'more', 'jumping', 'add', compared to these very concrete
animal names that we used previously. This also resulted in gestures that were a bit
more exaggerated in the first study so you see here also the robot is standing
so it's able to use its entire body to perform these gestures while in the
second study the gestures were a bit smaller, the robot was sitting more in a
peer-like fashion, close to the child, so it was more limited in performing these gestures.
So to follow up on this we decided to kind of replicate this first
study but using a bit more complex words from the second study.
Also what we noticed in the meantime was that if you look at human gesturing behavior
there is some variation there. As a very concrete example you see that people
perform gestures for example for the word 'pencil' from the character viewpoint
so you actually become the object, you become the pencil, but you can also do
this from the observer viewpoint. The character viewpoint
is more of a child-like way of performing these gestures: it's bigger
it's more like becoming the object
while the observer viewpoint is something that we as adults tend to do so you're
trying to take this outsider looking in point of view for example by showing
that the pencil is a cylindrical small thing followed by the act of writing
with this pencil. So we see that there is some variation there.
In the present study, first of all we wanted to replicate what we saw in 2018, so we went
back to basics with the game of "I spy with my little eye", but instead we
included a more diverse group of target words so not only animal names but also
some things like 'bridge' or 'stairs' that are maybe a bit more complex and also
harder to find gestures for. We included three conditions. Based on the previous study
we wanted to again compare between a robot that performs gestures and one
that doesn't, but we also included a condition in which the robot performed
different gestures for the same concept to see if it helps if the robot uses
these different strategies and focuses on different aspects of the
concept. All of these gestures were taken from a data set of human-performed
examples that we recorded last year. I'm gonna give you an example for the
word 'stairs'. You see that there are some variations in the way people in our data set
perform gestures for the word 'stairs'. This was outlining the shape of the
stairs, this is still kind of showing off from this observer viewpoint what stairs
look like, now we move more towards the character viewpoint where you see a
person actually climbing a ladder which is kind of comparative to stair climbing
and also actually walking up stairs but at the same time indicating that it's
something that's high up, and finally this is again from the observer
viewpoint showing that stairs go up and that they are big and high.
So if we look at the results, we see that children learn in all conditions and this is consistent
with our previous work where we showed that by training these words with the
robot children learn new words. On average they learned a little bit more
than one new word, measured immediately after the interaction. If you measure
again one week later they know a bit more, so they learn on average 1.4
new words in English however we don't see any significant differences
between the conditions with and without gestures. We do see that gestures lead
to more engagement, that's also consistent with what we saw in 2018.
We divided this into task engagement and robot engagement and we measured this by
taking small clips from the interaction at round 4 and round 24 of
the game of "I spy with my little eye" and you see that, as can be expected, the
engagement drops as you go. It's a very repetitive game, it gets
quite boring, so you see that children get bored with this and they lose
engagement, so that's normal. But if you look at robot engagement you also see
that overall the average engagement through time is higher if the robot uses gestures.
Whether this is repeating the same gesture or using variation that
doesn't really matter, it's higher in both. Because we see that people of
different ages have different strategies when it comes to gesturing, we also
looked whether there's an effect of age and indeed you do see that there is an
effect of age when it comes to using gestures to learn English words.
What we did here was we plotted the age in months at the time of the experiment against
their performance on the different post tests so this is the immediate post-test
and this is the delayed post-test, and we took the difference score so we
subtracted between the delayed post-test or immediate post-test and the pretest
scores. So this is learning progress basically, that's why you can also have
negative values here. And what you see here, and this pattern becomes especially
apparent in the delayed post-test, is that children who were in the conditions
where the robot used gestures, if they were older they seemed to benefit from
these gestures so this is the the no gesture condition you see that the ones
that receive gestures perform better whereas the younger children in our
sample seem to have even a detrimental effect of these gestures so you see that
they score lower than the no gesture condition and this effect of age is
significant but only for the gesture condition so the slope of this line
without gestures is not significant. To conclude we see that gestures do
appear to work but only for the older children in our study. The younger
children they have some problems with the gestures and we're not sure yet
exactly what the cause is but we think they either might not understand the
gestures yet, they need to develop this skill, or they struggle with the fact
that it's a robot and not a human performing them, or maybe there is some
sort of cognitive overload of having to learn a second language
while at the same time observing these gestures and taking advantage of them
We also realize that it's useful to now revisit previous studies with the
knowledge we have now, for example this effect of age, so we did do that and
we do see the same effect only for the gesture conditions in our previous study
in 2019, and also in 2018, so also there the older children benefited from
gestures more than the younger ones. When it comes to variation we didn't see
any additional benefits compared to repetition of gestures however we're
very curious to see if this maybe affects the perception of the robot so
if you have a robot that varies its behavior more maybe it's perceived as
more likeable or more human-like than one that doesn't but that's very hard to
investigate with children of 4-6 years old so we propose to run a similar
study with either older children or adults to see whether the perception of
the robot is maybe different if it moves around more and if it actually
introduces variation. Thank you for listening! If you have any questions feel
free to contact me, and I hope to see you soon!