Hi, my name is Nikhil and I'll be talking about our paper on
learning socially appropriate Robo-waiter behaviours using
real time user feedback. Let's get started. Humanoid Service
robots are becoming more and more prevalent in the
hospitality industry. Whether it's conducting deliveries to
guest rooms, being robot waiters taking customers orders, or
delivering food to the tables in restaurants and cafes. HSRs
donning more active roles in serving humans in daily
interactions. To effectively navigate in human environments,
these robots need to learn behaviours that are considered
socially appropriate by humans. This may include navigating in
human environments, learning appropriate approach behaviours,
navigating through complex real world home scenarios, or as
Robo-waiters facilitating customer orders and assisting
them in service settings. In this work, we focus on one such
setting where we evaluate Pepper as an effective Robo-waiter,
navigating through a restaurant learning socially appropriate
approach and positioning behaviour with respect to
humans. For this, we simulate a restaurant environment by
designing a nine cross nine grid, placing the customer table
right in the middle. The robot needs to learn how to approach
the customer table and position itself appropriately to interact
with the customers. One challenge in such settings is to
determine what might be considered appropriate
behaviour. For this, we adopt a crowdsourcing approach to gather
appropriateness ratings for robot behaviour in our grid
world scenario. For each position in the grid, we
generate images or GIFs for that position, and ask the
participants to imagine being served by the robo-waiter from
that position. These edited images or GIFs are presented to
participants using Google Forms, where they're asked to provide
annotations for the appropriateness of the robot
positioning itself or speed, as well as their effective
responses to robot being in that particular position. Each
individual form is annotated by 8 to 11 unique participants. So
what do these annotations look like? Let's look at positioning
first, this grid shows the average appropriateness ratings
on a scale of one to five, we see that there's a clear
preference for positions in front of the table. Furthermore,
participants rate positioning robot positioning itself closer
to the table as more appropriate. While positions
behind the table are considered to be less appropriate. For
robot speed, we see that appropriateness of the speed of
the robot varies with its positioning. For instance, low
to medium speeds are considered more appropriate when
robo-waiter is in front of the table, while faster speeds are
considered to be more appropriate when it is behind.
Finally evaluating the effective responses of participants
towards robot positioning, we see that the participants report
a higher valence value when the robot is in front of the table
while high arousal values are reported for positions behind
the table. Looking at both arousal and valence together, we
see that for positions in front of the table participants report
positive valence with calm to neutral arousal, indicating that
these positions are preferred. On the other hand, participants
report low valence and high arousal that is feeling agitated
or unhappy when the robot is behind the table. Having
collected these ratings, it is important to evaluate how
reliable these ratings might be. For this, we compute the Fleiss'
Kappa to measure the reliability of agreement between
participants on these ratings. We split the grid into quadrants
and compute these values for different regions within these
quadrants. Overall, we see a moderate agreement between
participants across different ratings. Position and various
ratings are more reliable with a higher kappa value compared to
arousal and speed ratings. In particular, we see front right
or quadrant one receive relatively higher kappa values
compared to other positions. Yet, this agreement is still at
moderate level, resulting in a lot of scope for allowing
robo-waiters to consider person specific adaptation while
modelling interactions with them. To learn appropriate
behaviours, making use of the annotations collected, we design
a reinforcement learning problem where the top right quadrant is
used to represent the restaurant settings. The action space for
the robot in this environment is defined by moving by moving up
down right or left at different speeds. At any given position
and speed, the average crowd-sourced appropriateness
ratings are used to reward the agent to be in that particular
state. To implement the behaviour generation model, we
compare three different reinforcement learning
strategies. More details on these can be found in our paper.
We train these different models with the collected data over
20,000 episodes, and compare them in terms of the rolling
episode score achieved by these models. As can be seen, DQN
performs the best and is the most stable. Although A3C is the
quickest to converge, it converges to a suboptimal
solution, while DQN+ HER is relatively unstable. We train
these models only using the position value matrix we saw in
the previous slide. With no real time adaptation in the model
yet.
To evaluate how we can learn and adaptive behaviour, with human
feedback, we conduct an HRI study with Pepper as a robot
waiter. The restaurant grid settings are realised with
pepper interacting with the customer sitting at the table.
We recruited 21 participants to interact with pepper across four
different interactions or tasks. Firstly, the pepper welcomes the
participants to the restaurant takes their order from a menu of
options, goes away and returns with their order and finally,
returns to collect dishes and receive feedback. Since at the
time of the study, COVID-19 restrictions were in place, no
actual food or dishes were used, and instead, the whole scenario
was acted out with the participants. Each participant
repeated the four tasks with the pepper under four randomly
ordered conditions. Thus in a within-subjects design, the
participants saw four different versions of Pepper performing
the same tasks, one after the other. The first condition was
the control condition where pepper randomly positioned
itself throughout the four tasks. For the second condition,
we use the pre trained reinforcement learning model
trained on crowd-sourced data to govern pepper's behaviour. For
the third condition, we use pre trained reinforcement learning
as a starting point for the robot and let it adapt based on
the explicit verbal feedback provided by the participants.
And finally, we evaluate the facial affect of the
participants particularly valence to determine their
evaluation of robot behaviour implicitly, and use this to
adapt robot behaviour. Both C1 and C2 represent non adaptive
settings where the robot adapts in real time with user feedback
under C3 and C4. For explicit feedback, we evaluate the verbal
feedback provided by the participants and quantify the
reward given to the robot using a keyword spotting approach. For
implicit feedback, we use an off the shelf facial affect
recognition model. To compute deviations in participants
valence measurements. The reward function was modified to include
both crowd-sourced appropriateness rating as well
as explicit or implicit reward based on the condition assigned.
This allows the robot to adapt in real time to use a feedback.
With the HRI study, we aim to evaluate whether pre training
robot behaviour on top state actually improves its
performance compared to static behaviour. Furthermore, we wish
to evaluate whether using explicit or implicit feedback
improves interactions compared to using only crowd-sourced
data. And finally, is there any benefit of using either explicit
or implicit feedback over the other? To answer these these
questions, we record participants' responses on
questionnaires, collecting ratings for enjoyment,
sociability, adaptability and appropriateness of robot
behaviour. And these evaluation help us understand how
individuals view their interactions with robots. For
enjoyment ratings, we see that adapting with explicit or
implicit feedback results in overall improvements. Pre
training also on average improves over using static
behaviour. For sociability ratings, we see that an overall
increase in ratings across dimensions under pre trained or
adaptive conditions. In particular, using explicit
feedback significantly increases the sociability of the
interactions. Adaptability ratings show a similar trend
where explicit feedback condition seems to emerge as the
most preferred condition, with the robot being rated
significantly higher in terms of understanding and adapting to
what the participant said. Implicit condition performs the
best at understanding how the participants might have felt
fulfilling the requirement from the robot under this condition.
Finally, for appropriateness stating as well, we see the
adaptive conditions performing significantly better in terms of
both robot positioning and speed.
Our interactions with the participants post experiments
also brought our interesting opinions about the waiter. In
general, participants found the interactions to be enjoyable and
interesting, despite repeating the same four tasks multiple
times across the conditions. They all mentioned that they
could imagine being served by Robo waiter like this in the
near future. Some participants expressed inhibitions to the
robot approaching them face on terming them to be frightening.
By the same participants well the robot movements particularly
upper body movements to be more human like. To conclude we
present the first work investigating real time
adaptation strategies for Robo waiters to learn socially
appropriate behaviours. We show that using classic identity
crowdsource annotations can provide a generalised
understanding of what is considered appropriate. This
understanding can be improved individually when interacting
with individual users by factoring in explicit or
implicit feedback. While explicit feedback seems to be
better at as it improves enjoyment, sociability and
adaptability for robot waiters. Next steps for us will include
considering multimodal analysis of user behaviour, to capture
implicit feedback, as valence alone might be insufficient at
to do so, also with COVID-19 restrictions hopefully lifting
we would want to extend our evaluations to wider and more
diverse audiences using more realistic restaurant settings
involving food and other social aspects of the interaction.
Finally, I'd like to thank our sponsors and my co-authors for
their contributions, particularly Emily, who
conducted these experiments as part for MPhil dissertation.
Thank you so much for listening.