Hello.
I'm Katherine Kuchenbecker from the Max Planck Institute for Intelligent Systems, and it's
my pleasure to present this paper as part of HRI 2022.
Our co-first authors are Valerio and Maija, and the other authors are Diar, Tommaso, Chie,
Alan, Max, and myself.
Together, we were motivated by a future scenario where, imagine, I'm working on my car, and
I have a robot helper.
And I say, "Hey, robot!
Can you please bring me that medium screwdriver?"
And the robot grabs that tool and brings it over to me.
And I turn around, and what I hope happens next is that I can enjoy a quick handover
that enables me to securely hold the object that the robot brought.
And not only that, but also that I can be able to use it immediately, so I can immediately
go ahead and do the next step in my task, like screw in a screw, continue with my projects.
And, if a robot or a human, for that matter, managed to do this really well, I would be
very fond of it and really grateful for the support that it showed me.
Now, other researchers have also been looking into robot handovers, between humans and robots,
and humans and humans.
And we took some key learnings from that prior work.
For one, Valerio recently led a project where they experimentally showed that how a robot
grasps objects definitely affects the success of the handover.
And ideally the way the robot grasps should account for the receiver's task, whatever
they want to do with that object.
Now, there have also been other studies that have looked into how object orientation affects
handover success, and also the pose of the robot, the position of the robot's arm.
Now together, that's already quite a lot, but there were two pieces that we thought
were missing.
And they're both about empathy for the human.
And so one is about the visibility of the object.
Not just, I can't immediately perceive the orientation; I have to see it with my eyes
from my current vantage point.
And then second, not all parts of the tool have the same importance.
The handle is where I want to grasp it if I want to go ahead and use the tool to do
something else, to do my task.
And so the research question that we wanted to address in this work was to investigate
how the visibility of the handle of a tool affects the planning, execution, and then
later the perception of a grasp as a robot brings a tool and hands it to a human user.
Now, we worked on solving this problem, we're addressing this problem by conducting a study
in VR.
So we had our participants wear a high-quality head-mounted display that provides a good
three-dimensional view of a simulated robot.
So we didn't have to have a physical robot and keep resetting it.
We could show many scenes in quick succession and test a much wider variety of stimuli more
quickly.
And then furthermore, the head-mounted display does really good hand-tracking, so we could
also have nice data of how the user responded to each of the objects that we presented them
with.
Now the virtual environment looks like this.
The participant always had to start their hands in these spheres at their sides.
And then the robot would appear and present the object; here's a hammer.
Then the user needed to reach out and grab the object as though they were going to use
it.
So for a hammer, that means to hammer a nail.
For scissors, to grab the scissors like they were going to cut paper.
And for a screwdriver, to grab the screwdriver like you would screw in a screw.
So this is a rather constrained grasping task.
As soon as they actually reached out and made contact with the object, like you see here,
the robot and object would disappear, and they would be presented with two subjective
questions, the first about the holdability of the object, and the second about the usability
given the grasp they had just performed.
We also of course looked at where they grabbed on the object, and here you see visualized
different participants in different colors, where for the same stimulus people made different
decisions about where to try to grab this object.
So, now I'm going to show you videos of two trials.
The waiting begins here.
Then the robot appears.
The user plans.
This time they decided to grasp the handle; that was pretty holdable.
Sorry, not the handle: the head of the hammer.
Ah, but not particularly usable.
And now here we see the scissors.
It's presented pretty straight forward.
You could grab it, and you could hold it very well and really immediately go ahead and use
those scissors.
So that's what trials looked like.
We had a total of three objects, each of which has its red handle region and blue non-handle
region.
We used a physics-based grasp simulator that we recently developed and published called
PrendoSim to simulate more than one thousand grasps for each of these three objects.
We then defined a metric that enabled us to quantify how visible the handle is in this
particular object presentation.
And the way we calculated it is from a single user viewpoint, we count up the number of
pixels of the handle that can be seen and divide that by the total approximate maximum
number of pixels that are visible at the same position but in an orientation to the side,
where it's very visible and where there is no occlusion by the robot hand.
Now, from these one thousand grasps simulated for each of the objects, we sampled representative
sets of ten in each of four visibility classes.
And so the least visibility is when the number of pixels you can see is less than forty percent
of that maximum number.
These objects on the left, visibility number one, these are quite hard to see.
And then we go from forty to sixty, sixty to eighty.
And then for visibility level four, it's above eighty percent.
You can see a number of pixels that is more than eighty percent of the number of pixels,
so it's very visible.
And you can already probably tell, these grasps are probably much easier to perceive and much
easier to reach out and grab.
Now this resulted in a total of one hundred and twenty task trials that every participant
saw in random order interspersed with some other trials and with breaks to ensure they
didn't get too tired.
Seventeen participants took part.
They were all adults, and they all did the study with their right hand.
For each trial, we have a nice recording of how their hand moved and how their fingers
moved, which lets us also calculate where they contact the object.
Here are all the metrics that we calculated.
We looked at two time-based metrics: how long it took them to start moving, and then how
long it took them to execute the grasp.
The distance their hand traveled and how much it deviated from a perfect straight-line path.
How much their head moved in total and especially laterally, because they tended to do that
to get a better view of the object when it wasn't very easily visible.
Where they grasped, handle versus non-handle.
And the answers they gave to how holdable is that object and how usable is that object.
The paper describes the detailed statistical methods that we used, and here I'm just going
to highlight a few of the main results.
On the objective metric side, we were able to see that as the visibility level of the
handle increases, the amount of time it takes to plan a grasp and also execute a grasp significantly
decreases.
Furthermore, the distance your hand travels also decreases a bit, and you take a much
more straight-line path when the handle is very visible.
We found very similar results for head motion, all of which tells us that the planning and
execution of a grasp is much more efficient when the handle is easily visible.
We also see that higher visibility levels result in many more grasps, significantly
more grasps on the handle itself.
And if the handle isn't very easily visible, the user will be more likely to grasp on the
non-handle part, and then, which means they are going to need to re-grasp in order to
make use of that tool.
Now on the subjective side, this is how participants perceived the grasps as they were presented
with different visibility levels.
The results really match.
So, this is by object now.
We see that as the visibility level increases, the objects are perceived to be much more
holdable, or the resulting grasps, and furthermore usable.
So the usability levels are generally lower than holdability, because there are more constraints,
but we see the same trend that higher visibility levels lead to better holdability and better
usability for the object.
There are some differences between objects, which we think have to do with their geometry
and size and constraints, but the trend was true for all three objects.
So in conclusion, when we are programming the robot helpers of the future, they should
make sure to try to show the handle of a tool they are trying to hand over to the user,
which means estimating the viewing, the vantage point, the viewing position.
And show it to them, not just point it straight at your eyes, but let you see it so you can
easily estimate it and grab that handle.
Now this is only one study.
There is lots of additional interesting work, and in particular we think that in the future
people should look at how visibility of the handle interacts with the position of the
object and the angle, because we didn't vary the object position, and we didn't independently
vary the angle.
We presented a wide range of object angles, but it did tend to somewhat relate to visibility.
And so to further decouple these different aspects, we think future studies are needed.
In summary, I hope you enjoyed learning about this research, called, "Robot, Pass Me the
Tool: Handle Visibility Facilitates Task-oriented Handovers".
I'm really grateful for your attention, and please reach out to any of the authors if
you'd like additional information.
Thank you.