Hello everyone, my name is Daniel Weber and
I welcome you to this talk called
"Exploiting Augmented Reality for
Extrinsic Robot Calibration and Eye-based
Human-Robot Collaboration", which is a
joint work together with Enkelejda
Kasneci and Andreas Zell.
So what is the setting?
As you can see on the image, we have a
robot in front of a table with various
objects on it
and we now want to point out objects to
the robot, which the robot should detect
and interact with, for instance an object he
should grasp.
To point out the object of interest we
want to use the human gaze.
Therefore, for the general interaction
between human and robot we use the
HoloLens 2, which is a pair of augmented
reality glasses from Microsoft with a
built-in eye tracker.
Before we can segment the objects of
interest, we first have to solve two
other problems.
The first one is gaze mapping.
So the HoloLens provides gaze as gaze
vector and hitpoint with the
environment in the coordinate system of
the HoloLens.
Now, a robotic system typically has many
3D coordinate frames that change over
time, such as a camera frame a gripper
frame or head frame and so on.
For the robot to know at which object
the human is looking at, we need the gaze
information in a robot's frame of
reference.
This should be independent of the
human's current view, so that he can move
around freely.
That means, we have to know both, the
transformation in between HoloLens and
robot as well as the transformation
between robot and the camera attached to
it that points at the scene with the
object.
Thus, the second problem is extrinsic
calibration of the camera.
There are different ways to calibrate
robot extrinsics. For example through
indirect collocation, using natural
points in the environment or markers, or
analytical methods or manual measuring.
All have in common that they are often
time consuming or use expensive software
or hardware.
Also if the sensor, in our case the
camera, is moved the calibration must be
laboriously repeated.
In this work, we tackle both of these
problems together.
The idea is to use the HoloLens as
bridge between the coordinate frames. We
create virtual counterparts
corresponding to the real ones. That
means we place a QR code at the robot's
torso, where the transformation to an
existing robot frame is already known,
and we do the same with the QR code at
the camera.
On this image you can see the view from
the human through the HoloLens. On the
right side we have the user interface
and on the left side you can see that
the real camera is overlaid with a
virtual model of the camera.
Now this is the pipeline we are working
with.
The QR codes determine the pose of the
virtual robot and the virtual camera.
Since the pose of the virtual and real
objects are the same, we can calculate
the real transformations from the
position and orientation of this virtual
versions, which we know in the frame of
the HoloLens.
Finally, we can publish this
transformation and also the gaze
information via our user interface.
So on the one hand we can calibrate the
position of the robot and the camera in
relation to each other and on the other
hand we can establish a direct
transformation between HoloLens and the
robot, which means that the robot is
aware of the gaze point at all times
regardless of the field of view.
In the bottom left is a visualization
of where we are at the moment.
We know the poses of the robot, camera,
and human wearing the HoloLens, and we
can track the gaze.
Here, the green arrow is the gaze vector
and the intersection with the
environment is the hitpoint that means
the point where the human is looking at.
Now we are able to continue with the
segmentation.
We address this task by enhancing
existing segmentation methods on the
robot's side with gaze information from
the human collaborator.
Thereby, everything should work
continuously and in real time and the
assistance, that the robot receives from
the human, should be limited solely to
the provision of the gaze information.
Apart from that, the segmentation should
only take place on the robot's side.
This makes sense, due to the higher
resources and computing power of the
robot compared to head mounted devices
like the HoloLens.
So the segmentation pipeline is as
follows.
As input we use the point cloud obtained
from the camera.
In the second step we apply a pass
through filter and a voxel grid filter.
This is not absolutely necessary, but it
reduces the computing time significantly.
In the next step, we transform the normal
of the surface, on which the objects are
located, from world frame to the camera
frame, which is possible due to the
previous calibration.
Afterwards, we use the normal to remove
said planar surface using RANSAC.
In the last step, we find the closest
point to the gaze point within the point
cloud with a K-d tree. This point combined
with Euclidean clustering finally yields
us the object cluster.
Here, we can see what the results look
like. On the left side we have some
segmented objects, where the human was
looking at.
On the right side the view of the human
through the HoloLens is shown, and as you
can see it is possible to give feedback
to the human in form of an augmented 3D
bounding box.
Thus, the human knows immediately what
object the robot thinks the human is
looking at.
In order to evaluate our calibration we
use OptiTrack, which is actually a
motion capture system.
For this, we attach multiple markers to
the robot and its camera, in this case
an Azure Kinect, that are tracked by the
system. You can see an illustration in
the image on the right.
The precision of the tracked markers is
below one millimeter, but post processing
increases this to two to four millimeter.
In this way, we can get some kind of
ground truth with which we want to compare
our method to. We do this separately for
the translation and the rotation.
For both, translation and rotation we do
the calibration in three different poses
of the camera: vertical, horizontal, and
tilted. For each of these poses we do 20
calibrations and record the spatial
distances.
In the box plot on the left the
distribution of the individual Euclidean
distances to the ground truth are shown.
As you can see, the vast majority of
values has a precision below five
millimeter and all the medians are
within the range of the accuracy of the
ground truth.
On the right side you can see the
rotation error.
In general, each rotation can be
expressed by an axis of rotation and an
angle of rotation. This rotation angle
can be considered as a measurement of
the similarity of two orientations.
Here, except of a few outliers, all values
are below 2.2 degrees and the medians
are in the area of 1.9 degree and
below.
Overall, the calibration is therefore
suitable for use in real-world scenarios,
which we will also see later in a
demo.
In order to evaluate our segmentation
method we trained several neural networks
on household items from the SUN RGB-D
dataset.
Our experiments have shown that the
results claimed in the respective papers
are difficult to achieve in practical
applications.
One possible reasons could be that, due
to comparability, the evaluations are
usually conducted on the same few
categories with large objects like table
and couch. As a result, performance on
other classes is often significantly
worse.
In numbers that means that the mean
average precision of Frustum ConvNet at
an intersection over union threshold of
0.25 was only about 28 percent and for
VoteNet it was even below one percent.
As a consequence, a meaningful comparison
was therefore not possible.
So we came up with two alternative
approaches. We first labeled the 3D
bounding boxes in the data by ourselves,
and to avoid vulnerability regarding a
bias in labeling we also projected the
points segmented by our method onto the
2D image plane and compared the resulting
2D bounding box with Faster R-CNN.
Here, you can see the segmentation
results.
In the table on the left you can see the
intersection over union between the
bounding boxes obtained by our method
and the respective ground truth shown
for 10 test objects.
In 2D all intersection over union values
are 0.78 or above with a mean of 0.81.
In the 3D case the values are naturally
a bit lower, but still in the area of 0.6
and above with a mean intersection over
union of 0.69.
If we take a look at the recall curve as
a function of the intersection over
union we can see that both curves show a
recall of one at an intersection over
union threshold of 0.5.
This is particularly impressive in the
3D case, as we recall that the threshold
of 0.5 is twice the threshold typically
used for 3D object detection as well as
in the previously mentioned papers.
In general, an object is already
considered to be detected at a 3D
intersection over union of 0.25.
Finally, I would like to show you a small
demo of the system.
For the calibration we just have to
select the two markers.
The one at the robot's torso and the one
at the camera sensor.
It is fast, simple, and very flexible,
which makes it possible to change the
tilt of the camera sensor even during
runtime.
After the connection is established, the
human can move around freely and all
gaze information is available to the
robot in real time.
Leveraging this gaze data, the robot is
eventually able to segment the objects
that the human is looking at.
The robot also provides visual feedback
to the human's augmented view in the
form of a 3D bounding box.
Thank you very much for your attention.