Hi, I am Mirhan and I'm going to be presenting our joint paper with
professor Işıl Bozma which deals with 3d point direction estimation.
In the paper, we address the detection of 3D hand pointing direction by a mobile robot.
Pointing direction estimation improves human robot communication with possible applications
in areas like human guided object learning and environmental modeling. In our model only the
hand data is used to estimate the direction although other parts of the body may give some
useful insights such as alignment of the head. This is mainly because robot may not be able to
see the whole body in all cases. To this end, a three-step algorithm is developed. Differing from
previous work the robot is able to estimate the 3d hand direction but accurately and fast regardless
of the proximity background variability or the detectability of specific human parts.
We also present a sensitivity analysis of the final estimation to the sensor noise. Two new
data sets for pointing gesture classification and continuous 3d pointing direction are
made publicly available as there are no such datasets to the best of our knowledge.
Here is a table summarizing related literature about pointing direction estimation. Applicability
of these methods in robotic applications depends on several issues as listed here.
It can be seen that our approach is the first 3d pointing direction estimation method
with the least amount of assumptions to the best of our knowledge.
We do not have assumptions about the background or do not require hand to be close to the camera etc.
We are using rgbd data coming from a kinect sensor.Using this data, first a novel sequential
network-based learning model is developed for the simultaneous detection of hands and humans in rgb
data then hand regions are labeled as pointing or non-pointing by a cnn-based classifier.
Following, a density-based segmentation algorithm is applied to the pointing hand regions to get
the 3d hand segment. A new geometric method for estimating the 3d pointing direction
from depth data off the detected hands is then applied to get the final estimation.
Direction estimation starts with finding hand and human regions from rgb image. We use cnn-based
state-of-the-art YOLOv4 architecture for this job. To train the architecture a data set with
both human and hand labels is required. However there were no such data set in the literature,
so we first created a hand and human data set and trained YOLOv4 to get the final detector.
This is achieved in three stages first a hand detector is created by training the network on
the COCOhand dataset. Next the hand detector is run on the human images of COCO dataset
as to create an annotated dataset with hand and human labelsn This dataset is then used to train
the network once more. Hence the network learns to detect both hands and humans.
After finding hand bounding box robot needs to determine if the
hand is performing a pointing gesture to avoid non-pointing hands to be processed
further. To this end we trained a convolutional neural network with our own data set consisting
of pointing and non-pointing hand regions. We try to include images with varying conditions such
as illuminations and hand distance to the camera. The architecture of the cnn is given in the figure
which is inspired by another pointed gesture paper and it is similar to the Alexnet
architecture which is quite popular in the literature.If the hand is performing a pointing
gesture hand bounding box region is mapped to 3d space using depth data. Then a density based
segmentation algorithm which is an improved version of an existing one is applied. Original
algorithm is designed for 3d points without color information. We improved it to consider the color
data as well. Algorithm basically applies a connected component algorithm in the spherical
coordinate system with the given distance metric in the slide and the adjacency decision procedure.
Using this criteria, connected component algorithm is applied and the segments in the region are
found. Then among the found segments we need to determine which one is the hand segment. For this,
we utilize the observation that hand should be the biggest object in its bounding box as well as the
closest object to the camera.A score is assigned to each segment to measure these two properties
and the segment with the biggest score is assumed to be hand segment. After having hand segment
we determine the 3d pointing direction. In our method, we try to find the pointing part of the
hand. What do we mean by the pointing part of the hand? Well, the common thing about the pointing
gesture is that significantly big part of the hand is gathered around the line in a pointing gesture.
We define points around that line as pointing part of the hand. Now, let me explain how do we get
this point in part of the hand.First eigenvectors and eigenvalues of the covariance matrix of the 3d
hand points are found. Since principal eigenvector gives the direction where the data is mostly
dispersed we expect our final direction estimation to be close to the principle eigenvector. This is
key to our algorithm. We assume that pointing part of hand is a collection of hand 3d points
with maximum cardinality whose orientations close to the principle eigenvector v0. This assumption
makes sense since we assume v0 is close to the pointing direction and big part of the hand is
gathered around the line in pointing gesture. With this assumption the pointing part of the hand is
found through finding two planes whose normals are orthogonal to v0 and whose neighborhoods
contain the largest amount of 3d hand points. Finally the effect of the sensor noise on the
final prediction is investigated namely a bound of errors of the principle eigenvalue and eigenvector
is found. We approximate the covariance matrix of the noisy data as in equation two where the
sigma is the true covariance and ns is the covariance of the noise which is assumed to
be a multiple of the identity matrix. Then, we use weyl inequality to find the bond on
principle eigenvalue error and Davis-Kahan equation for the error on the principal
eigenvector. The success of the pointing gesture classifier is investigated on the pointing gesture
data set which is divided into training validation and test data sets. The number of images in each
sets are given in the table. As it can be seen we achieve a 94.0 accuracy in the test dataset
Our 3d point direction dataset includes images with varying hand distances
to the camera. We group each image depending on the hand distances into the three groups close far
and distant. In the first table you can see the number of images in each of these three groups.
Our procedure on the experiment is as follows: we classified a direction estimation
as accurate if the angle between the estimated direction and the ground truth direction is
less than a threshold. We used three different thresholds 10 degrees 15 degrees and 20 degrees
results of which given in the middle table. As it can be seen we achieve 83.7 percent accuracy for
the 20 degrees threshold we also investigated the accuracy rates for the different distance groups
we also ran our 3d point interaction algorithm on the publicly available IPO
data sets this data set includes images where the hand distance to the camera is less than 1.5
meter. In the table, our results as well as the results of an existing algorithm is given on the
data set.As it can be seen results of the both algorithms are close but the existing algorithm
performs slightly better than our method. But, it should also be noted that the existing algorithm
only works for the images where the hence distance to the camera is small.
Let me conclude my speech with the final remarks. Our three main contributions in the paper are
a new 3d point interaction estimation method with the least amount of assumptions
a sensor noise analysis on the final predicted direction and two publicly available data sets
that can be used as a benchmark for future studies. As a future work to this paper a
human-guided object learning may be built on top of the presented method and also the
temporality of the incoming sensor data may be used to include to improve the accuracy.
Finally, I want to thank Özgür Erkent for his comments on the methods and Mehmet Yasin Özkan for
his contributions during the labeling of the datasets. Thank you for listening.