Hello everyone, this is Erwin from Carnegie Mellon University and Tokyo Institute of Technology.
Today, I'm going to introduce a new hand pose interface for wearable devices.
Before we start, I want to acknowledge the collaborators from all over the world.
We have Ye and Prof. Kitani from CMU, Huishyong from University of St Andrews,
Prof. Quigley from University of new south wales, and Prof. Koike from Tokyo Tech as well.
This work cannot come true without any of you, thank you.
OK, As we may know, Human's hand is the most common tool for us to interact with other objects.
Nowadays, we use our hands to operate touch screens for smart devices, using gestures to control the TV or home appliances.
Also, in the field of mixed reality, hand interaction is the most natural way to control a virtual avatar.
However, the current hand pose detectors still have a lot of limitations.
Head mounted displays or some gloves can detect hand pose precisely but will disturb the user due to the extra equipment.
On the other hand, some wearable cameras are used to track the hand,
but most of these cameras are placed at a weird place to observe the hand, which can only be used in limited conditions.
One most natural way might be a 3rd person view camera, which doesn't need to have disturbing devices on human's body.
However, these camera are mostly fixed therefore cannot be used in the wild.
Some recent works also try to solve this problem,
Chan et al introduced the cyclops ring,
which uses a ring-like fisheye camera placed between the fingers to observe the hand to classify the hand gesture.
Chen et al. used an elevated camera to observe the dorsal hand from above to recognize the finger angle.
However, these two setups are still hardly to be called a natural way to detect hand pose.
Okay, before we proceed, I have a question for you.
We could see that most previous work estimates the hand by tracking the fingers,
but, do we really need to see the fingers?
I want you to raise your hand and observe the back of it carefully.
Next, try to tap your finger periodically.
Do you see your skin, your bones, your vessels are moving together with your fingers?
Yes, that’s exactly what we want to use in this paper.
We want to extract the deformation of the back of the hand and estimate the 3D hand pose using a single RGB camera.
. There are a few smartwatches in the market which have a side camera for photography,
we believe this technology could be applied on these wearable devices.
In our previous work, published in last year UIST,
we built a prototype which used a depth camera to classify the gesture from the back of the hand.
This time, we want to further this work to full hand pose for a single RGB camera.
Here you can see the overview of our approach, we used a wide-angle RGB camera on the wrist,
of which the image is shown in the middle here,
and we calculate the motion history from the hand pose and pass these 2 information to our network,
finally we regress the 3D hand pose from the image.
Let’s take a glance at our hardware, as mentioned before,
in this paper, we employ a 120-degree fisheye lens camera for prototyping,
because the current field of view of a normal camera on a watch is not enough to see the whole hand, as we saw in the previous video.
However, we fixed the camera as the same height as a real watch and we do believe that this camera could be easily embedded into a wrist-worn device in the future.
Before we input the image into a network, we performed some real-time preprocessing to reduce noise and extract more features.
First, we used an encoder-decoder network to remove the background and get a masked hand image.
Next, the motion history image is calculated from the past 5 frames to extract more motion features.
This video here shows the real-time result of the image preprocessing.
First you can see the hand masking.
We can tell that it is also robust to different lighting conditions.
Next, these are the motion images of tapping different fingers,
you can realize that different part of the back is changing when tapping different fingers.
Ok, finally we have the major contribution of this paper, the “Dorsal Net”.
We use a two-stream CNN network, of which each stream is a simple ResNet 18,
to extract the information from the hand images and motion images.
After that, the network is followed by an LSTM layer for temporal feature learning.
To enhance the temporal feature learning, we used a LSTM-based Kalman filter network instead of normal LSTM.
A by-pass is also added to the network to choose whether to use Kalman filter or not.
Finally, the network outputs a 19 dimension finger representation,
and is recovered to 3D hand pose by an inverse kinematics based hand simulator.
Hereby, we used a special 19D hand representation, of which the first 16 values stand for the joint angle for each finger.
The DIP and PIP have only one value while the MCP has two, a vertical rotation and a horizontal rotation.
And the thumb is represented by a 3D vector independently, because it is quite difficult to see the changes of thumb from a wrist camera.
Therefore we want to ease the work of the network to simply estimate a thumb vector and leave the rest to the hand IK.
Good, let’s take a look at the real-time result of this work.
To show the effect of our system. We performed several studies for evaluations.
. First of all, we collected data from several users and the setting is shown in the left figures
the ground truth data is collected by the leap motion IR camera.
We collected data from 5 users, of which each user did 15 gestures including 10 static gestures of American sign language number
and 5 dynamic gesture of each finger tapping.
All gestures are recorded 30seconds for 5 sessions, which results in 225,000 frames in total.
For the evaluation, we first studied the accuracy of our system on each finger and joint.
The MAE stands for mean absolute error of angle while the SD is standard deviation.
By comparing all the fingers, the ring finger achieves the lowest error while the index and middle fingers didn’t fall much behind.
The accuracy of pinky finger as well as the thumb is worse because the deformation related to these fingers could be hardly observed from dorsal hand.
On the other hand, for each joint, the vertical MCP performs the best,
however, the horizontal MCP rotation has a quite large error, which was out of our expectation.
Next, we compared our network with other baseline such as the nearest neighbor,
a simple direct regression, and other state of the art network.
where our network performs better both with and without Kalman filter.
In order to show the robustness of the network, we also run the test in 5 different lighting conditions:
Outdoor daytime, outdoor nighttime and Indoor dark condition together with the baseline indoor condition.
The outdoor condition with sunlight cannot be quantitively evaluated because the leap motion IR camera cannot work under sunlight,
but we already saw the real-time result from previous video.
In the rest 4 conditions, the average error increases in dark condition with bad lighting
but there wasn’t a huge difference between the best and worst lighting, which is within 1 degree.
We also ran an ablation study on different network architecture and inputs.
Starting from simple direct CNN regression to two stream ResNet and LSTM.
By comparing the first 2 architecture, we can see that ResNet has a similar error with a much faster inference time.
Next, comparing the 3-5 lines with different inputs, it indicates the Motion input has a greater effect on accuracy than the RGB only condition,
of course the two-stream input has the best accuracy.
Lastly, the last 3 lines indicate that the Kalman filter and our Data augmentation has effect on improving the accuracy.
Besides those gesture, we also noticed that when your hand is grabbing some items,
the dorsal hand is also quite different which depends on your grasp type.
So we also evaluate our system on grasp recognition. The right figure shows the confusion matrix of 50 tests for each gesture.
In classifying 11 grasp types, our system achieve a 75% overall accuracy.
While the worst case appears in T1F, T2F and T3F which is the picking motion of the Thumb with another finger.
To show the usability as well as the potential of this research, we developed several types of applications.
Well, let’s summarize today’s contents.
Last but not least, we still have some improvements to do for this work.
Currently, we just performed a simple wrist angle estimation by the center of hand, and cannot deal with the roll of the forearm,
in the future we are going to build a full hand pose estimation system by also including the movement of background.
Also, the system now works in a dim dark environment, but since it is using an RGB camera,
it will not work in a completely dark environment, which needs to be improved.
Another problem is the number of subjects, due to the corona pandemics, we only collect 5 subjects from the university,
more variation in hand shape and skin color is required to prove the robustness of the network.
And of course the last one is that we hope to implement this system on real watches.
That's all for the presentation, feel free to contact me if you have any question, thank you!