Hello everyone, this is Erwin from CMU and TokyoTech.
Today, I am going to introduce and demonstrate a new hand pose interface for wearable devices.
Here, I said demonstrate which means during the presentation I will use the proposed system instead of the mouse to control.
As we may know, hand is one of the most natural way for us to interact with other objects.
Therefore, thousands of study were done to recognize human's hand-pose.
However, there is still lacking a natural way to detect user’s hand which can be embedded into a wearable device.
Some of the previous works are too bulky.
Some of them are placed in a weird location,
and the remaining ones are fixed therefore cannot be used in the wild.
In this paper, we will introduce BackHandPose,
These camera could be included into some wearable devices such as a smartwatch.
This video here shows the real-time result of the image preprocessing.
First you can see the hand masking.
We can tell that it is also robust to different lighting conditions.
Next, these are the motion images of tapping different fingers.
You can realize that different part of the back is changing when tapping different fingers.
Now we introduce the “Dorsal Net”.
A two-stream CNN network, of which each stream uses a ResNet 18 as a backbone to extract the information from the hand images and motion images.
After that, the network is followed by an LSTM layer for temporal feature learning.
To enhance the temporal feature learning, we used a LSTM-based Kalman filter network instead of a normal LSTM.
Finally, the network outputs a 19 dimension finger representation, and is recovered to 3D hand pose by an inverse kinematics based hand simulator.
Good, let’s take a look at the real-time result of this work.
To show the effect of our system, we performed several evaluations on 5 different users.
The ground truth data is taken from a leap motion camera and the right tables shows the average angle error of doing 15 different gestures comparing with other baselines
Where our method with Kalman Filter outperforms all the other methods both in the individual model and the general model.
In order to show the robustness of the network, we also run the test in 5 different lighting conditions.
The accuracy become worse in darker condition but it was within 1-degree error comparing to the best condition.
To note that, our system even works under strong sun light condition where the IR-based leap motion cannot.
Besides specific gestures, we also evaluate our system on grasp recognition.
The right figure shows the confusion matrix of 50 tests for each grasp.
In classifying 11 grasp types, our system achieve a 75% overall accuracy.
While the worst case appears in T1F, T2F and T3F which is the picking motion of the Thumb with another finger.
Finally, to show the usability as well as the potential of this research, we developed several types of applications.
That's all for today's presentation, thank you.
Please feel free to contact me with any question and you can also read our paper for a more detailed explanation.