Hello everyone, I’m Fang Hu from SciFi Lab,
Today I'm going to present FingerTrak
a joint work done by Cornell University and the University of Wisconsin-Madison,
FingerTrak is the first minimal-obtrusive wristband
that is able to reconstruct continuous 3D hand pose,
As shown in the video,
blue hand in the top left is ground truth,
which means the target hand pose we want to reconstruct
red hand in the top right is prediction made by our FingerTrak system
in other words, the hand pose reconstructed by our system,
To reconstruct hand pose
is a focus in human computer interaction area
as it is a cornerstone to a multitude of interactive applications
For instance, we can use hand pose to control VR/AR
and to interact with robot
Hand pose is also essential in sign language translation
as users use different poses to express ideas
All of these applications require high quality reconstructed hand pose
especially in mobile settings
However, existing systems are either camera-based
which needs to observe all the fingers
thus can easily suffer from occlusion
or are based on glove or external sensors
which cannot move or are bulky
All current solutions cannot reconstruct hand pose in mobile settings,
So our research question is
how to design a minimal-obtrusive wearable device
that can continuously reconstruct hand pose
without the need of seeing all fingers
The answer is silhouette!
Here, our intuition is that
hand silhouette contains key information about hand pose
As shown in the left column
we observe hand contours from four different views
can we infer what the hand pose is?
Yes, it looks like this, pose 1,
Now we move to the next multi-view figures
the corresponding pose is pose 2
It’s the same for pose 3. The silhoutte matters!
Based on this intuition, we developed FingerTrak,
It consists of two major parts,
First, a wristband mounted with four thermal cameras,
which captures hand silhouette from different views,
The reason why we choose thermal cameras is that
thermal images area easier to segment than RGB ones
despite their low resolution
But other types of cameras also work, which I’ll cover later,
When we collect data, Leap Motion provides us with ground truth,
and second, machine learning model
which takes multi-view silhouette as input, it uses renet 34 as backbone and then
connect to two fully connected layers. Lastly, it outputs hand pose parameters.
Now we know how it works, let me show you two demos,
The first demo is when hand is empty,
The red skeleton is reconstructed hand pose by FingerTrak,
As shown in bottom left corner,
red area is hand and green one is background. This is the video demo.
There's another video shows reconstruction results
when hand is holding objects, like opening a book or holding a pen
Then how do we prove that FingerTrak works in real world scenarios?
We designed four test settings to evaluate FingerTrak’s real world performance,
In the first three settings, hand is empty.
When testing background is the same as that of training,
it achieves 1,20cm pMAE, that is, joint position mean absolute error,
The blue hand skeleton is the ground truth and the red one is our prediction,
We also test it under different background to simulate its real world use case,
Moreover, users may take off the wristband and then take it back on in real life
so we design re-mounting session.
Lastly, we test the system when hand is holding objects,
Apart from user study results,
there are several interesting FingerTrak's design factors worth analyzing,
One of them would be camera setting,
FingerTrak's prototype has four cameras, which may seem a lot,
So the question is,
can less number of cameras achieve similar performance,
in other words, do we really need four cameras?
We changed the quantity and positions of the cameras we used in the evaluation,
We labeled the four camera used in the study from C0 to C3. (C0 : top, C1: bottom, C2: right side, C3: left side)
We trained different models with different combination of the camera settings
and calculated the relative reconstruction error in each set of camera setting,
We used the evaluation results from all 4 cameras as 100%
and compared other reconstruction errors to the original setting,
C1 is the most informative camera among the four cameras,
The results indicated that with minimal number of cameras like one or two
the system can still reconstruct hand pose
in the cost of sacrificing some performance
and using three cameras can achieve very similar performance as using four,
Another interesting design factor is camera type,
Thermal cameras preserve hand silhouette but its resolution is low,
We prove that with RGB camera whose resolution is higher,
we can potentially have better reconstruction results,
Our preliminary results show that
using RGB cameras can achieve an average accuracy of 92,62%
in distinguishing 12 challenging micro-finger poses,
much higher than thermal one, which is under 50%,
Finally, FingerTrak empowers lots of exciting applications,
For example, it can be used for recognizing fine-grained daily activities
such as table-top human object interaction,
An example is shown here,
To summarize, FingerTrak is the first wearable system
that can continuously track 3D hand pose in mobile settings,
We demonstrated the feasibility of reconstructing the entire hand pose
by only observing a few hand silhouettes from the wrist.
Thanks for watching!