Hi. I'm Keunwoo Park from KAIST.
Today, I will present our work DeepFisheye,
a system that tracks multiple fingers near a touch screen.
This is a joint work of KAIST and Imperial College London.
Most smart devices use a touchscreen as an input interface.
However, a touchscreen's input space is limited to the size of its physical area.
Therefore, some studies have expanded input space to above the touchscreen,
which is a near-surface space.
Moreover, some studies expanded input space to multiple fingers.
For example, a smart device can distinguish which finger is contacted on a touch screen,
and each finger can be used for different input.
These two approaches are both attractive
because they expand input space while enabling natural interaction.
For example, Air+touch and Handsee demonstrated how mid-air interaction can expand an input space.
MagTouch and TriTap are examples of practical finger identifying technology.
Existing mid-air finger tracking technologies can recognize a fingertip,
but they cannot distinguish which finger it belongs to.
On the other hand, finger identifying technologies cannot track fingers in mid-air.
How can we support both interactions with a single technology?
If there is a single technology that can track all the fingers’ locations above a touchscreen,
it can support both interactions.
We defined a term Near-Surface Multi-finger tracking (NMFT) technology.
It tracks each finger's locations above a touchscreen,
which can support both mid-air and finger-aware interactions.
We had these four criteria for NMFT on mobile devices.
First, it should support various sizes
because there are many different sizes of smart devices.
Second, it should work in various environments.
Smart devices can be carried around anywhere.
Third, it should have a minimal physical footprint to fit into a smart device.
Lastly, it should not require any additional accessory for a user to wear.
We thought the vision-based method could satisfy these criteria
because current camera modules are very small, and it doesn't need any gloves or rings.
However, it is hard to install a camera on a smart device
and capture a whole hand's image due to its limited field of view.
Therefore, usually, a camera is installed at a distance from the interaction space.
We solved this issue by using a fisheye camera.
A fisheye camera with 180 degrees of field of view
can capture the hand image no matter where the hand is located.
Now, I will introduce DeepFisheye prototype.
A fisheye camera is attached to the bottom of the touchscreen.
It tracks finger joint locations above the touchscreen using a deep learning model.
It can also classify which finger is in contact with the touch screen.
Moreover, it can classify hand posture of the touching hand.
I will explain the working principle behind DeepFisheye.
Everything starts from a fisheye RGB image.
First, it goes through the Pix2Depth network,
which converts an RGB image to a depth image.
Then, it isolates the hand part from the RGB image using the depth image.
The two images are combined and fed into the hand pose estimation network.
The hand pose estimation network outputs the 3D location of all the joints.
From this output, it is possible to classify contact finger and hand postures.
I will explain the essential parts of this pipeline in more detail.
The Pix2Depth network follows the encoder-decoder model.
It encodes a hand pose information and generates depth from the encoded data.
The hand pose estimation network is a modified version of the ResNet network.
Please see our paper and supplementary document for the complete description of the networks.
Audiences might wonder why we generated depth image in the middle,
rather than directly estimating the hand pose.
Let me share some part of our design process.
Most of hand pose estimation methods crop hand part of the image
to improve hand pose estimation accuracy.
However, it is hard to crop hand and run hand pose estimation with fisheye images
because the hand’s image changes significantly base on its location.
For example, the hand posture in picture a, b, and c are all the same,
but hand images are all different.
Therefore, we need a new technique to improve hand pose estimation accuracy in the whole image.
And this is how we ended up with adding depth image as an intermediate output.
So does generated depth image improves hand pose estimation accuracy?
In our experiment, yes it does.
The blue and pink lines represents the result of the pipeline
with and without Pix2Depth network, respectively.
PCK is a percentage of corrected keypoints at a certain threshold.
A keypoint is assumed as corrected, if its error is less then the threshold.
The hand size is distance from the wrist and the start of the middle finger.
When we tested two pipelines, with and without Pix2Depth network,
the pipeline with Pix2Depth network shows higher PCK, which means it was more accurate.
We used synthetic images for this test, which will be explained in the later part.
For the contact fingers, DeepFisheye can classify all five fingers.
And also it can classify three hand postures, which are open, pinch, and point.
DeepFisheye uses simple rule-based classifiers.
The contact finger classifier just finds the nearest fingertip from the touchpoint.
The hand posture classifier utilizes which fingers are folded to distinguish the three postures.
For example, if only middle, ring and little fingers are folded, it is pinch posture.
Every deep learning needs a proper dataset.
Here is how we created our DeepFisheye datasets.
Fisheye images are very different from normal pinhole camera model images.
Therefore, hand images took with a fisheye camera were necessary for us.
Even though there are many different hand datasets,
we could not find the one with fisheye images.
So we created the DeepFisheye datasets.
There are two datasets.
The first one is the synthetic dataset.
We used a virtual hand model and collected images in the virtual environment.
Specifically, we collected color image, depth image, and 3D hand joint data.
There are many benefits to this approach:
it is possible to collect a large number of images with a small cost,
it is possible to collect accurate hand joints data,
and it can randomize background and the skin color easily.
However, the problem is that synthetic images are different from the real images.
Therefore, when a model was trained with only the synthetic data,
it did not perform well with real images.
The real dataset fills this domain gap.
It comprises fisheye images and 3D joint data,
which was collected with the Leap motion device and the fisheye camera.
The synthetic dataset contains 233 thousand images,
and the real dataset has 42 thousand images.
The bottom images show sample images of the datasets.
We trained the deep learning model with synthetic datasets first
and fine-tuned the model with the real dataset.
The synthetic dataset was used first because
it has more data with more diverse hand poses and backgrounds.
This process let the network to learn basic knowledge
of the shape of the hand and distortion of fisheye images.
And as mentioned earlier, the real dataset teaches the network
how real images are different from synthetic images.
We evaluated DeepFisheye through a user test.
We recruited 12 participants.
We evaluated three things: 3D finger tracking accuracy,
contact finger classification accuracy, and hand posture classification accuracy.
We collected images from four different locations.
We collected 3D joint locations with the leap motion device.
Participants moved their hands above the leap motion device and fisheye camera for 2 minutes.
We collected data with hands opened because, in a previous study,
users prefer to open their hand when they use finger-aware interfaces.
For the classification accuracies,
we collected fisheye images and touchpoints for each finger and posture.
Participants had to touch the touchscreen and move their fingers all around the touchscreen.
5 to 700 samples per trial were collected.
I want to emphasize that collected data was only used for testing, not for training.
The four different locations were a workshop, cafe, seminar room, outside terrace.
We chose these locations to consider different backgrounds
and lighting conditions, and these are the sample images.
We analyzed the accuracy of three different sizes of area:
large tablet size, small tablet size, and smartphone size.
It doesn’t mean that we used three different devices,
but we analyzed data in theses target areas.
First, let's see fingertip tracking errors.
We analyzed Euclidean error and PCK error.
The average Euclidean errors were around 2cm.
And 80% of errors were less than 3 cm.
The second is finger classification accuracy.
The average accuracies were around 83% across the three device sizes.
The thumb showed high accuracy because it is far apart from other fingers.
The accuracies were high at the center of the screen and low at the boundary.
It is because when the hand is far from the camera,
some parts of a hand was occluded and highly distorted, as you can see in picture C.
Hand posture accuracy showed a similar trend.
The boundary parts showed low accuracy for the same reason.
Overall accuracies were around 90%.
The pinch gesture and point gestures were mostly confused due to its similarity.
We implemented a few interaction scenarios using DeepFisheye.
Let's say a user is using a painting app, and it has a single button.
A user approaches his fingers to preview what functions are mapped to different fingers.
Then a user selects a tool that he wants.
As another scenario, modifying shapes can be done naturally with DeepFisheye.
A user can load the menu to put a shape on the canvas with the thumb finger.
Then a user can move a shape with the pinch posture, and copy with the point posture.
The fisheye camera that we used is too big to be attached to smart devices.
However, we think it is possible to shrink down the camera size,
just like the ones in the smartphones.
Then, DeepFisheye can be applied to smartwatches and smart phones.
In conclusion, we propose DeepFisheye, a near-field finger tracking solution for mobile devices.
We evaluated DeepFisheye’s tracking and classification accuracies,
which proved that DeepFisheye is a feasible concept.
We share the code and datasets for other researchers
who are interested in the hand pose estimation with fisheye images.
For the code, datasets and more information, please visit our GitHub page or our project page.