Hi. I'm Keunwoo Park from KAIST.
Today, I will present our work DeepFisheye,
a system that tracks multiple fingers near a touch screen.
This is a joint work of KAIST and Imperial College London.
Most smart devices use a touchscreen as an input interface.
However, a touchscreen's input space is limited to the size of its physical area.
Therefore, some studies have expanded input space to above the touchscreen,
which is a near-surface space.
Moreover, some studies expanded input space to multiple fingers.
For example, a smart device can distinguish
which finger is contacted on a touch screen,
and each finger can be used for different input.
We wanted to support both interactions with a single technology.
And Near-Surface Multi-finger tracking (NMFT) technology,
which tracks each finger's locations above a touchscreen,
can support both mid-air and finger-aware interactions.
We had these four criteria for NMFT for mobile devices.
First, it should support various sizes.
Second, it should work in various environments.
Third, it should have a minimal physical footprint to fit into a smart device.
Lastly, it should not require any additional accessory for a user to wear.
We thought the vision-based method could satisfy
these criteria because current camera modules are very small.
However, it is hard to install a camera on a smart device and capture a whole hand's
image due to its limited field of view.
Therefore, usually, a camera is installed at a distance from the interaction space.
We solved this issue by using a fisheye camera.
A fisheye camera with 180 degrees of field of view
can capture the hand image no matter
where the hand is located.
Now, I will introduce DeepFisheye prototype.
A fisheye camera is attached to the bottom of the touchscreen.
It tracks finger joint locations above the touchscreen using a deep learning model.
With the 3D finger joint location data and simple rule-based classifiers,
DeepFisheye can classify touching fingers and hand postures.
I will explain the working principle behind DeepFisheye.
Everything starts from a fisheye RGB image.
First, it goes through the Pix2Depth network,
which converts an RGB image to a depth image.
Then, it isolates the hand part from the RGB image using the depth image.
The two images are combined and fed into the hand pose estimation network.
The hand pose estimation network outputs the
3D location of all the joints. From this output, it is possible to classify
contact finger and hand postures.
Even though there are many different hand datasets,
we could not find the one with fisheye images.
So we created the DeepFisheye datasets.
There are two datasets. The first one is the synthetic dataset.
We used a virtual hand model and collected images in the virtual environment.
However, when a model was trained with only
the synthetic data, it did not perform well with real images.
That is because the synthetic images look different from the real images.
The real dataset fills this domain gap.
It comprises fisheye images and 3D joint data,
which was collected with the Leap motion device and the fisheye camera.
We trained the deep learning model with synthetic
datasets first and fine-tuned the model with the real dataset.
The former process let the network to learn
basic knowledge of the shape of the hand and distortion of fisheye images.
Next, the real dataset teaches the network how
real images are different from synthetic images.
We evaluated DeepFisheye through a user test.
We recruited 12 participants.
We evaluated three things: 3D finger tracking accuracy,
contact finger classification accuracy, and hand posture classification accuracy.
We collected images from four different locations.
The four different locations were a workshop,
cafe, seminar room, outside terrace.
We chose these locations to consider different
backgrounds and lighting conditions, and these are the sample images.
We analyzed the accuracy of three different
sizes of area: large tablet size, small tablet size, and smartphone size.
It doesn’t mean that we used three different
devices, but we analyzed data in theses target areas.
First, let's see fingertip tracking errors.
The average Euclidean errors were around 2cm.
The second is finger classification accuracy.
The average accuracies were around 83% across
the three device sizes.
The thumb showed high accuracy because it
is far apart from other fingers.
The next one is hand posture classification accuracy.
The average accuracy was around 90%.
The pinch gesture and point gestures were
mostly confused due to its similarity.
In conclusion, we propose DeepFisheye, a near-field
finger tracking solution for mobile devices.
We evaluated DeepFisheye’s tracking and classification accuracies,
which proved that DeepFisheye is a feasible concept.
We share the code and datasets for other researchers
who are interested in the hand pose estimation with fisheye images.
For the code, datasets and more information,
please visit our GitHub page or our project page.
Thank you very much.