DeepFisheye is a system that tracks multiple fingers
near a touchscreen.
I think touchscreens are one of the most
revolutionary inventions in modern computers.
However, its input space is limited to the size of its physical area.
So I wanted to expand its input space.
In HCI, mid-air interfaces and finger-aware interfaces were suggested
to expand the input space of toushscreens.
Using DeepFisheye, it is possible to expand the input space of
touchscreens by enabling mid-air interactions
and finger-aware interactions.
Now, let's see how it works.
This is the DeepFisheye protytpe.
We attached a fisheye camera to the bottom of the Microsoft Surface device.
This application just visualizes the output of DeepFisheye model.
And the actual model runs on the different computer.
I will show you the intermediate outputs of the DeepFisheye model.
This is the image from the fisheye camera.
DeepFisheye use this single RGB image.
From this RGB image, DeepFisheye generates depth images.
As you can see in these images.
From these two images, DeepFisheye generates heatmaps and 3D finger joint locations.
Now I will expalin basic funcntionallites of DeepFisheye.
As I said, DeepFisheye can track multiple fingers' locations.
So when I put my hand above the touchscreen,
it shows the fingertips' locations.
And you can see that these circles
follow each fingertip.
This is a closer look for finger tracking.
The next one is finger classification.
DeepFisheye can classify which finger is touching the touchscreen.
The result of finger classification are shown on the
left top corner of this application.
So when I touch with the index finger,
it shows it index finger.
middle finger
ring finger
pinky finger
and thumb
Now you can see the closer look of this application.
The next on is posture classification.
DeepFisheye can classify which posture a user is using
while touching a touchscreen.
It supports three postures.
The first one is open hand.
The second one is pinch hand.
The third one is point hand.
When I touch the touchscreen with open hands,
it says the hand is opened.
And 'pinch' for the pinch posture.
'point' for the point posture.
This also works in different parts of the touchscreen.
And you can see the close look of this demo.
These are the basic functionalities of DeepFisheye.
Using these basic interaction elements,
we implemented sample interaction scenarios.
And I will show you the scenarios.
This is a painting application that we created
to demonstrate what interactions are possible with DeepFisheye.
If you see this application, most parts are canvas.
And we have a single button for most of user interfaces.
Because in this way, a user can have more space to draw.
Let's see how this application works.
You can access all the tools by using this single button with different fingers.
But it is hard to know what fingers are mapped to which tools.
If you approach your finger to this button,
the button shows tools that are mapped to each finger.
You can draw with your index finger.
And you can erase with your middle finger.
If you want to change the color of the pen,
you can tap the button with the index finger.
If you want to change the width of the pen,
you can tap the button with your middle finger.
Now I will chang the color to green.
We added a special tool, rather than just a pen, which is a spray.
You can select the spray tool with your ring finger.
If you select the spray tool, your hand becomes a spray.
So if you do the pinch posture, you can see the area
that the spray will spray ink.
And if you do the point posture, it starts to spray ink.
This painting app also supports shape addition and modification.
If you tap the thumb anywhere on the surface,
It pops up the shape menu.
And this menu follows the fingers.
By tapping different fingers,
you can place different shapes in any location.
You can move this shape with pinch posture.
And you can copy this shape with the point posture.
You can delete a shape with the ring finger.
There are the basic functionalities of our painting app.
This was demo for DeepFisheye.
I hope you enjoyed.
If you are intereted in mid-air interactions,
or finger-aware interactions
more broadly, hand pose estimation for fisheye camera, fisheye images,
please chekout our paper.
Thank you very much.