Hi everyone, I am Yilei Shi, I am from the University of Auckland.
Today, I am gonna present our paper, Ready, steady, touch, Sensing physical contact with
a Finger-Mounted IMU.
This work was done by me with my colleagues, supervised by Professor Suranga Nanayakkara.
Today, touch interaction is becoming a more and more common interaction modality.
It has been used on a variety of interactive devices, such as our smartphones, smartwatches, computers, everyday appliance, and so on.
In academia, touch interaction has also been enabled on everyday surfaces.
One example is augmenting sensors onto surfaces of an object, then the human touch could be
sensed within the augmented area.
A second method is to use optical sensors.
There is usually a sensor, such as a camera, already set up in the environment, and the
touch events within the area could be tracked.
Finally, researchers also developed wearable sensors to sense touch interactions on the
go.
Two examples shown here need to cover the fingertip.
A recent research used IMU, but they mainly focused on discret touch event.
In this paper, we propose a new touch sensing technique, that can continuously detect finger
touch.
The theory is based on two findings.
First, when hovering the finger in the air, the finger tends to move in a very small scale,
although sometimes we can not see clearly.
However, it is stable when pressing against an object.
Second, when a finger moves in the air, this no sound created.
But when moving along a surface, they usually can create some acoustic signals.
These difference could be captures with a fingernail mounted IMU.
You can see the difference in the video.
Based on this finding, we implemented our technique with a very simple IMU board, which
could be attached to the fingernail.
We use a Convolutional Neural network to train our touch classification model.
A simple demo is show in the video.
With the orientation sensing capability of the IMU, we can roughly track the movement
of the finger and control the computer cursor.
We conducted several studies to evaluate the system.
In the first study, we evaluated different places to mount the IMU.
We tested 4 fingers and 3 places on each finger.
We did not test on thumb, as thumb has special size.
Except for phone, thumb is not a common finger for touch input.
The result shows that for the fingers, index finger has the best sensing accuracy.
Index finger is also the finger we commonly use to perform touch input.
For the places, fingernail has the highest sensing accuracy.
Thus, we decided to augment on the fingernail
In this study, we evaluated different gestures.
The first one is performing different gestures.
The second one is when performing the touch action, whether resting the elbow on the table.
We think this causes different level of freedom to move the finger.
The result showed that, for moving gestures, there was no significant difference.
For elbow, the elbow lifted in the air has higher touch sensing accuracy.
We also analyzed the accuracy of different frequencies, by downgrade the frequency.
The result showed that higher frequency had higher sensing accuracy
In this study, we evaluated our model trained in the previous study on 7 different surfaces
and 2 non-touch scenarios.
The results showed that the overall accuracy was around 96%, which means it is generalizable.
Finally, we combined our touch sensing technique with the motion data from the IMU to see whether
our technique is good enough to sensing different touch gestures.
We used 28 gestures, including the graffiti uni-stoke gestures as well as left and right
swipe.
The result showed that the accuracy was around 90%.
We also built some applications to show the rich potential of our technique.
In this application, we use the finger as a mouse to control the computer cursor.
Users can right and left click, move the cursor or scroll.
In this demo, we extend the input space of a smartwatch to the back side of the hand.
In this demo, we transfer our hand into a controller.
Users can use different hand gestures to perform different commands.
Here we control the music volume and pause the music.
That’s all for my presentation, thanks for attending