I'm Yizheng from Tsinghua University in China.
Today I will introduce a smart ring
embedded with an IMU sensor.
The ring can sense finger touch on any physical surface,
for example this table or the wall.
So in this work I collaborate with Chun, Zhipeng, Shuchang,
Xiaoying and Yuanchun.
Here's the video.
The ring sense finger touch by the vibration
at the touch moment.
The accuracy is around 99%.
The methods can award unintentional touches.
Users can also tap with various tapping postures.
For example tapping with the middle fingers
or knocking with the knuckle.
So we first envisioned a use scenario to motivate our work.
So mixed reality headsets can render user interface
on any physical surface.
In the future, it is promising for users
to interact with any surface with direct finger touch.
To sense touch, it is straightforward to use cameras
on the Mixed Reality headset.
However, cameras have inherent drawbacks.
First, the occlusion problem.
Second, the accuracy problem especially in the depth
dimension.
For example, the state-of-the-art work explore
touch sensing.
With depth cameras of Hololens, a touch
is declared if the distance between the fingertip
and the surface descends below 10 millimeters.
These methods cause missed touches
and unintentional touches, which does not
provide a satisfying solution.
So to adjust this problem, we merge
the combination of the Mixed Reality headset
and a ring embedded with an IMU.
The camera on the headset is to detect the finger location
and the posture, while the smart ring is
to detect a touch moment.
We focus on sensing the touch moment
and achieve a high accuracy and a low-latency.
So we are not the first to detect touch by smart ring.
However, prior work did not focus on touch contact sensing.
They used simple threshold methods to sense touch,
getting an accuracy of up to 90%.
In this paper we used machine learning
to achieve an accuracy of 99%.
There are two parts in our work.
In the design part, we investigate user preference
on the tapping postures and the ring placements.
We identify a set of useful tapping postures.
Our technique can sense the touch moment
of all these postures.
For ring placements, users like to wear the ring
on the proximal phalanx of their fingers.
In the technical part, we find that the SVM-based methods
can outperform previous work for touch sensing.
To optimize the accuracies, it is the best choice
to wear the ring on the proximal phalanx of the index
finger and middle fingers.
I will introduce our work in three-user experiments.
In experiment 1A, we collected tapping postures
that most users are willing to use in daily life.
We first defined a comprehensive set of tapping posture.
For each posture, participants perform the posture
and then rate them in a questionnaire.
We explored the posture in a three-dimensional design space.
First, which fingers touch the surface?
Second, which part of the finger touches the surface, including
the finger pad, fingertip, knuckle, and the fingernail?
Third, is the hand in a closed fist palm
or in an open palm position?
So finally we had 70 types of posture to be evaluated.
20 participants took part in the experiment.
They touched with 70 postures on a desk and on a wall,
and then rated for each posture through a questionnaire.
The questionnaire evaluated each posture on a 7-point Likert
scale by three scores--
the physical and mental ease of performing the posture,
the ease of remember the posture, and the willingness
to use the posture.
And here is the result of the user preference.
The left figure shows user preference
of all the postures in horizontal and vertical
orientations.
The red color represents a better user preference.
Users prefer to touch with the index, the middle finger, two
fingers, or three fingers.
They do not like to touch with the ring finger,
and they accept only touching with the finger pad,
fingertip, and the knuckle of a finger.
So it is not a good idea to design a touch gesture that
involved the finger nail.
This figure shows the top 10 tapping posture.
The result may help the designs of touch interaction
in the future.
In the technical part of this paper,
we also sense the touch events of all these posture.
We conducted experiment 1B to investigate user preference
of ring placements.
Here is the result. Users prefer to wear the ring
on the proximal phalanx of their fingers.
They do not like to wear a ring on the distal phalanx
of their fingers.
Now we step into the technical part.
We collected motion data from the smart ring
when participants touch with different tapping posture
and different ring placement.
Based on the motion data, we designed the touch
sensing algorithm.
Before we look into the details, we have the conclusions.
First, a valuable information in the smart ring
is which machine learning can largely
improve the touch sensing.
Second, it is the best choice to wear the ring on these two
positions.
They optimize the performance of recognition.
Third, after a touch, the vibration
can transmit to any ring placement
within 20 millseconds.
Thus, ring-based touch sensing can be low-latency.
12 participants touch on the surface
and we collected the motion data.
The experiment had three independent variables--
the surface orientation, the ring placement, and the tapping
postures.
And we also collected mid-air gestures as negative samples,
such as swiping in the air or drawing a circle.
This figure shows our apparatus.
The participants wore a smart ring and touch
on a low-latency touch screen.
The IMU ring can collect angular speed, acceleration,
and the direction of the gravity.
The frequency was 200 hertz.
The touch screen collected the ground truth of the touch
down event.
The latency was below 5 milliseconds.
We used the leap motion to check the hand.
Next, I'll introduce the touch sensing algorithm.
We have the details in the paper.
Here I'll just present the final algorithm.
We first train a SVM model to classify the tapping moment
and the mid-air gestures.
The left figure shows an example of the positive samples.
The positive samples also include
the other tapping posture like knocking or using other fingers
to tap, while the right figure shows the shows an example
of the negative sample.
And this figure shows the motion data one touch.
The red line represents the ground truth
of the touch moment.
We sample a time window of 50 milliseconds around the touch
moment as positive sample.
We sample the time window before the touch moment
as negative sample to avoid reporting touching events
in advanced.
For each dimensions of the IMU data,
we calculated its maximum, minimum, mean, skewness,
and the kurtosis.
We obtain a feature of 45 dimension
and use SVM to train the model.
The figure shows the accuracy of the model over ring placement.
The red color shows the best ring placement
in terms of accuracies.
Their accuracies are around 99%.
The right figure compare our methods with two baselines.
The two baselines are the threshold methods
based on baseline smart ring and the optical methods
based on camera.
The results show that our method is better.
The classifier was not enough for sensing touch in runtime.
First, you might trigger a repeated touch event.
Second, the classification accuracy of 99%
was not enough to avoid unintentional touches.
To address this problem we packed a classifier
into a simple algorithm.
First, the algorithm does not report touch events
if there has been a contact in the past 50 milliseconds.
Second the algorithm reports touch events
only if the classifier detects two consecutive frames
of touches.
And we evaluate the algorithm in the third experiment.
In this experiment, we evaluate our algorithm
by the "Piano Tiles" game.
We compared our methods with the camera-based method.
In the task the participants can see his virtual hand
in a display.
The objective was to tap on the black tiles
as they appear from the top of the screen,
while avoiding the white.
The screen moved manually at the rate
which the tiles were touched.
Participants touch 100 black tiles to finish the game.
They were asked to touch as fast as possible.
And here is the result.
Our method outperformed the camera-based methods
in accuracy and completion time.
The task in this experiment required participants
to touch really quickly.
The camera methods could not handle this situation well
because sometimes the users did not
lift their fingers over the first half
of the camera-based methods, which is 10 millimeters.
So the delays of our methods is low and stable
while the delay of the camera methods was not stable.
Some participants report that the camera-based methods
sometimes even sensed a touch in advance.
So here is the conclusion.
In this paper we focus on touch events sensing by IMU ring.
We identify usable tapping posture
according to user preference.
We find that it is the best choice
to wear the ring on the proximal phalanx of the index
or the middle finger.
Our work has some limitations.
The major one is that we can only detect touch down events.
It will be valuable to sense touch up in the future
so that more touch gestures, such as long press and swipe,
can be recognized too.
So that is all of my presentation.
Thank you for listening.
I'm happy to take the questions.
Thank you.
It was a great talk.
I was wondering if you did any experiments
on putting the IMU on the wristband
and trying to see if you can still detect touch
using this method?
Sorry?
Did you do any experiments on putting the IMU
on like a wristband instead of a ring?
Sorry?
So instead of putting the sensor on the finger,
if you put it on the wrist.
This is a good question.
At first I did try to use a watch to sense this
but it is not working.
So the vibration of the touch can only
transmit to this nice feelings of the finger.
And no matter how soft the user tapped the desk,
there is a vibration.
But sometimes it is the peak of motion data is high,
sometimes it is low but the machine learning
method can handle it well.
But if we use a watch, smartwatch, to send it,
the motion data is just random sometimes
when a user touch really soft.
So it's Dan Vogel from the University of Waterloo.
So I really, I especially like the study
of different type of touch gestures,
that's tapping gestures, that's going
to be really interesting and really useful.
When you did your experiments to understand
the accuracy rate what was--
I can imagine many false positives, right?
If I tap against my pants, or if I pick up a cup.
It's very impressive 98%, but is that
using the type of wiggling the fingers in space as you're not
tapping ground truth?
Or did you use a more diverse set
of realistic and motions and touches?
Thank you.
That's a good question.
So in our work, the negative samples
are the mid-air gestures.
We did not correct the gesture including the collision.
Sometimes when a collision occurs,
it will be a false positive sample.
So we think that we can add a button on the ring.
When you are not using the ring, you can just turn it off.
And when you turn it on, once you don't collide or hit
your finger, there will be no false positive.
Yes, thank you.
Is this on?
OK, awesome.
Thank you for the talk.
So some of your motivation was coming from mixed reality input
where people would want to be tapping
not only on a table or a hard surface but also on the body
and potentially softly.
Did you test these and did you only evaluate in the criteria
that you collected your test data in?
Or did you also do some aspects of testing in the wild?
Thank you.
So you mean the surface, the various surface?
In this work I just test the rigid surface,
like the desk or the wall.
And after the submission I have tried it on a screen.
If you tap on the leg, it is also working.
It was actually just exactly the same question, so great talk,
thank you very much.
My question kind of builds on top of that question.
What sort of data points were collected in terms
of positives and negatives?
And how well does it generalize across surfaces?
Sorry?
So how many data points you collected for positives
and for negatives.
And how well does it generalize across users,
or do you need to fine tune it per user?
We have the details in the paper.
In the paper, we describe the negative sample,
how the public collected the negative sample
and positive sample in details, so maybe we can see the paper.
The data is not that large.
We could tell participants to thread the data.
One participant collected data for one hour, and it is enough.
All right.
Let's thank the author.