(clears throat)
- Okay, hi everyone, my name is Jun Gong.
I am third-year PhD student at Dartmouth College.
Today, I wanna talk about Pyro,
thumb-tip gesture recognition using
pyroelectric infrared sensing.
This work was in collaboration with
Yang Zhang from Carnegie Mellon University,
Xia Zhou, and my advisor, Xing-Dong Yang.
Before I talk about motivation and related work,
first of all, I just want to show you
what is Pyro and what Pyro can do for us.
So Pyro is a sensing technique that uses
passive infrared sensor to detect
micro thumb-tip gestures.
Users can use the thumb-tip to gesture
on the tip of the index finger, like drawing a circle.
In this video player app, users can draw a circle
to launch the app.
Users can also rub the fingers to pause the video
without touching the screen.
I will talk more details about implementation
and applications later in this talk,
but before that, let's get back to the motivation.
Smart watches are getting more and more popular these days
because they are small, wearable, and powerful.
However, the small factor of the smart watch
makes it hard to manipulate it.
Many previous work has tried to enlarge the input volume
of the smart watches.
Next, I'm gonna show you a few examples.
Digits uses just one camera to
capture small finger motions.
It can achieve very high accuracy.
However, it needs a lot of computational resources.
Thumb-tip gestures can also be sensed
by using magnetic sensing, but the user has to
instrument their fingertips with magnets and sensors.
The closest related work of Pyro is Soli from Google.
It can also track tiny finger motions
by generating active radar signals,
and it has already been integrated into wearable devices,
like smart watches.
However, we realized that it is power consuming
to generate such a high frequency radar signal,
especially in a wearable context.
This partially motivates our work.
So in this work, we explore a different technique
called pyroelectric infrared sensing.
We use this technique to detect
micro thumb-tip gestures.
Compared with existing methods,
pyroelectric infrared sensing has two unique benefits.
First of all, the passive infrared sensor,
referred to as PIR sensor, made of pyroelectric elements,
is quite sensitive to the thermo infrared signals
radiated from our fingers.
Compared, sorry
radiated from our fingers.
Without generating active signals,
the sensing technique itself
is totally passive and power efficient.
Second, cooling is a known challenge
for wearable devices.
As you can see in this image,
of course, this is a little bit exaggerated,
but you can see my point.
However, PIR sensor generates little heat
so it requires no cooling.
This is an important benefit for wearable devices.
Before I share more,
before I talk about the sensing principle,
I want to share more on the passive infrared sensor.
So actually, passive infrared sensors
are everywhere in our daily life.
They are commonly used to detect
whether a human has moved in or out of the sensor's range.
We can usually find them in many alert systems
or automatically activated lighting systems.
So here is an example showing
a common usage of a PIR sensor.
As you can see in the oscilloscope,
the signal, digital pulse will be generated
when human motion is detected.
So instead of using the digital output of the PIR sensor,
we built a sensing system from scratch
so that we can get and adjust
the analog output of the PIR sensor,
which contains more information.
As you can see here, the PIR sensor
is quite sensitive to small finger motions.
So next, let me explain the sensing principle.
PIR sensor itself has two pyroelectric elements
which are quite sensitive to the
thermo infrared signals radiated from the human body.
When the sensor is idle, both of the elements
detect same amount of the thermo infrared signal
radiated from the environment.
So, no differential change will be generated.
When one body passes by, it first intercepts
with one half of the sensing area,
which generates a positive differential change.
When one body leaves the area,
the reverse happens.
A negative differential change will be generated.
So, by taking advantage of this sensing principle,
Pyro supports six fine-grained micro thumb-tip gestures.
Drawing a circle with the thumb-tip,
drawing a triangle with the thumb-tip,
rub the fingers, drawing a check mark with the thumb-tip,
drawing a rectangle with the thumb-tip,
and drawing a question mark.
So, before coming to the implementation details,
here is a short demo.
From the analog raw data, we extracted
a series of features, which are passed into
a machine-learning classifier.
The classifier will perform real-time classification
of six micro thumb-tip gestures.
As you can see here, the classification result
is shown on the left, and the raw data,
the bottom graph shows the raw data.
The blue bars on the top of the display show
the extracted features.
So here is a triangle,
rub fingers,
a check mark,
a rectangle,
and a question mark.
If you pay attention to those graphs,
you will notice that the raw data and extracted features
of six micro thumb-tip gestures
are quite different.
So next, let's look into the implementation details.
So here is the Pyro system.
The Pyro system consists of the PIR sensor
with a Fresnel lens and a customized sensing board
including a microcontroller, Bluetooth module,
and the analog frontend.
So the thermo infrared signal radiated from fingers
is concentrated by the Fresnel lens,
which makes our system become more responsive
to the tiny finger motions.
The signal is then passed into a band-pass filter
to reduce the dominant noise caused by the power line.
After noise is removed, the signal is then amplified.
We carefully tuned the amplifier gain,
to have an optimal sensing range
from 0.5 centimeters to 30 centimeters.
All of the raw data are streamed into a laptop
for gesture recognition.
Before we feed the data into the machine-learning classifier
to extract relevant features from it,
we decided to use a feature extraction package
called Tsfresh, which can extract hundreds of features
in time domain and frequency domain.
We pick the top features based on the feature importance
provided by the machine-learning classifier
and then we use the random-force classifier
as our classifier,
since it has already been proved accurate and robust
in many gesture recognition applications.
So, in order to see how accurate Pyro is,
we conducted a study.
We looked at ten participants and asked them
to perform six micro thumb-tip gestures,
as I have introduced in the previous slides.
Except for gesture direction, we did not give any
instructions on how to perform these gestures.
Each gesture was repeated for 20 times in each session.
And in total, we collected three sessions.
Participants were required to leave the desk
and walk around the lab between the sessions.
In total, we collected 3,600 gesture samples for analysis.
So here are the evaluation results.
The y-axis shows the accuracy.
For each participant, we conducted two-fold
cross-validation, where half of the data,
we used for training,
and the remaining half were used for testing.
The average accuracy is about 93.9%.
So in order to measure the system's reproducibility,
we also calculated the leave-one-session-out accuracy,
by training the model using two sessions of data
and testing it on the remaining one.
We want to see if the model training a different time slot
can work.
So the average accuracy is about 84.9%,
which can demonstrate more realistic system performance.
So here are the confusion matrix
of cross-validation and leave-one-session-out.
In order to compare,
you can see in both graphs, the trend is similar
between the cross-validation accuracy
and the leave-one-session-out accuracy.
As you can see here, the rectangle achieves
the highest accuracy among the
six micro thumb-tip gestures.
A potential reason is that it consists of
many sharp turns, which will make the signal
more distinguishable than the others.
Besides the system's accuracy,
we also conducted a series of studies
to initially evaluate the system's robustness
against the environmental noises.
We recruited one participant and
collected two sessions of data,
in a controlled lab environment
without presenting any ambient noise.
And then we collected one more session of testing data
in different situations presented with
one specific ambient noise.
As for the ambient light, we collected
one more session of testing data in a dark room,
with luminosity from zero lux to 20 lux.
We also conducted one more session of testing data
in a bright room,
with luminosity of 200 lux to 300 lux.
Our study result shows that we can achieve
similar accuracy at leave-one-session-out accuracy,
which initially demonstrated our system's robustness,
against the ambient light.
We are also interested in nearby hand movement.
As you can see here, we ask one more person
to wave his hand behind the participant's finger.
And you can see there is still some wave over there,
but this will not affect the classification result.
So, we also considered the influence of hand temperature.
Imagine yourself just holding a cup of hot coffee
or a cup of iced Coke.
We are trying to see if the dramatic change
of finger temperature will affect
our system's performance.
It turns out that, in the hot condition,
our system works pretty good,
but in the cold condition, our system won't work
if the finger temperature is as low as the room temperature.
It is mainly because the PIR sensor signal
is proportional to the difference
between finger temperature and the room temperature.
So here's a video showing a system's performance
under different finger temperatures.
I'm trying to decrease the hand temperature.
The system still works, because the finger temperature
is still higher than room temperature.
Right now this is a cup of hot water.
I'm trying to increase the finger temperature.
So the system works.
So next, I'm going to show you two demo applications
to showcase the potential of Pyro.
We built a smart watch prototype
with a two-inch TFT display, 3D printed case,
and the Pyro system.
Our first application is a video-player app
on a smart watch,
as I have already showed you at the beginning of the talk.
As you can see here, I'm drawing a circle
to launch the app,
rub the fingers to pause the video,
rub again, the video can be continued.
Our second application allows user to interact with
the hand-worn display using thumb-tip gestures.
We augmented Google Glass using Pyro.
The sensing unit is placed beside the touch bar
near the ear.
So in our application, a check mark is a shortcut
for taking a photo,
a check mark with the index finger touching the touch pad
will take a photo and automatically
share it on Facebook.
So, like many sensing techniques,
Pyro has its own limitations.
First of all, its false positives.
Off screen movements like a human passing by the sensor
may generate similar signals
as the thumb tip gestures.
But we believe we can solve this problem
by triggering the sensing only when the hand is up.
Second, our study shows that different persons
may perform the same thumb-tip gestures
in different ways.
This means that the model needs to be trained
for each user.
In the future work, we are trying to develop
a more generous model to solve this problem.
Last but not least, although our supplementary study
initially demonstrated our system's robustness
against the environmental noises,
further evaluation should be done
in more diverse and more realistic settings,
like testing our prototype
in different outdoor temperatures.
For example, we can test it in Orlando or Alaska.
So, here is several take-home messages
that I want to share with you guys.
Pyro is an effective technique to sense
thumb-tip gestures using infrared signals
radiated from the fingertips.
Six micro thumb-tip gestures can be accurately
identified using Pyro.
Pyro has been initially proved to be
robust against environmental noises
in a controlled lab study.
So this is all I want to share about for today,
and I'm more than happy to take any questions.
(applause)
- [Audience Member] Hi very cool,
Jeff Blum from McGill University.
I'm wondering if you have any sense
of how much of the contribution
for the differentiation is from the amount of time
it takes to do the gestures,
versus the waveform that you have,
because the rectangular one for instance
seems like it might take the longest to do,
so you might be able to differentiate them better
just by having longer and shorter gestures.
- You mean, gestures in different time periods?
- [Jeff] Or just yeah, well it takes longer
to do the rectangular one right?
So the machine-learning algorithm may just say,
"Oh that was really long, I think it's the rectangle."
- Yes, could be one factor because
the rectangle costs a lot of time compared with rub fingers,
but because we did not give any instructions
on how to perform these gestures,
so different persons may perform at different speeds,
but our study results also shows that,
okay, we can still get a pretty good accuracy.
But this could be one feature that can
be extracted from the raw data.
- [Jeff] Thanks.
- We have time for another question.
- [Audience Member 2] Hi John Season from
Columbia University.
I wonder, do you have a sense of the sensitivity
of your method, if I move just one millimeter
can you sense it, if I move like 0.5 millimeter
can you also sense it?
- You mean...
- [John] The sensitivity
- Sensitivity, you mean will it cause
a lot of false positives or?
- [John] I mean if I just move slightly,
like 0.1 millimeter, can you sense it?
- Uh..
- [John] Like very tiny little finger
- Very tiny, what do you mean by very tiny
- [John] Finger motion, like this kind of motion,
very tiny, I was just wondering
to what extent you can sense it
- Could be, I have tested this sensor,
for now I'm trying to do more than gesture recognition,
maybe we can do some tracking stuff,
it could be, it can achieve very high resolution
but I'm not so sure about
the tiny movement you pointed to,
one millimeter or two millimeter.
- [John] Or maybe we can talk offline, thanks
- Yup, thank you.