- I'm Simon Per-Roh and I will be the chair
for this session.
I'm very happy to see so many people.
I would like to apologize, I tried to have
a session plan at 1PM, but unfortunately
we have to get it very early in the morning,
so thank you for coming.
This session will be especially exciting,
because we have a lot of honorable mention.
First talk will be SkinTrack
by Carnegie Mellon, and will be presented by Yang Zhang.
- Thanks for the intro.
My name is Yang Zhang, and I'm a PhD student
at Carnegie Mellon University.
Today, I will talk about a project called SkinTrack.
This work has been done with my colleagues,
Junhan, Gerard, and my advisor Chris Harrison.
As we all know, smart watches are small,
but powerful computers.
However, in the case of smart watch, the screen
is so small, but our fingers are big.
We wondered, can we spill interactions
out onto the surrounding skin, and how to do it
efficiently and robustly.
We're not the first who considered
this interesting possibility.
Lot of previous approach has been tried.
For example, the most intuitive way
is to put a touch sensitive overlay
on top of the skin, however, it's always desirable
to use remote sensing, and avoid
instrumenting our skin directly.
For example, people use bio-acoustic vibrations
to localize finger touches, and detect
directional swipes, however this approach
is very low resolution.
The only work we saw that can support
continuous finger tracking, use camera
worn on the shoulder.
As we can see, it's quite obtrusive,
and it needs a clear line of sight to our skin.
Lastly, we could use a proximity sensor
to enable discrete touch buttons on the skin.
However, this approach only detect four touch locations.
Through this previous work, we can come up with
some ideal sensing properties.
First, it should support continuous finger tracking.
Also, it should not depend on a clear line of sight
to the skin.
And it should be non-obtrusive,
and ideally it can be integrated into a smart watch.
We developed a small tracking system
that we attempt to achieve all the above
sensing properties.
We call this system SkinTrack.
As we can see in this image,
SkinTrack has two components.
A signal emitting ring, on the right
and the sensing band.
Before I talk about how system works,
let me first explain what the physics behind.
Our sensing principle is based on the fact that
electricity takes time to travel.
In the case of a electrical signal,
it will reach different physical locations
at different times.
Even though this time latency's very small,
but in the case of high frequency signal,
it will result in a observable phase shift.
And in the case of human body,
the skin has a Dielectric constant of 17.
That means if we have a 80MHz AC signal
it will travel through our skin with an
wavelength of 91 centimeter,
and each centimeter it travels, it changes phase
by roughly four degree.
We conduct very early experiment to verify this.
Here is the setup.
We put a signal emitting ring to the finger.
The signal propagate out of our finger
and radiate out through the skin.
It is then collected by the two electrodes.
On the oscilloscope, the yellow signal is collected
from the left electrodes and the blue signal
is collected from the right electrodes.
The difference between these two signals
is shown in purple.
We can see the phase difference
between these two signals, but more importantly,
the phase difference changes correspondingly
to finger movement.
And here, how it works in SkinTrack.
We put a signal emitting ring shown in yellow
and first one pair of electrodes, shown in red.
When the finger touches on the skin,
note the distance between the finger
and the two electrodes are different.
This result in an observable phase difference.
When the finger moves to a next location,
the phase difference changes.
This can be used to detect Y position of the finger.
Now we add more pairs of electrodes
to be sensitive to X axis movement.
With this 2D tracking ability,
we can power interactive applications,
for example, we can draw an "M"
to quick-launch a music app.
We conducted series of experiments
to verify our sensing principle.
Here, we draw six, seven dots
on the user's wrist, along the wrist band
and we compare what we measured
with our predictions.
In this chart, the red line is based on
our theoretical model, and all the blue dots
would be measured when we touch on these seven dots.
As we can see, these two signals matches each other
pretty well, pretty well.
Here's also another experiment, when we move
the finger horizontally along the arm.
Again, these two signals matches pretty well.
Even though we still see mismatches
between the two signals, that is because
the electrode we use are copper patches
instead of the absolute point we're assuming of our model.
And also our arm is pretty curved.
It is not absolutely homogeneous,
which makes the signal propagation more complicated
in reality than in theory.
Now, I'm gonna show a live demo of SkinTrack.
Here is the wrist band of SkinTrack.
As we can see, there are several electrodes underneath.
I'm gonna put it on my body.
Because different user have different body,
SkinTrack has to be calibrated per user.
I will show you how this calibration's done.
Now I'm gonna put on the signal emitting ring.
As we can see, when I touch on my skin,
the signal changes dramatically.
So we have a classifier, whose only job
is to detect whether there's a finger touch
on the skin, and this data is fed
to that classifier.
Now I'm gonna calibrate this system real quickly.
I'm gonna touch on this nine crosshairs sequentially.
Now my finger is away from the skin,
so nothing show on the screen.
When this hover, it shows
and then I'm gonna touch, move to the right,
to the left,
up and down.
I'm gonna draw a circle,
hover and then away.
As we can see I only trained this
system very quickly, but it should
give you an idea what the accuracy of SkinTrack.
That's my live demo.
(audience applauding)
Now let's talk about how we made these.
To give little bit more detail about SkinTrack,
we used an oscillator powered by a battery
to make our signal emitting ring.
As we can see, it uses a fairly big battery,
but it gives us a 15 hour battery life.
As I will tell little bit more at the end
of this presentation, there are always ways
to shrink this battery.
Now for the wrist band, we use four pairs
of electrodes, two pairs of them are
sensitive to X movement, and the other two
are sensitive to Y movement.
We used four phase comparative chips.
Each of them takes input from one pair of electrodes,
it outputs voltage to indicate
the phase difference between the two signal.
These voltages are then sampled by
this Ath-Now microcontroller,
then sent to a laptop through
Bluetooth low energy for further computation.
We conducted a user study to evaluate SkinTrack.
It involved 12 participants.
As we can see, smart watch naturally
segment the user's arm into four locations.
There are front and back of our arm,
and front, back of our hand.
We evaluated SkinTrack on all four input locations.
For each input location, we collected three
training round, then followed by three
live testing round, then we asked
the participant to remove the wrist band,
relax, and put it back on.
Followed by three another testing rounds.
Not gonna go into detail
about this, but in short,
we collected three touch states.
They are finger out of range, hover and touch.
In total, we collected more than 100,000 test instances,
all labeled in real time.
And here is our result.
The result I'm gonna talk about here,
are all generated in real time by live classifier.
For touch state detection, here is the computation matrix.
It shows that SkinTrack is able to detect
with three touch states
at 98.5% accurate.
If we only care about touch and not touch,
we can just like attach screen desk,
we can combine hover and out of range together,
and that accuracy's even higher.
For touch tracking accuracies,
here are the four input locations
and each location has nine crosshairs.
We've rendered all touch points
from our 12 participants here.
As we can see, these touch points clustered
over across this crosshairs,
but still we see some variances.
That's because, first, the tracking inaccuracy.
The user error.
And finally, different user have different body,
thus different offsets.
Now we draw two sigma ellipses here.
In this case, smaller ellipses indicate higher accuracy.
We calculate the diameter button must be
in order to achieve 97.5% accuracy.
Here's the result.
The result indicates that SkinTrack is
average button size of 21 millimeter
to achieve this accuracy.
Luckily, previous work also used this evaluation,
so we can easily compare SkinTrack with them.
One result shown in blue is from OmniTouch,
which is a computer vision based system.
As we can see, our result is about comparable.
Also, Holz et al report a similar result,
but a capacitive touch screen is a 15 millimeter
button diameter to perform accurately.
Remember that we collected three testing runs
after the band was worn by the user.
The reason we want to do this is because
we've found most biosensing systems
don't work at all after they're taken off
and put back on.
Almost all system need to be recalibrated
each time it's worn, which is clearly not ideal.
Here is the result for SkinTrack.
We can see the button diameter increased
from 21 millimeter to 32 millimeter,
which is just not as accurate,
but still, it works.
We also run a series of supplementary evaluation
for SkinTrack.
And I'm gonna go through this result very quickly.
First, we double the density of the crosshair
from 30 millimeter interval to 15 millimeter interval.
And in this case, a bigger circle
indicate higher error.
As we can see, touch point's
farther away from the wrist band
has higher tracking error,
which is predicted by our theoretical model.
Also, we evaluated SkinTrack to detect
the discrete finger touches
and the result shows an accuracy of 91.8%.
We also evaluate SkinTrack when there's
a thin clothing between the finger and the skin.
And it shows no significant difference
between the two conditions.
Also, skin moisture doesn't have any significant effect
on the touch tracking accuracy.
We even put the ring on a different finger.
Here, we put the ring on the ring finger,
and still used index finger to touch,
and there's no significant difference between them.
With this 2D tracking ability,
we proposed some interaction modalities
based on top of that.
Just to remind you what SkinTrack can do,
here's a video shows skin track can detect hover,
touch, and even it works through thin clothing.
Now, with SkinTrack we can support directional swipes.
We can detect discrete touch positions.
Also 1-D slider will be a piece of cake.
Finally, it support continuous finger tracking.
Like many sensing systems, SkinTrack has limitations.
For example, the major obstacle comes from
the sensing stability across time.
We saw the touch tracking accuracy decreased over time,
and to solve that, we hope to use longer training process,
superior electrical materials, and better ergonomics.
Also, for the power consumption,
remember that this ring can be
used for 15 hours, is quite long,
but it emits a signal once it's powered on,
so it's a constant signal.
In order to reduce the power consumption,
we can reduce the duty cycle,
or we can even dynamically emits the signal,
only when it detects the fingers touching on the skin.
And finally, we found the physical
contact condition between this finger and the skin
can affect the reported position.
In order to solve that, we hope to
develop superior receiver filter
on the receiver side, and also we can
try different frequency on the emitter side,
or even combine different frequencies for the emitter side.
I want to conclude with some example applications.
To show what we can do with SkinTrack.
Here, with SkinTrack, a user can navigate
through a list of applications.
It can select by a right swipe,
list this of music, user can scroll up and down
and confirm by right swipe.
We can create shortcut on the skin,
for example, by dragging icons onto the skin,
then do a quick launch by simply
touching on the skin.
We can even play AngryBird,
which you know how hard it would be
if we play on this tiny little screen.
Also, we can use our backhand as a number pad
to dial a phone number.
Lastly, we can use SkinTrack
to support spatial gestures, for example,
we draw, "N" to quick launch a news app,
or silence our phone by drawing an "S."
We draw an "A" to quick launch an address book.
Here we can scroll normally on the screen,
or do a rapid scrolling by touching on the skin.
Thank you for your attention.
I'm happy to take questions.
(audience applauding)
- Thank you very much for this presentation.
We have time for two questions.
Do we have people who want to ask questions?
Yes?
- [Audience Member] (indistinct speech) Excellent work
and excellent presentation.
You talked a little bit in your study
where you had them take the band off and put it back on.
Can you also take the ring off and put that
back on (mumbling)
you talked a little bit about ability over time
and the session (mumbling).
Add to that question, talk a little bit more detail
about how you would address the session dependence.
- That's a good question.
In the uses of it, we didn't take the ring off.
The reason for that is because we didn't find any
influence if we take the ring off
and put it back on, but the major obstacle
comes from the reworn of the wrist band,
so that's the most important thing we want to evaluate.
Also, how to solve these instability across section.
For now, we use a mylar layer
between the electrode and the skin,
so that ensured only a capacitive coupling between
the electrode and the skin.
That makes our system more immune to
the accumulated change across skin.
Yeah, definitely more work should be done
to solve that instability over time,
across user or across different reworn condition.
- [Daniel] Hi, Daniel Ashbrook from the Rochester
Institute of Technology.
I'm curious if the back of the hand
performance changes if you bend your wrist,
so it's no longer in plane with the ring,
or with the wrist band.
- It's affect a little bit, but
especially comes from the contact condition
between the electrode and the skin.
Also, it changes the path, the distance between
finger touch, and our
electrodes.
Again, I think one way
to solve that is just like the regression model,
or machine learning model being exposed
to different contact conditions,
and then it could be more robust across time.
- [Daniel] Thank you.