- I'm Jess McIntosh, I'm from the University of Bristol,
and I'm here today to present my work SensIR,
which is detecting hand gestures with a wearable bracelet
using infrared transmission and reflection.
And this work is done in collaboration with my colleague,
Asier Marzo and my supervisor, Mike Fraser.
So let me begin with a brief introduction
and set the motivation for this work.
So we can use hand gestures explicitly to conveniently
control wearable devices, as seen in previous work,
and during this session with Pyro.
So we can use this to control devices
in our surroundings too.
But we can also detect our actions that we do implicitly,
and this allows us to monitor daily activities,
or keys for contextual awareness, or bad habits.
But for either use case, it's clear
that there are mobile applications.
And therefore, the sensing apparatus must also be portable.
So smart watches are already seen as a socially
acceptable placement for a wearable, and the
watch strap conveniently allows space for our sensors.
And so for these reasons, a lot of work has gone into
placing the sensors here to detect the hand pose.
So there's a variety of different sensing methods
that's being used to achieve this,
and there is associated properties
that determine the method's strengths and weaknesses.
But in this work, we don't compare the sensing methods.
Instead we simply try to improve the accuracy
of the existing infrared sensing method,
without sacrificing size, efficiency, or practicality.
So there's been lots of previous work on wrist-worn
gesture detectors using infrared, but the basic principle
of operation is the same for all of these.
And indeed for other techniques too.
So this principle relies on the deformation of the
shape of the wrist, as gestures are performed,
due to the moving anatomical parts within.
So let's look at the previous work in a bit more detail.
So here's a cross-section of a wrist, and this might be
a typical bracelet with the infrared sensors embedded.
And there will be multiple pairs of
emitters and receivers of infrared light.
So these each sense the distance to the skin, and
thereby sensing the deformation of the shape of the wrist.
It is important to note here that each pair
works individually in this configuration.
So I will refer to this from now on as a
one-to-one emitter to receiver configuration.
But we know that infrared light of particular wavelengths
can actually pass through flesh diffusely.
So inspired by this, we made small changes to the previous
configuration, where instead now, every infrared sensor
measures the light level when any one LED is emitting.
So now in addition to what we had previously,
the adjacent sensors also pick up reflections from the skin,
and we also get some amount of light being transmitted
through, which infers distance, and possibly captures
some of the morphological changes that
happen within, such as bends moving.
So the device can multiplex through each LED
to collect a matrix of brightness values.
So we have a ray for each emitter,
and each column representing the receiving elements.
So then we feed the raw data values from this matrix
into a neural network, in order to classify gestures.
So here's what our prototype looks like.
There are 14 segments in total.
And each segment has an LED and a photodiode,
with a peak sensitivity of 860 nanometers.
And this is all connected to a signal generation and
acquisition circuit, which then relays the information
back to a PC for classification.
So to test this configuration, we conducted a user study,
which the participants found incredibly interesting.
So we chose a variety of gestures,
including pinchers, multi-finger, and wrist.
And each of their gestures were performed 10 times.
So after collecting the data from the study,
we wanted to compare two things.
So we first need to compare the predictive accuracies of
the system, when using the traditional one-to-one system
against our new one-to-many system.
And secondly to see how the number and arrangement of the
different segments might affect the predictive accuracy.
So here are our results.
So it's clear that using all the features from the
one-to-many system is better than using
only single reflective measurements,
and significantly so for the first three.
It is worth noting that in fact the smart watch
configuration did not suffer from a major decrease in
performance, nor did actually halving the elements here.
But actually four segments did suffer quite a lot, but that
could be used for gesture onset detection for instance.
So in addition to this study, we did
some additional preliminary studies to investigate
the effects of cross session misalignment,
non-sedentary postures, and skin coupling.
So sensor misalignment occurs when the device
is taken off and back on between sessions.
So there is usually some misalignment between the sensors,
and this can lead to a classification error,
due to a shift in the features.
So detecting this misalignment is the first step towards
calibrating the device to account for the shift.
So we tried using a neural network regressor on the data
from the neutral hand pose, and we found that
we could in fact detect the shift reasonably well.
So the second study that we tried
to do was a non-sedentary study.
So during the main study, the users were sitting down
in a fixed position and instructed to keep their arm still.
But people are likely to be moving around and moving
their arms in different positions when in real use.
So we tested the system with three different
arm elevations and three different arm rotations.
And we found that the accuracy still remains quite good
in both cases, but as expected the rotational changes
to the forearm makes classification more difficult.
And finally, there may be some circumstances when the
device is maybe not tight around the wrist in some places,
and the sensors aren't quite touching the skin.
So we conducted some simple tests, purposefully
introducing gaps and even putting some latex
between the band and the skin.
And we found that the accuracy was largely unaffected still.
So to summarize, we've demonstrated that a one-to-many
system is superior to the previous one-to-one systems.
And this only requires a minor modification
to actually achieve this.
And finally, some preliminary work suggests
that calibration could work quite well,
and the technique is fairly robust to natural movement.
Thank you very much for listening.
I'll take any questions you have.
(audience clapping)
- [Audience Member] Hi, my name is Kon Park from KAIST.
I think for your presentation, I have
a question about the lights through the wrist.
You said two kinds of lights pass.
First one is reflected by skin, and second one
is a light that goes through the wrist.
My question is that how much does the light
through the skin affects to the final visual?
Is it really important to measure
those lights that goes through the skin?
- So you're asking me about the
dynamic range of it or something?
- [Audience Member] No, I mean is it important to measure,
I mean for example, if the light is emitting this part,
and you are also measuring the sensor from this part right?
Is it important to measure those far...
- Yeah it does depend on the distance.
In some cases, some users wrists were quite large,
and therefore the light doesn't
transmit all the way through.
So often times it's just sometimes the ones around the
edges, and not quite opposite, but sort of four o'clock,
five o'clock position can be detected.
- [Audience Member] So you're saying that major part
of the, I mean is it important to
measure the adjacent sensors, right?
- Sorry?
- [Audience Member] I mean you're saying
it is important to measure values in adjacent sensors?
- Yeah.
- [Audience Member] Okay, thank you.
- Any other questions? I have one for you.
So to what extent is this robust to environmental
interference, like from other light?
Like if I'm on a bright, sunny beach,
and I have a lot of ambient infrared and things?
But it's all kinda inside the wrist.
So is it actually pretty safe against all that?
- So there is some light leakage, but I think you can
use techniques taken from other fields,
such as the infrared brain imaging stuff.
And they use techniques where they can mitigate
this kind of light leakage by just subtracting
the background noise from your readings.
- Okay, great.