hello I am jeremy chan
a junior researcher in the knowledge
visualization department
of the Know Center which is a research
institute based in Austria.
In this presentation, i will be briefly
walking through the main findings from
my paper
titled, Designing a sensor glove using
Deep Learning.
To start off, a sensor glove can be
defined as a glove with sensors to
extract information
concerning hand movements. Here i present
a self-made glove of our own. It is
fitted with an excessive amount of
sensors,
which includes IMUs, flex sensors
pressure sensors,
gyros, magnetometers and etc.
With data of hand movements, one can
attempt to recognize hand gestures.
Some examples of such are illustrated
from the figure,
where a gesture is defined as a movement
of the hand to express
an idea. The objective
for gesture recognition is to classify
the data from the hand movements
into specific gesture classes. Such data
will be in the form of a time
series where the gesture would occur
within a period of the time series.
Data of 31 gestures were collected from
a total of 22 participants.
For each gesture the participant is
first shown a video clip
of how to perform it, they then perform
the gesture either 5 or 10 times while
being recorded.
With the collected data, we process it
with the use of machine learning to
classify each time series sample to a
gesture class.
Specifically through the use of neural
network, which is a mathematical model
that can take a time series sample as an
input and output a probability
distribution regarding
which gesture the sample
belongs to. Through this, we have achieved
high true positive rates for
participants
by using data from all the sensors, which
we
know consists of some redundancy.
information from the model to make a
better glove design.
With the experiment and the data, we are
aiming to answer
three main research questions which can
be summarized as:
Number one, is it possible to create a
design with minimal
sensors whilst retaining a degree of
prediction accuracy?
Can the same be done where the focus is
to create a cost effective design?
Number two, is it possible to create a
personalized glove design tailored for
an individual solution?
Can the same be done for a general
solution? Number three,
how different are the set of sensors
from the different designs
and the different solutions? And we
answer them
through the use of important estimates.
They are a form of measures to rank the
significance of sensors
in relation to the classification of
gestures.
We achieve this by introducing a layer
of weight in between the
input and the neural network. Within the
input,
there are multiple data channels which
represent the sensors.
Some sensors take up multiple data
channels, for such cases
variable sharing is done between the
channels.
The weights are all initialized to ones,
essentially serving as an identity
function in the beginning.
When considering cost effectiveness,
another neural network is used to
account for the additional constraint.
Again an additional layer is used which
utilizes the weight
from the original important estimate
layer.
Instead of the standard way of training,
we train in two phases.
The first phase, we freeze the important
estimate layer and train the model
for a baseline result. The second phase,
we take an existing baseline,
freeze the model and train just the
importance estimate layer.
After the training, the important
estimates which originally serve as the
identity function by being initialized
as ones,
will now vary from the training. The
value
of the weights can then be normalized
and ranked.
Here is a heat map of the important
estimates. On the y-axis,
we have two participants, coded JQ28 and
AL29,
they are examples of solutions to be
used for a personalized glove design.
On the y-axis, we also have the average
of all important estimates from all
participants, labeled 'All',
which is an example of a generic
solutions for a general glove design.
On the x-axis, they are the
data channels which correspond to a
sensor on the glove.
This is depicted by a 's' followed by a
number.
The shade of blue signifies its
significance,
where the darker shade means it has
greater significance.
In this presentation, we are going to
focus on one participant,
labeled JQ28. With the importance
estimate, we can proceed to
remove sensors that have ranked poorly
in terms of significance.
Here we remove 20 and 40 percent of the
sensors.
In the binary heat map of the
participants, the y-axis represent a
different
percentage of sensor removal and the
x-axis consists of
the data channels which correspond to a
specific sensor of the glove,
labeled with a 's' followed by a number
again.
For the heatmap, the highlighted sensors
mean they are retained
in the removal, whereas transparent
sensors mean
they are removed. The sensors removed are
also highlighted
in the diagram of the glove above. For
this particular participant, we can see
that
the flex senses at the base and tip of
the thumb,
the pressure sensors of the middle and ring
finger,
as well as the IMU on the wrist are
being targeted
for removal. We verify the effect of
removal by comparing the metrics with
an original baseline. Here we compare
models that focused on classification
accuracy, the heatmap on the left, and
models that focus
on cost effectiveness, the hit map on the
right.
On the heatmap, it depicts the true
positive rates for the same participants.
The x-axis consists of 31 gesture
classes and on the y-axis
they are the different true positive
rates from the baseline model, as well as
the models with different percentage of
removal. Here we can see that as more
sensors are removed,
especially around after 40 percent
the performance of the model also
diminishes.
This effect is more visible from the
model that focus on prediction accuracy,
which is
the heat map on the left.
In conclusion, we introduced the use of
important estimate
to identify significant data channels in
time series data.
We have illustrated it with an example
of a sensor glove for gesture
recognition.
We have demonstrated a method for
removing redundant sensors with a way to
verify if they are
indeed redundant. And for future work we
aim to perform more analysis
on important estimates to perhaps
improve upon the ranking method for
removing
redundancy. Thank you for attending my
presentation,
I hope that it was informative. More
details and information regarding the
method is available on the paper itself.
We hope other researchers will find it
useful and adopt our methodology in
their design.