we've got left and right
not registering anything forwards and
backwards
up and down all right
let's give a high five a shot
oh my gosh that's awesome i can't
believe that worked
in the first video in this series i
talked about how we can use deep
learning to recognize complex patterns
in data
and i sort of jokingly stated that one
example
could be to recognize and count high
fives using an accelerometer
well it turns out it wasn't so much of a
joke because
this is me testing it out on a network
that i just trained
and it's working surprisingly well so in
this video i want to talk about how this
was accomplished
the data pre-processing i did and why
transfer learning made it possible for
me to solve this problem in just a
couple of hours
so i hope you stick around for it i'm
brian and welcome to a matlab tech talk
for this project i'm using a simple mems
accelerometer that's wired to an arduino
the arduino reads the sensor and prints
the measurements onto a serial bus
which is then connected to my computer
via usb and i'm reading the serial bus
with matlab
and plotting the last second and a half
of data onto the screen
and you can see as i move the
accelerometer around how it changes the
plot accordingly
now this is just sort of random motions
but what i really want is
something that can recognize a very
specific sequence of accelerations
the high five
now of course this was kind of a simple
high-five motion since the cables i'm
using are quite short
but this is going to be the motion that
i'm going to teach a deep neural network
to learn
and you're just going to have to use
your imagination that this is a
full-scale
five in all of its glory
okay so this is the acceleration pattern
that i'm looking for
and the general approach i'm taking to
solve this problem is to turn this
three-dimensional signal
into images that i can use to train a
deep neural network
and this is similar to what we did in
the last video where we pre-processed
audio signals
with the short time fourier transform
and we used it to create
a spectrogram and i could do that here
as well and it would probably work just
fine
but what i'm going to do instead is use
the continuous wavelet transform
to create a scalogram a scalogram is
another time frequency representation of
signals
but it differs from the spectrogram in a
way that makes it more suitable for
signals
that exist at multiple scales that is
signals that are low frequency and
slowly varying
but then are occasionally interrupted
with high frequency transients
they're useful for signals like ecg's
that have this characteristic heartbeat
pattern
and as it turns out they're useful for
the occasional high five
in an otherwise slowly moving hand
now i'm not going to go into the details
of the scalogram in this video
since there's a whole bunch of other
things that i want to cover but i left
several great resources in the
description that you should check out if
you want to learn more
all right so the first thing i want to
point out is that we need
three images to represent a high five
and therefore
i have to feed three images into my
network architecture
but luckily many architectures are
already set up to input
color images which have a red blue and
green channel
so to just fit with those existing
architectures what i did was assign the
x
scalogram to the red channel y to the
green
and z to the blue and then combine them
to create a single color image and
this sort of pinkish volcano with some
smoke coming off of its peak
is the scalogram representation of a
high five
and this image looks pretty cool right
but i think it gets even cooler
if we view the scalogram on the
real-time acceleration stream
each sample time i receive a new
measurement from the sensor
i then update the buffer of the last few
seconds of data
and i create the scalogram you can see
how the patterns and colors
change as i move the sensor around and
what's really cool is that you probably
couldn't tell if i was doing a high five
by looking at the raw acceleration data
streaming by but i'm sure you can see
the characteristic pink volcano
streaking past every once in a while
it's a very obvious pattern and we can
now take this process data and
train a deep neural network to find that
obvious pattern
but in order to do that there are at
least two things that i need still
i need some training data with labeled
scalograms of high fives
and some with no high fives and i need
an architecture let's talk about the
architecture first
rather than design and train in
architecture completely from scratch
we can build on what already exists with
transfer learning
transfer learning is modifying an
existing architecture
and then retraining it to accomplish
your task rather than the task it was
originally trained for
and to understand what that means let's
revisit what we talked about in the
first video with the very
basic description of what these
image-based architectures are doing
they are looking for patterns in images
and it does that by looking for
primitive features in the early layers
like blobs and edges and colors and then
as you progress through the layers
it combines them into more complex
features and then ultimately combines
those into final patterns that can be
labeled
again this is an oversimplification of
these networks but this is a useful
oversimplification for describing
transfer learning let's assume that this
is a network that's fully trained
to recognize flowers in images now
obviously a network trained to recognize
flowers won't do a great job of
classifying high five patterns in our
scalagrams
but here is the interesting thing blobs
and color
and loops and lines and features like
that exist in pretty much
all images including our scalograms
it's only the last few layers in the
network that combine those features
and does the final classification that
is very specific to the type of images
you want to classify so we can think of
this first portion of a trained image
classification network as a general
feature recognizer and therefore we can
keep them
and then chop off the last few layers
responsible for the specific
classification
and replace those with new layers that
can output
the labels that we are looking to
classify
now theoretically training this network
should be much faster and require much
less training data since there is a lot
less that the network has to learn
it doesn't need to learn how to
recognize the primitive features
it only needs to learn how to combine
them to recognize the larger patterns
you're looking for
which is great because most image
networks require
millions of training images in weeks of
time using several high-end gpus to
train
and i don't have that kind of time or
the hardware or
the arm strength to create millions of
high-five training images
and then train a giant network so
this is what i'm going to do i'm going
to start with an existing network
and then modify the last few layers and
then
retrain it with a much smaller amount of
training data
so ready let's get to it
there are many different pre-trained
image classification networks that i can
start from
and they each have their own execution
speeds database sizes and prediction
accuracies
and i'm going to go with google net for
this project
but i assure you this choice didn't come
down to any logical reasoning
it's just because the matlab example
that i'm basing my project off of uses
google net and transfer learning to
recognize patterns in ecg
signals so to keep it in line with this
example
i'm going to do the same all right the
first thing i want to do is load
google net into matlab and just try to
give you a sense
of how massive this network is we can
open it up in the deep network designer
app to visually see all of the layers
and how they are connected
and check out this architecture it has
144 layers and there's a bunch of
different things going on in this
architecture
there's some convolutional layers at the
beginning followed by a series of
parallel
inception layers and there's a number of
these groupings just one
after another now luckily we don't
really need to concern ourselves with
most of these layers for our transfer
learning example
the important ones are at the end
this layer here is a fully connected
layer which means that
each input of which there are 1024
connect to each of the outputs of which
there are
a thousand so simplistically the way we
can think about this
is that there are 1024 different complex
features that this network has learned
and it uses combinations of those
features to classify
a thousand different objects so
whichever output value from this layer
is the largest
it assigns the label that is associated
with that particular neuron
and it finds the maximum values with
this probability layer
and then determines the label with this
output layer
you can see the labels here their tinch
and goldfish
and great white shark and 997 others
so for this project we only need to
replace two layers
the fully connected layer and the output
layer
and i can drag in a new fully connected
layer into our network
and then by clicking on it i can change
its parameters
i want to set the output size to be 2
which essentially means that we want
this layer to combine the 1024 input
features
to recognize just two main patterns and
since this layer hasn't been trained at
all
i'm going to increase the weight and
bias learning rate factor to five
this will allow this layer to make
larger changes with each
training cycle now the second layer we
need to replace is this final output
layer
so that it only has two labels high five
and
not high five and it's going to
determine what those labels are
automatically from the training data so
we don't have to specify them here
and that's it we're going to use the
rest of the network as
is now we're almost ready to start
training
but first we need to create our training
data
the ecg example used about 160 images
for training and validation so to stay
in line with that order of magnitude
i opted to record a hundred high-five
images
and a hundred non-high-five images i
created a script
that loops a hundred times recording one
and a half seconds of acceleration data
each loop
and then saving off the scalogram in a
data folder
and i got pretty good at timing my high
fives so that the event happened
near the middle of the window but i did
have to go through the full list of
images and prune out the ones that
didn't look so good
i didn't want those bad images to
corrupt the training
but overall you can clearly see that the
pink volcano
appears in all of these and about in the
center of the image
and i did the same thing for the other
label no high five
here i just recorded various things like
no motion or
slight random motion or other faster
motions that are similar to
but not quite a high five and that's
it with my training data created i can
start
the training let's go back to the deep
network designer app in matlab
under the data tab we can import all of
the training data that was just created
and all of my data exists within the
data folder
and the high five and no high five
subfolders
and the names of the subfolders is where
it's pulling the labels from
and if i wanted to i could also augment
all of this data by scaling translating
or rotating it
in order to cover all of the possible
solution conditions but i'm just going
to leave it as is
and lastly i'm going to have it randomly
select 20 percent of the data to be used
for validation
so this is a snapshot of my training
data
we have 80 examples of high fives and 80
examples of no high fives
and on the training tab i can now set my
training options
here i'm making a few adjustments to how
i want to train this network
but i'm just changing them to match the
training parameters that were in the
matlab example that i showed you earlier
and the link to that example is in the
description of this video if you want to
check it out
it goes over all of this stuff in really
good detail
all right so with the options set we can
start
training this network so let's kick it
off
and while we wait for this to train
let's talk about the steps we took to
get here
we took a network that was trained to
recognize objects like goldfish and
sharks
and repurposed it to find patterns in
three-dimensional acceleration data
and while i used it to create something
rather silly
this process can be used to find
patterns in any data
google net in particular requires a
color image as
input and so as long as you can format
that data into an image
you could use this transfer learning
process
you could use it for object detection
like pedestrians or street signs
you could use it for predictive
maintenance to find patterns in data
that indicate when a component will fail
you could use this to look for defects
in materials
you could use it to find patterns in
data from wearable electronics
like determining if a person is walking
or running or if their head was impacted
too hard
there's a million use cases for this
there's
also networks that are pre-trained to
classify sounds
like the vgg-ish network and we can use
transfer learning with it as well to get
it to classify
our own sounds in audio signals
and what's really interesting about this
whole thing is how quickly you can get a
trained network up and running
i started this project by following
along with an existing matlab example
but i essentially went from nothing to a
trained network
in about two hours and this was with a
single cpu
now your project might require more
training data than i used
and maybe a gpu for training in a
reasonable amount of time
but starting from an existing network
will almost certainly require less time
and data
than having to start from scratch and
starting from an existing matlab example
of which there are a lot you can jump
into these types of problems much
quicker and i left a link to this
example list in
the description
all right so we got ourselves a trained
network now and it took about four
minutes to train
also the network is 97 accurate
using at least the 20 of the data that
we set aside for validation
so it missed one of the 40 images
not too bad and now this brings us back
to the initial video i showed you of me
testing out the train network on the
acceleration data stream
each sample time i would update the
scalogram based on the latest
acceleration data and then feed that
image into the classify function
and have it return a label if the label
came back as a high five
i displayed that and incremented my high
five counter
and that's pretty much the whole thing
it turned out to be easier than i was
expecting and the whole high five
counter
is actually pretty satisfying to run so
i hope this has helped you understand
the benefits of transfer learning and
maybe got you thinking about your own
particular pattern recognition problems
and whether a technique like this could
help you with classification
this is where i'm going to leave this
video but in the next video i want to
talk about verification of these trains
networks
and how we can be confident that they're
going to work so if you don't want to
miss that or any other tech talk video
don't forget to subscribe to this
channel
also if you want to check out my channel
control system lectures i cover more
control theory topics there
as well thanks for watching and i'll see
you next time