in the last video i showed how i built a
high five counter by retraining an
existing deep neural network
i went from basically nothing to
something that runs in about two hours
which is pretty awesome
but simply getting a model that can
classify your data
isn't the end of the deep learning
workflow some other things that we need
to consider
are that the trained network is usually
part of a larger system
and we want to be able to incorporate it
into the complete design
we also want to have some amount of
confidence that the model will work
on unseen data and that it's going to
interact as expected with the other
system components
and ultimately we want to deploy it onto
a target device which requires certain
performance characteristics
and each of these require additional
work and considerations
beyond what i did in the last video and
so that's what we're going to talk about
now
i hope you stick around for it i'm brian
and welcome to a matlab tech talk
we can sum up this video with a question
i have a trained network what's next
and before we get into it i want to just
reiterate my usual caveat that this
video isn't intended to cover everything
i just want to introduce some concepts
and get you thinking about the overall
problem
and i've left links in the description
to resources that go into far more depth
on all of this alright let's get to it
at the end of the last video we were
left with a google net model
that was retrained to recognize two
patterns it's looking for these pink
volcanoes and classifying them
as high fives and then anything else we
send through the network gets classified
as no high fives
so what's next well i think it's
reasonable to want to know
whether the network works or not and we
know that it works in some fashion
because during the training process it
was using 40 validation images to assess
the model accuracy
and it only mislabeled one out of the
40. and this is great for sure
but there's two things that i want to
point out the first is that even though
we have some confidence that this model
works
for the 40 validation images we don't
necessarily
know that it works on images that
haven't been seen yet
now if those 40 validation images does a
good job of covering the entire solution
space
then we could be confident that the
network will only come across situations
that are similar to one of these images
however in this case 40 images was
enough to gain confidence for the video
i was making
but i don't think it was enough to cover
all of the possible arm motions
that the network might see in real life
and so we're going to want to do some
additional testing
if we're going to deploy this network
into the field
the second thing i want to point out
with this model is that we validated it
with
scalogram images that are uncorrelated
to each other
and what i mean is that each one comes
from its own standalone arm motion
but that's not quite the case with how
i'm going to use this network
remember i'm looking to pass in
essentially a continuous stream of
scalograms
and i want to count the number of high
fives in my data
that means that as the data streaks
across the window
i'm going to pass in multiple frames
with the exact same high five in it
and my network is going to return a high
five label several
times in a row and beyond that due to
the errors in this network
that was the one out of 40 that it
mislabels some of these frames are going
to be mislabeled
and both of these things will cause my
counter to be off if i just
assume each scalogram that is labeled a
high five is
actually its own unique motion
so the neural network is just part of
the overall logic that i need for my
counter
and this is almost certainly the case
for all deep neural network solutions
you're going to develop a model that can
classify data
and you know you obviously want to know
that that works but beyond that
there's other logic that makes that
network an actual
product and it's important that the
system as a whole
works as well now my project was rather
simple and the stakes were pretty low if
it didn't function correctly
but let me walk through the way i
approached building out the rest of the
hi5 counting logic
i built a matlab script that would read
in the acceleration data
and then keep the most current 1.4
seconds of data in a buffer
and then every third sample time i'd
pre-process the buffered data
into a scalagram and then make sure that
there were no pixel values greater than
one
and then run it through the trained
network with the classify function
then i only counted a new high five if
it had been more than 1.4 seconds
since the last high five this is how i
ensured that i only counted each high
five a single time since it would have
completely left the buffer
before i look for a new one and i did
this in matlab because i only needed a
few lines of code for this project
but this whole thing could have been
developed in simulink as well
here is a simulink version of my high
five counter
well it's almost exactly the same the
difference is that rather than read the
acceleration
live i've opted to just read in a five
second
acceleration profile that i saved off
earlier the idea being that i could
collect
several of these five second snippets
where i know exactly how many high fives
are in that data
and then use them in regression tests
whenever i update the network
or the other logic and other than that
the rest of this logic is doing the
exact same thing
i'm updating the acceleration buffer
pre-processing it into a scalogram
classifying it with my trained network
and then counting the number of high
fives
so let's open up the scope and run it so
you can see it in action
the top graph is showing the red in
acceleration profile which has a single
high five
right here at the beginning the counter
only registers a single high five as
expected
even though you can see that several
scalar grams were labeled as high fives
as that pink volcano was streaking
across the image
and the other thing to note here is that
there's a delay between the actual high
five and when it was counted
and this is due to the fact that my
network was trained to recognize the
high five
when it was in the middle of the image
or a few tenths of a second
after it occurred all right so now
what's nice about simulink
is that i can package this logic into
its own subsystem
and i can use it as part of an even
larger system
like for example this model shows the
implementation of a multi-loop pi
controller
acting on a robotic arm and maybe if i
was so inclined i could use my new
subsystem
to determine if this robotic arm ever
made a motion
that looked like a high five and once i
was happy with
this implementation i could use simulink
coder to build embedded c code
and deploy all of this logic including
my deep neural network
to the arm itself now i'm not actually
going to build this out any further
because i'm just making a video to
demonstrate a few things
but if i did want to keep going on this
project my next steps would be to
systematically try
all of the different arm motions i could
think of both high fives and non-high
five motions
and i'd probably even have other people
try it out from different cultures that
might
high-five differently in every instance
where the user motion was misclassified
by the system
i would save off that data and add it to
my training data set to retrain
and refine the network now in this way
i'm never guaranteed
to have a perfectly functioning network
but i am increasing the solution space
over which i'm confident
that it will perform and this is a
standard approach for deep neural
networks
we don't yet have a good systematic way
to verify them
so we rely on these sampling methods
like monte carlo approaches
to gain confidence in the network over
the entire solution space
and this is more than likely going to be
the case for your project as well
whether you're looking for material
defects or picking out verbal commands
in audio
or classifying rf modulation schemes
you're going to integrate the trained
neural network into your full system
and test it in a variety of situations
but as you can see no matter how many
different tests i run
there will always be sections of the
solution space that haven't been tested
and so this is where synthesized data
can be so powerful
remember in the second video where we
synthesized rf data
and the idea there was that we would use
it for training a network
well now we can use it to generate
millions
of different test cases and produce a
really dense
sampling of the solution space and that
would just give us a huge amount of
confidence in the system
well you know as long as the synthesized
data actually reflected
reality now i can't really do that with
my high five project because i don't
have a good understanding of the
acceleration patterns that are possible
with high fives and therefore can't
easily
synthesize it for my project it was
easier to just
physically test it rather than
synthesize the test data
but regardless of whether you can
synthesize test data or not
you're going to want to ultimately test
the system in the real world
on the real hardware which brings me to
the other thing i want to talk about
deploying your network and other code to
a target computer
deployment is important because it
doesn't matter if your code works
if it doesn't work on the hardware it's
supposed to run on
and there's a bunch of good information
that i link to in the description on
generating optimized code
for deep learning networks directly from
matlab and simulink
for embedded gpus cpus and
fpgas so i'm not going to go through
that here
instead i want to take the last few
minutes to talk specifically about the
size and speed
of the trained network for a high five
counter that's going to be deployed to
an embedded cpu on a watch
it's important that the size of the
network is taken into account
i started from google net which if i
open up the deep network designer app
i can quickly get a sense of the size of
this network
this neural network has 7 million
parameters
which is pretty massive for a simple
high-five counting program
now if memory space or execution speed
is a concern
then i'm going to have to find a way to
make this smaller and there are
a few options for one i could just start
from a smaller pre-trained network
something like squeeze net which only
has just over a million parameters
and then again use transfer learning to
retrain it for high fives
the idea would be that maybe i don't
need the accuracy
or feature details that the larger
google net provides
and here i did the exact same transfer
learning steps that we took in the last
video
and i retrained squeeze net and at least
for these particular parameters this
network is
90 accurate so a little less accurate
than google net
now i can export the retrain squeeze net
to the workspace
and run it in my matlab or simulink
simulation
but check this out we can see that this
million parameter network is still about
three megabytes and if this size is
still a problem
instead of finding an even smaller
pre-trained network i could also try
reducing the size of my network
by pruning it or quantizing it
pruning is removing some of the
parameters in the network that don't
contribute much
to classifying your particular data and
quantizing
means taking the single or double
precision weights and biases in your
network
and quantizing them to 8-bit scaled
integer data types
the idea is that we can still get the
same performance from the network
without having to use high precision
data types
just to give you a sense of what
reducing the network could look like in
one instance
let me show you the result of quantizing
my trained
network here i'm using the deep network
quantizer app
to pull in my trained model and then
quantize it to 8-bit scaled integers
now this took a few minutes to run which
i'm skipping over
but the important metric here is that
the network was compressed by 75
and it had no measurable impact on its
accuracy
so it has the same number of parameters
but it's one quarter the memory size
and in addition to this i could also try
pruning the network to reduce the number
of parameters
in a way that also doesn't impact
accuracy but i'm going to leave this
model as
is for now and hopefully you can start
to see that with a pre-trained network
and transfer learning and pruning and
quantizing you might be able to get to a
model that is sufficient size
and efficiency for your application but
if you can't then the last option is to
build your own network architecture
from scratch this option requires the
most
training data and the most training time
since at the beginning the network has
no concept of
anything so it has to learn everything
and the other downside of starting from
scratch is that it takes a good
understanding of different network
architectures to create an
efficient one but this is where
trade-offs can occur
how much specialized development do you
need to do from scratch
versus how efficient and fast do you
need your architecture
to be my high five counter might benefit
from a smaller and specialized
architecture
since it has to be deployed to an
embedded processor on a watch
whereas a system that is looking for
material defects
could run on a dedicated desktop
computer with a gpu
and it may not even need to run real
time if getting instantaneous results
back
is not necessary so i guess the one
thing i want you to take away from this
is that there are things you need to
consider with deep learning
some of those things are how deep does
the network need to be in order to be
able to find and classify the patterns
in your data
and can you get by with a pre-trained
network and transfer learning
and how are you going to access labeled
training data
and how will you ensure that the data
covers the entire solution space
in general you're going to need more
training data to train larger networks
and to train networks from scratch and
then
how are you going to gain confidence in
your network and in the system as a
whole
and can you synthesize test data and can
you run simulations
or does all testing need to be done in
the field
and there's no one answer for every
project but
hopefully you can start to see the
benefits and possibilities
for deep learning perhaps you have an
engineering problem that you're working
right now
where the solution comes down to being
able to detect and label complex
patterns in data
if so deep learning is an approach you
might want to consider
as part of your trade studies it might
be easier than you think
all right that's where i'm going to
leave this video for now don't forget to
check out the resources that i left in
the description of this video
there's a lot of good stuff there and if
you don't want to miss any other tech
talk video don't forget to subscribe to
this channel
also if you want to check out my channel
control system lectures i cover more
control theory topics there as well
thanks for watching and i'll see you
next time