in the last video we talked about how we
need
data to train a classification network
however in many cases data in its raw
form
the form in which it was collected might
not be ideal for training a network
there are some changes that we can make
to the data that are often desired
or sometimes necessary in order to make
training faster
simpler or to ensure that it converges
on a solution
in the first place and so in this video
that's what we're going to talk about
specifically i want to highlight a few
different ways to pre-process data
and talk about why it's so important to
the deep learning workflow
now like every video in this series this
isn't intended to cover everything you
need to know
but hopefully this will at least get you
thinking about your own particular
engineering problem
and the pre-processing that you may have
to do in order to successfully implement
deep learning so i hope you stick around
for it i'm brian
and welcome to a matlab tech talk
data pre-processing is a pretty broad
term
it's basically anything you do to the
raw data prior to inputting it into your
specific machine learning operations
and it's hugely important for at least
three reasons one pre-processing can
transform the data into a form that is
suitable for the network architecture
two it can help reduce the dimensions of
your data and make
patterns more obvious and three
it can adjust the training data to
ensure the entire solution space
is covered so let's walk through an
example of each of these
so that hopefully they make a little
more sense
the input into a network is fixed in
terms of the number of elements you feed
into it
this means that your data needs to be
formed into discrete packets that all
have the same number of elements
if you're working with images each image
needs to be the same size
which might mean that part of
pre-processing is to crop
pad out or resize images that don't have
the correct
dimensions and it's the same if you're
working with signals and not images
the length and sample rate of the
signals need to be consistent
or again cropping padding and resampling
is required all right so this is just
one quick example to highlight how the
network architecture
and the data need to be consistent with
each other but there's many
other things as well like making sure
the units are correct
but this sort of data reformatting is
one
reason to do pre-processing for the
second reason
dimensional reduction it's helpful to
remember that deep learning trains a
network
to recognize patterns in data and so any
information that isn't needed to
recognize the patterns you're looking
for
can be removed without impacting the
overall classification
and removing extraneous data helps make
the remaining pattern more
obvious and that's going to help the
learning process
in general if the patterns are more
obvious to a human they're going to be
more obvious to the deep learning
algorithms
as well but the other benefit of
reducing the dimensionality of your data
is because of the so-called
curse of dimensionality where more
dimensions means more features
and variations of each feature and
therefore more training data is needed
to cover all possible combinations
so not only is the data itself larger
with higher dimensions
but you need more of it to train the
network
so overall it takes more network
complexity more data storage and more
time to train
for example take these two images of the
number four scaled to two different
resolutions
we can tell that both of these are the
number four despite the one on the right
having far
fewer pixels or having a lower
dimensionality
and this is because the defining
features that make up the number four
still exist at both resolutions in this
way a network trained on the lower
resolution image could still function
as a number classifier just as well as
one trained on the higher resolution
image
now let's look at a slightly different
four again at both resolutions
you can see that the two instances
differ slightly in their details
which is picked up nicely at the higher
resolution but is mostly lost
in the lower resolution and this is
where the curse of dimensionality comes
into play
a network trained on the higher
resolution images may have converged on
a solution
that thinks that this little tail is a
defining feature of a four
and therefore misclassify this one since
it doesn't have that tail
but since that detail is missing in the
lower resolution the network almost has
no choice
but to converge on broad details that
truly define the number
and you might be thinking well why
wouldn't the higher resolution network
learn to recognize the broader features
also
well it can it just takes more training
data for the network to figure out that
these small details that we're feeding
into it
don't affect the larger classification
now a drawback to dimensionality
reduction is that you have to understand
your data well enough
that you can reduce the dimensions but
not so far
as to accidentally remove critical
information from
your data set for example say we want to
train a network that can visually
identify manufacturing defects in hex
nuts
it wouldn't be a good idea to reduce the
size of these images by just
scaling them down the flaws or the
patterns we're looking
for are quite small and we would lose
the detail that distinguishes them
in this case a better dimensional
reduction approach might be to crop
the images instead now there
are a lot of different ways to reduce
dimensionality like for example removing
noise or removing trends from the data
but however you do it the key takeaway
is that we want to remove any components
from our data set that we
know aren't important so that the
network doesn't have to learn
that it's not important all right the
last benefit of data pre-processing that
i want to talk about
is using it to cover a larger portion of
the solution space
a network can only learn from the
training data that you give it
and so if you want a network that can
recognize the number four in various
different handwriting styles
it makes sense that you need to have
several examples of different ways that
people write the number four
however writing style isn't the only
variable that can affect a good
classification sometimes the number
might be written
rotated in one way or another or maybe
it's a little bigger
or smaller and so the problem is that a
network trained on all of these
uniform fours would fail to classify
these rotated and scaled fours
since it wasn't trained on it now rather
than
collecting real labeled data for every
variation
we can simply duplicate the training
data and pre-process it
by rotating and scaling across the
entire solution space
or on the flip side we may take
unlabeled data
which can be scaled and rotated in any
way
and then pre-process it to put it into
an orientation and size that the network
has been trained for
so in these two examples we were able to
manipulate the data
in order to cover a larger solution
space
okay so that was a quick overview of
some of the benefits of
data pre-processing and the main
takeaway here is that you want to use
some of your expertise and knowledge
to change the raw data in a way that
makes learning possible
faster and more accurate and to give you
a sense of what this could look like in
one case
i want to talk about the pre-processing
that is done on the audio data within
the matlab example
speech command recognition using deep
learning
this example shows you how to train a
convolutional neural network to
recognize a given set of verbal commands
and you can go through the whole thing
yourself if you want to see all of the
details
but what i want to show you is how much
pre-processing is done
in order to prepare the audio signal for
training
the convolutional neural network is
looking for patterns in an image
so we have to transform this audio
signal into a 2d
image that contains recognizable
patterns that can be used to distinguish
between
each of the possible verbal commands
this particular image is called a
spectrogram and i think it's worth
explaining how this image is created in
some detail
because i think it nicely demonstrates
the types of things that you might
think about when you are pre-processing
your own data
all right what i have here is an audio
waveform of me saying the word
allow it's recorded at 44.1 kilohertz
and is about 0.8 seconds long
and the first thing we want to do is
make sure that this audio segment
is the same length as all of the other
audio that we may want to classify
with our network and since some words
may take longer than 0.8 seconds to say
i decided to extend this out to exactly
one second
and i did that by padding the beginning
and the end of the signal
with zeros and if we take the fft of
this signal we can get a sense of where
there is the most
frequency content since this was
recorded at 44.1 kilohertz
there is frequency information up to
about 22 kilohertz
but as you can see there isn't a whole
lot of information in the higher
frequencies
so i'm going to resample the audio
signal at 16 kilohertz
which will capture up to 8khz signals
and it won't cause any major loss of
quality
all right so this is the frequency
content for the entire signal
which isn't quite what we want as we
speak
the frequency content changes based on
the sounds and the syllables we're
saying
and we want to be able to pick out those
individual sounds and words so we need
to see how the frequency
content changes over time and we can do
that
with a short time fourier transform
we start by selecting a window size that
is smaller than the full signal
and then running an fft on just that
subset of data
i'm choosing a window that's a little
over 180 milliseconds
and you can see what that bit of audio
looks like here
now there is a problem with running an
fft on this audio segment
exactly as it is and that's that the fft
is expecting the signal to be repeating
and if i line a few of them up one after
another you'll see that there's this
discontinuity that we've created
this jump is going to artificially add a
bunch of high frequency content and make
our spectrogram much
noisier than it should be so to fix this
we
apply a window function there's a bunch
of different window functions but in
this example i'm using a
han function and the details don't
really matter because
the general idea behind all window
functions is mostly the same
they start and end at zero and have some
kind of scaling in between
and since the window function starts and
ends at zero when i multiply it with the
audio segment
it guarantees that the resulting signal
also starts and ends at zero
which means there won't be that
discontinuity when it repeats
now this scaling fixes the discontinuity
but you can see that we're losing some
information near the edges of the window
but as i'll show you shortly that's why
the windows overlap
okay so now we can take the fft of this
scaled signal which is this red line
and we can see that there isn't a whole
lot of content which we knew
because the word hasn't started yet but
if we scroll back up
we can move on to the next window which
i'm overlapping with the previous one by
50 percent
once again we take the time data and we
scale it with the han function
notice that with these overlapping
windows the part of the signal that was
lost in the first window
is present in the second so we are
capturing that information
all right we keep this up by hopping the
window across the entire signal
and then applying the window function
and then taking the fft
here i'm just showing the first four
windows so you get the idea
but if we go all the way across the
whole signal what we're left with is the
frequency content by
window but here once again we have more
information than we actually need
each fft produces a spectrum with
thousands of values
and we don't need that level of
granularity
so one common way to reduce the amount
of information here is by splitting up
the spectrum
into a number of bins and then scaling
and summing the frequencies
in each bin with a mel filter bank
which is a set of triangular bandpass
filters that are spaced
closer together at the lower frequencies
and then gradually get wider and further
apart
as frequency increases essentially this
is sort of to model the sensitivity of
the human ear
where we're more sensitive to lower
frequencies than higher
so we're just basically capturing finer
resolutions at the lower frequencies
also with the triangular filter the
frequency content in the middle of each
bin
is weighted more heavily in the
summation than the frequencies near the
edge of the bin
and to capture this information near the
edge once again we
overlap the bins so what we end up with
after all of this bending and filtering
and summing
is something that looks like this one
value per bin
that represents the frequency content
for that small bit of spectrum
and here i've colored these squares
based on the magnitude of the frequency
but they all look black since there
isn't that much information here
but if we apply this binning and scaling
to each of our windows
you can start to see that there is some
interesting content here
and that this frequency content changes
from one window
to the next and now that we have all of
this information
by bin and window the last thing to do
is to put it all
into a single image the first window is
placed on the left side of the image
with the lowest frequency bin on the
bottom and then we place the next window
beside
it and the next until we've gone across
the entire signal
and we've created a spectrogram
which is pretty cool right i think it's
kind of neat that we can create an
image of audio signals like this
okay so hopefully you can see when we go
back to the matlab example how this blue
image
represents the frequency content of this
audio signal over time
but more than that hopefully you can see
obvious
patterns in this spectrogram much more
so than you can
in the audio waveform and these patterns
are so unique in fact
that i bet you could even use them to
determine what word
is being said and to prove it check out
this
spectrogram of me saying allow aloe
alloy and ally and despite how close
these words are to each other
the patterns they make in the
spectrogram all have defining features
that make them different from each other
and if i say one of these words a second
time
you could probably determine what i just
said from the spectrogram
alone and this is exactly the uniqueness
and patterns that a deep learning
algorithm
and a convolutional neural network could
thrive on
all right that's where i'm going to
leave this for now if you want to learn
more i've left a bunch of links in the
description to different matlab tools
that help with
feature extraction and data
pre-processing and i've left several
examples that show these tools in action
on images audio and other signals
in the next video i want to talk about
how we can build off of an existing
pre-trained network with transfer
learning so if you don't want to miss
that or
any other tech talk video don't forget
to subscribe to this channel
also if you want to check out my channel
control system lectures i cover more
control theory topics there as well
thanks for watching
and i'll see you next time