JASON MAYES: So you just built your first deep neural
network that was able to classify images.
That's a huge step in the right direction,
but let's see where there's still
room for some improvement.
Well, first off, you may not be always dealing with a 28
by 28-pixel, grayscale, square image.
Here you can see a real photo taken of a person also
wearing a t-shirt, which is probably
more representative of a real-world situation
than the image on the left in which you
want to recognize the presence of a t-shirt on a real person.
Next up, you can clearly see the lady in this image
is not in the center of the image.
Of course, you've seen in earlier chapters
how you could use existing models to find the human
and crop them out, but what if it
wasn't a human you were looking for
and a model did not already exist to find that thing?
Maybe it's a car, a handbag, or a plate of food,
and they're not in the center in the shot.
Now, from your understanding of how a multilayer perceptron
samples all the pixels in the input image
and adjusts its weights accordingly when training,
you can probably appreciate that it
will associate certain features with specific locations
in the image.
This is fine when all the images you are classifying
are resized, cropped, and centered,
but this is not typically how data will be presented.
Take an example from the MNIST data set on the left.
It's quite clear to you and I that the image on the right
is exactly the same 3, just a bit smaller
and in the top right of the image instead of the center.
However, to your multilayer perceptron,
this is something it's never seen before,
and it will most likely classify this incorrectly
if it's to try and run this through the network.
Again, here the image is rotated ever so slightly,
and to us it's clear this is a number 3 at an angle.
However, your current code from the previous section
in some cases, at least, would still misclassify this, too,
as there's no training data like this
in the data set that it saw.
So how can you create a machine learning
model that can work with larger images
efficiently whereby the objects might not always
be in exactly the same location or even slightly rotated
or scaled?
Well, thankfully new research has come out
that is very well suited for this task known
as Convolutional Neural Networks.
We see some special new layers that you've not seen before
but have been fed into what is essentially
a multilayer perceptron at the very end
to perform the final classification,
so it actually builds upon the knowledge you already know.
For now, however, let's just treat the CNN
as this magic gray box until we dive deeper to break it apart.
At its essence, a CNN is essentially
learning patterns of patterns.
The first layers in these machine learning networks
look for simple features in the image
presented-- maybe things like lines
at certain orientations-- which deeper layers can then
use to find shapes, and those shapes can then
be used to find objects.
Essentially, the deeper you go into the network,
the more complex of representations it can find.
Here you can see an attempt from Google AI Research
for visualizing the different layers of a very
complex and deep CNN on this slide,
but you can see how each layer recognizes high-level features
for deeper into the network you go from left to right
in this case.
Now, this is all very exciting, but first,
let's introduce some new concepts
so you can break down this gray box that
currently seems rather magical.
OK, to explain these concepts, I'll be using a 7
by 7-pixel image like the one shown here as input data
to keep things simple to understand.
But remember, the principles could apply to larger images--
say 256 by 256 and so on.
Now, remember, each pixel is just a number.
Let's say white is 1 and black is 0.
In this case, the numbers on the slide
are essentially what the computer sees.
If you overlay these two images that I
made to see what crossover there actually
is for the white pixels, you can see there is none.
As far as the computer is concerned,
these are entirely different images.
CNNs, however, try to match pieces of the image instead
of the whole image itself.
Here you can see some common features in the digits drawn.
In these highlighted 2 by 2 orange squares,
you can actually see the content is exactly the same.
And what about these green squares?
They also match.
And finally, how about these pink squares?
Maybe if an image has three features
in a similar relative positioning that
are not too far away from each other, maybe
you could call that a 7.
So now the question is how do you find these features
that matter?
Let's talk about filters.
So Convolutional Neural Networks will use filters--
or some ML folk prefer the word kernels--
to extract the most important features
from the image training data.
You can think of these as tiny images--
maybe 3 by 3 pixels in size--
like the ones shown here.
However, these are not actually images
as they can also have values like minus 1,
for example, as shown.
Filter 1 here is interested in parts of chunky diagonal lines,
and filter 2 is interested in parts
of lines that are horizontal.
You can see how they essentially overlap
with the different parts of the image that
match the ones in the filter.
And just like the weights in the neural networks
that you already learned about, the values
of the numbers in these filters are just random to start
but will be adapted over time through training
to change their values to find filters that
can find the best features that describe
the objects in the image.
Also, as an aside, if any of you are
familiar with traditional image processing,
you may have heard of a filter called the Sobel edge filter
detection, and this is basically the same sort
of filtering going on here.
Except in this case, the computer
learns for values to use rather than a human defining
the pattern in advance.
Now that you've got some filters, you can take each one
and overlay it with every pixel and its surrounding
pixels in the given image one by one that
matches the size of the filter.
In this illustration, you're overlaying
it to be centered on the highlighted green pixel
as shown in the right image.
Next, you multiply each of these nine pixels, in this case,
by the corresponding value in the filter
in the same position.
In this case, taking the top-left pixel
as an example that has the value of 0,
the same position in the filter has a value of 1.
So 1 times 0 is 0, which gives you your top-left result
value as you see on the right.
You then repeat this for all the other values in the filter
to get the overall result that's shown on the right-hand side.
Next, you can sum up the results numbers, which in this case
would be minus 2, as minus 1 plus minus 1 equals minus 2,
and all the other values are 0.
Seems like this filter does not approve
of this patch of the image, which makes sense as it's not
over a diagonal line.
So skipping ahead to a different path of image,
let's repeat the maths again.
In this case, the sum of the resulting numbers
would be 3, as there are three 1s, and the rest are 0.
Looks like you got a much better match than before.
Now, you could do this for every pixel in the image.
However, you may struggle with the ones at the very edge,
as filter on the edge pixels would not
have any pixel values to multiply with outside
of the edge of the image.
So what do you do in those cases?
Well, when dealing with CNNs, there's
a term called padding that allows
you to specify what numbers to fill
these out-of-bounds imaginary pixels with for the purpose
of convolution.
Here the padding is set to 0, so it just
fills in the space with this value.
Alternatively, you could only sample pixels
from inside the image that have enough data around them.
The choice is up to you.
So then after you've applied the filter
to every pixel in the image, the numbers on the left
would be the outputs.
Here I've highlighted in green numbers that are above 0
with a darker green showing higher numbers,
and in a similar fashion, negative numbers
are in red with darker red being more negative.
You can see that your diagonal filter has led to a new matrix
where the edges of the diagonal lines
are quite clearly highlighted by the darker green regions.
And in a similar fashion, if you perform
a convolution using the other filter you just defined--
the one looking for horizontal lines--
you can clearly see how it's able to pick out
the top of the 7 in this case instead.
Now, that's pretty powerful stuff.
Let's see how the same convolutions
deal with the smaller 7s that you also had.
Using the first filter on this new image,
you can see it's still able to find
the diagonal on the smaller 7 in a different position.
Good so far.
And once again, for the horizontal line filter,
it managed to find this part of the smaller 7 too.
OK, seems like you now have a chance of finding features
no matter where they are in the image, so what's next?
Well, the next concept to understand
is something known as stride.
This just means how many pixels you'll move by when
performing the convolution.
Let's say we start by analyzing the current green pixels shown
on the right.
With a stride of one, you would now
move one pixel to the right, and so
on for the rest of the image, adding padding
where it's needed to calculate the edge pixels,
and then move to the next row and continue.
And in a similar fashion, a stride of two
would just move the field two spaces at a time instead.
Starting as before on row 2, you now
move two pixels to the right, and now you have two rows down.
And notice how with stride of two,
it means you skip a row of pixels as the stride
acts in both axes.
However, as this is a 3 by 3 filter,
it still samples numbers from this row as shown.
Great.
So at this point, the only thing left to decide
is how many filters do you want in your system?
And what size will they be?
These are some new hyperparameters for you
to decide what to set when you start coding later,
just like you had to pick the number of neurons
in prior coding.
So to bring this all together then,
a convolutional layer in TensorFlow.js
is just a stack of trainable filters
that you're applying to some image-like data
whereby the relative positions of that data
actually matter in order to classify something.
Also note I've used 2D data here to keep things
easy to understand, but remember,
three-dimensional data might exist,
so you can also have 3D convolutions that
act in the same way that sample data from more dimensions too.
And it turns out that if you have many convolutional layers,
each layer can learn more and more complex
features from the ones found in the lab before it,
essentially learning patterns of patterns.
But if you have a large image, that's
a huge amount of processing to do between every layer.
Well, there's one more layer type
you can use to reduce the image size after a convolution has
been performed while retaining the key features as
found known as max pooling.
So in a similar way to convolution,
max pooling will work its way through the convolution output
data with a certain size and stride.
Here you see that with a size of 2 by 2, a stride of two,
and using padding, max pooling would
sample 16 positions in this convolution's result. Now,
all the max pooling is going to do is sample the numbers
and find the maximum value from those numbers and record it.
So the output of a max pooling layer
would be smaller than the original input
while still retaining the important information that
matters.
You can see here how for a given sampling of numbers,
it lets the highest number only pass through
in the final result on the right.
Now, if I remove all the colored squares so it's easier
to see the original data, you can
see that the max pool kept all the important information
that the filter found at the horizontal line data
towards the top of the image.
And on that note, even though it's smaller,
the important piece of that filter came through no matter
where they were in the image, and you can now
work with a layer that's much smaller in size,
greatly reducing the complexity of the model versus if you
didn't do this, while still having a chance of finding
the object even if its position has shifted from where you saw
it in the original training data.
And of course, if you have many filters,
the max pooling would apply itself
to the output of each convolutional filter's result
so you would have a stack of max pooling results, too, as shown,
with each one suitable at recognizing
a different interesting feature.
And these steps of convolution and max pooling
can be chained together to be repeated several times,
creating a deeper network capable of learning
more complex features.
So to bring all of your knowledge together then,
let's build a full diagram of a Convolutional Neural Network
with some realistic choices for the number
of elements in each section.
Here you've got a 28 by 28-pixel input image
like you've seen before.
This has been followed by a convolutional layer containing
16 convolutions that use filters that are 5 by 5 pixels in size
and produce a 28 by 28-pixel shape output
as padding is used.
This is then fed into a max pooling
layer that uses a 2 by 2 max pooling size
with a stride of two, which produces 16 feature maps that
are now 14 by 14 in size instead.
Essentially, it's made the convolutional results
a quarter of the original size while retaining
the key information from the 16 convolutions.
You can then feed the results of the max pool
into a second convolutional layer, this time using 32 5
by 5 filters, producing a 14 by 14 output
as it's sampling the max pool outputs that are already 14
by 14 in size.
This convolutional layer's results
are then fed into a second max pooling layer, again with a 2
by 2 size max pool with stride of two, producing 32 feature
maps that are now just 7 by 7 pixels in size.
And then finally, these 32 feature maps
from the second max pool can be flattened
into one huge array of numbers, which are then
fed into a regular multilayer perceptron
that you already used before.
In this case, this perceptron has 128 neurons
in the first layer and then 10 neurons in the output layer,
representing each of the 10 digits
it can classify just like you learned
in the previous sections.
And as before, the highest number on the output
represents the prediction of what it thinks
it saw in the input image.
In this example, it shows a 7 has been classified.
Now, so far you've been learning about using CNNs
with the context of image data.
However, CNNs can be used for any data
but is image-like in nature, meaning
that the spatial position of the data
is important in order to classify
the contents within it.
For example, a database table of customer addresses
is not suitable for a CNN because if you
changed the order of the rows of data,
they would still make sense.
In contrast, if you change the rows of an image of a dog,
you'd no longer have an image of a dog.
The order of the pixels matters for image data.
Also remember that data could be 2D-like images or 3D
for other types of sensor.
And one good example of other data
that could be represented as an image is sound.
It turns out you can draw recorded audio
as a spectrogram like the one shown here, which
is an image showing the frequencies of sound heard
over time.
Well, guess what?
Your CNN will work just fine on these, too,
and now you've got the ability to classify short audio clips
as well.
Pretty cool.
OK, that was a lot to digest, so in the next section,
you'll put all of this into action
and code a CNN that can recognize images more robustly.
See you there.
[MUSIC PLAYING]