LAURENCE MORONEY: Welcome back to Machine Learning
Foundations.
In the last episode, you got an introduction
to machine learning, and you saw how
it works from a programmer's perspective
by having you create answers and data,
and letting a computer infer the rules that determine them.
This is fundamentally different from traditional programming,
where you have to figure out the rules, express them in code,
and then have them act on data to give answers.
You created your first neural network, a single layer
with a single neuron, to figure out the relationship
between two numbers.
And then you had an exercise to see
if you could extend that to figure out house
prices in a simple scenario.
Before we get to this lesson, let's take a look
at the exercise answer.
You can find the notebook containing the answer
at this URL, and I'll step through it next.
So let's start by looking at the question.
The idea is for you to build a neural network that
will be trained to be able to roughly predict
the price of a house if we're saying the cost of a house
is a $50K flat fee plus $50K per bedroom.
So a one-bedroom house is $100,000,
two-bedroom house is $150,000, et cetera.
So here's the question.
So for example, we want to build a neural network that
can roughly predict the price of a house, where
the price is a simple formula of $50,000
plus $50,000 per bedroom.
So a one-bedroom house is $100,000,
a two-bedroom house is $150,000, et cetera.
Now, how would we create a neural network
that learns that relationship?
Now, one way that would make it work better
is if you scale the house price down.
So instead of you trying to figure out that a one-bedroom
house would cost $100,000, maybe you could predict that
a one-bedroom house would cost $1.00.
And then you say that's one $100,000.
So for example, a seven-bedroom house should cost $400,000,
but maybe you can predict a $4.00 off the seven instead
of $400,000 of of seven.
So here's the starter code that you were given.
And let's see what the answer for this
was going to look like.
So here's the answer.
So you're going to define your model in the same way
as we did in the lesson--
a single dense layer that has a single unit with an input
shape of 1.
You can compile that with stochastic gradient descent
and mean squared error.
And now we can do an array of just small float values.
Like a one-bedroom house is $100,00,
a two-bedroom house is $150,000, a three-bedroom house
is $200,000, and so on.
Then we can fit the x's to the y's.
We'll do it over 1,000 epochs.
And we can see what we get for seven.
So I'll run this now.
So after 1,000 epochs, we're at 1.2 time 10 to the minus 6
on our loss.
And our answer is very, very close to 4.
It's 4.001622.
So we've trained the neural network
now to be able to predict that house price.
I know it's a very simple formula.
House prices aren't really that simple.
But hopefully this was able to show you
how this type of prediction can be done on just
a very small amount of data.
Well, that was a very simple scenario,
matching x's to y's, or getting a house price
from an oversimplified formula.
The key thing here is that the pattern
of code that you used, creating a neural network,
compiling it with a loss function and an optimizer,
and then fitting the data to the labels
is the same pattern that you'll use in far more
sophisticated scenarios.
In this video, we'll start by looking at computer vision
and how a computer can learn to see.
We'll start simple, and we'll build from there.
So in understanding vision, ultimately it's
all seeing and recognizing things in a field of view.
So for example, consider the items of clothing here.
If I were to ask you which ones were shoes,
you'd probably know right away that it's
the high heel and the sneaker.
But how did you know that?
How could you teach someone who has never
seen a shoe before what a shoe is so that they could then
tell the difference?
What are the rules that make a shoe a shoe?
Looking back at the data, how would you
come up with the rules to determine a shoe?
If it's red, it's a shoe, maybe.
Well, that works for this data, but we all
know that that won't work for all shoes.
Computer vision is the field of taking pixels and recognizing
what's in them.
And the same kind of pattern matching we've been discussing
can be used here.
But in order to train a neural network
to recognize the contents of images, we need data
and we need labeled images.
Fortunately, there's an easily available data set
called Fashion MNIST.
This has 70,000 images sorted and labeled in 10 categories.
The images are small.
They're 28 by 28.
But despite this, they're still recognizable.
You can tell that this image is a boot.
And because they're small, we can train a neural network
quickly using them.
We'll start by loading the data.
In Keras, there's some built-in data sets
that can be loaded with a single line of code.
And here it is.
Python can return multiple values from a single call,
and four values are returned here.
First is a set of training images and training labels,
and then is a set of test images and test labels.
You might wonder why our images have been split into two sets.
There's one for testing and this one for training.
The idea here is really simple.
We'll take a subset of our images--
in this case, 60,000 of the 70,000--
that we'll train the neural network with.
We'll then use the other 10,000 to see
how well it performs on images it has never seen before.
Think about how you would teach someone
who has never seen a shoe before what a shoe looks like.
You'd probably get a bunch of shoes and show them.
But then there's no point in testing that person
on the shoes that you've just shown them,
because in that case, you've already
told them that those are shoes.
Instead, you'll get a bunch of stuff
that they haven't seen before, and have them pick out
which ones are shoes.
It's the same principle here, and it's one
that you'll commonly see in building neural networks
with machine learning.
So, back to our data--
we have four lists, one for training images,
one for training labels, one for test images,
and one for test labels.
Our images of the 28-by-28 pixel arrays for the clothing items
like the boot here.
And the labels are numbers from 0
to 9, indicating the appropriate item of clothing.
Now, think about this for a second.
Why do you think that it would be a number, such as 9, and not
a piece of text, like boot?
Well, there's two main answers to this.
The first is that computers deal better with numbers,
and we train our neural nets using numbers.
The second, a little more subtle, but very important,
is that, by using a string here that said boot,
we'd be introducing a bias that the neural network will
need to understand the English word for the clothing item.
But we don't want that bias.
So we use the number 9 instead of the text
for ankle boot, which I can then present here
in a few different languages.
Can you guess what each one is?
I'll leave that up to you.
So here's what our neural network
definition will look like.
In the last video, you saw a neural network
with a single layer containing a single neuron.
Now we have one with three layers.
The first layer is called a flatten,
and this is a special type of layer in Keras.
There's no neurons here.
But the idea of the function is to take the rectangular shape
of the data, the 28 by 28, and flatten that
into a one-dimensional array that can
be processed by the next layer.
Hence the name, flatten.
This is followed by a dense layer.
Now, note that there are 128 neurons in this layer.
It's significantly larger than the one-neuron single layer
that we used last time.
This is followed in turn by another dense layer, this time
one with 10 neurons in it.
There's a few important details here
that you need to consider and need to remember
when designing neural networks.
To understand what the network is for,
it's always good to look at the top layer and the bottom layer
for their shaping.
In this case, the top layer takes 28
by 28 for its input, which is the size
of the images in our data set.
Remember that, when building neural networks,
all data being fed into training has to be the same size.
In this case, it's easy because the data was already
in 28 by 28 for us.
But later, you'll see that you often
need to pre-process your data to get it ready for training.
On the output side, we have 10 neurons.
Now, remember that there were 10 different types
of clothing in the data set.
This is not a coincidence.
The job of each of these neurons will
be to calculate the probability that a piece of clothing
is for that particular class.
You're probably also wondering what these things are.
Well, they're another place where neural networks
do a lot of math.
But fortunately, most of it is encapsulated in parameters
for you by TensorFlow.
Layers in TensorFlow can have an activation function
run on them.
Now, this is code that executes while the network is learning.
They can be really useful, and can save you
a lot of time writing code for yourself.
The relu function here is pretty simple.
It basically says that if the output of a neuron
is less than 0, set it to 0.
The reason for this is that negative outputs
could skew results downstream, canceling out
positive outputs elsewhere.
So we'll just throw them away.
Softmax, commonly seen on the final layer
if there are multiple categories,
simply helps you find the most likely candidate.
Remember earlier that I said that each
of the neurons in the bottom layer
will be a probability that the item of clothing
matches that class.
That could give you something like this.
And instead of you writing code to look through each value
and find the largest, softmax will set the largest
to 1 and the rest to 0.
So now, to get the correct class,
you just need to find the 1.
To illustrate the learning a bit, think about all
of the pixel values, which are numbers from 0 to 255,
being fed into a neuron called n0.
And its parameters, n0 and c0, are guessed.
The same happens for every neuron in the network.
Each neuron is initialized with random parameters, m and c.
And when the pixels of each image in the training set--
and that's all 60,000 of them--
are fed in one by one, when all the totals from all the neurons
are added up, we'll get an answer.
The loss function then calculates how good
or how bad that answer is, and the optimizer then
tweaks the m's and the c's of the neurons
to give it another try.
Over time, the values within the neurons
will change so that the answers for the training data
are fit as accurately as possible to the labels
in the training data.
This is the same learning that we
saw in the single-neuron example in the last video.
For more detail on how this works under the hood,
check out this excellent video from Andrew Wang
at deeplearning.ai.
So on the loss function and optimizer,
we specify them when we compile the model.
This time we'll use an optimizer called
adam, which is a particularly good one for a task like this,
and a loss function called sparse_categorical_crossentropy.
Note that when you're classifying multiple
categories-- and we have 10 here--
you're going to need something categorical.
Don't worry if you don't understand what this
is what the options are yet.
Just go with it.
And over time, you'll learn good optimizers and loss functions
for specific scenarios.
Now, next up, we just want to fit the training data
to its labels.
And we'll run it for just five epochs
this time, because there's a lot of data there.
Last time when we did the y equals 2x minus 1,
we then used model.predict to test our network.
Now, that worked for a single value.
You can also use model.evaluate to pass it
in to a list of test data.
It takes the test images, gets its prediction,
compares that to the answer in the test labels,
and then calculates how many it got right
and how many it got wrong, returning you an answer.
It's a nice one line of code to help you test.
And if you want to see the predictions for yourself,
you can of course do a model.predict.
If you pass it in multiple images,
it will return a list of all of the predictions
for the respective images.
OK, so that's a run through your first computer vision neural
network.
Let's now see that code in action in a lab.
And after that you'll be ready to try the next exercise.
The lab is at this URL.
And I'll run through it now.
OK, now let's take a look at this Fashion MNIST example.
So here's the Colab that you'd be working from.
The URL was in the video.
And I'm just going to step through it.
So first we can start by importing TensorFlow.
And we'll get this warning that the notebook was not
authored by Google.
That's OK, you can run it anyway.
I've also asked it just to print the TensorFlow version
so we see what version we're working with.
Now, at the time I created this, TensorFlow
was at the release candidate 2 for TensorFlow 2.2.
If it takes a little while to run that,
just keep an eye on this area in the Colab.
And that's where it's assigning you a VM.
And that will give you status of the VM itself.
Another thing to do when you're training
is to take a look at runtime, and look at the runtime type,
and look at the hardware accelerator that you're using.
If your training is going to be really, really slow,
this is probably set to none.
You can actually improve your training
by setting that to GPU.
In this case, this isn't a particularly sophisticated data
set.
So I'm just going to keep it at the CPU that it's running on.
OK, first of all, I'm now going to load the data.
I'm just going to specify I want MNIST.
And I'm going to use the tf.keras.datasets.fashion_mnist.
I'm going to get my training images
and training labels, my testing images and testing labels,
just download them.
It's pretty quick.
If we want to take a look at a few of them,
this code will help you to do that.
First of all, I'm going to take a look at the training labels,
and then the training images, and print them out so we
can see what that looks like.
And if you look kind of closely here,
you'll see, even though it's cropped,
that it is actually the boot.
Now, these images are grayscale.
So they're stored as values from 0 through 255.
So for example, the top line of the image here is 28 zeros.
So that's all going to be black.
This number 9 is the label, because that's
what we're printing out first.
The second line is all black, the third line
is all black, et cetera.
And when I scroll down and actually look at the image,
we'll see the image was there.
And it's been colorized.
But you can see the dark image is
there, the dark at the bottom, or all these zeros.
And then all of these numbers are the numbers representing
the pixel values here.
It is a grayscale, but when I'm printing it out here,
I've colorized it.
So one thing that you want to do with your data
is to normalize it.
Neural networks work best if they're dealing
with values between 0 and 1.
And as I mentioned, the values and the pixels in this image
are from 0 to 255.
So a simple way to normalize that is just divide by 255.
And with Python, if you have an array
as we have here, or a list, if you just
divide the list by 255, it's smart enough
to divide all the individual values.
You don't have to loop through it and do it yourself.
Now we have our neural network is defined.
We're flattening, and we have our two denses, 128 and 10.
So I run that.
And now I will compile my model with my loss
function and my optimizers, and then I'll train it.
And I'm fitting the training images to the training labels.
And I'm just going to do it for epochs.
So it's pretty fast.
We can see, even on a CPU, it's only
taking about three seconds per epoch.
And over here is our accuracy.
So the first time around, it was about 82%
accurate, which is pretty good.
Because if you think about it, it's a 1
in 10 chance of getting it right.
You would expect it to be 10% accurate.
So it started off pretty good.
The optimizer did a good job of making the first guess.
But then by measuring the loss and optimizing again,
we can see that it's increasing.
So it got up to about 89% here once it's done training.
But those five epochs, around 89%.
It varies.
Sometimes it can be 91%, 92%, depending
on the initial initialization.
But here it's pretty good, at 89%.
But that's at the images that it was trained on,
so those images that it's seen.
How will it do with images that it hasn't seen?
And we can use model.evaluate to do that.
So here I've passed it the test images and the test labels.
It's measured its prediction for what it thought it would do.
And then it measured against the actual answers.
And it got it right 86% of the time.
So something here that we've kind of
done very simply and in just five epochs,
it's actually 86% accurate with data
that it hadn't previously seen.
So that's pretty good.
And if you want to work through some other exercises,
you'll see there is some exploration exercises
further down in the notebook.
You can work through those yourself
and try to figure them out.
And that will get you ready for the exercise that
comes with this video.
OK, so you've now seen how to classify items of clothing
from Fashion MNIST.
There's a similar data set, with 28-by-28 grayscale
images in 10 classes, and it's handwriting of the digits 0
through 9.
See if you can now build a classifier that
recognizes handwriting digits in this exercise.
Don't worry if you're stuck.
I'll give you the answer in the next video.
And if you have any questions, please leave them
in the comments below.
Whatever you do, don't forget to hit that Subscribe button
for more great TensorFlow content.
Thank you.
[MUSIC PLAYING]