LAURENCE MORONEY: Hi, and welcome back
to Machine Learning Foundations.
I'm Laurence Moroney, your host and guide.
And I'm here to step you through the basics of machine learning
from a coder's perspective.
Over the last few videos, you've looked
at convolutional neural networks and how they can
be used for computer vision.
You built classifiers for fashion, horses and humans
and cats and dogs.
But one theme that was common was the concept of overfitting.
We're given limited data to train on.
Your neural network could get very
good at recognizing this data that it was trained on,
but not necessarily very good at recognizing any data
that it had not previously seen.
To help this make sense, consider this scenario.
Say your whole life, you had only ever seen
shoes that look like these.
Then to you, that's what shoes would look like.
They'd cover the ankle.
They'd have flat soles that have laces--
all those kind of things.
So then if I showed you this picture,
you'd instantly recognize everything here is a shoe--
the big ones, the small ones, the ones on their sides--
they're still in your mind, shoes.
But what if I show you this picture?
Now, we know this is a shoe, because we've been exposed
to shoes that look like this.
But again, if all you had ever seen
is the shoes I mentioned earlier,
you wouldn't recognize this.
There's no laces.
The sole is all wrong.
And it probably doesn't cover your ankle.
You would then be overfit in the data
that you think makes what a shoe is.
The same happens with a neural network.
If it hasn't been exposed to a label sample that
looks like a high heel shoe, then it's
the same as our hypothetical person
who has only ever seen ankle boots as shoes.
Of course, if you don't have the data, you don't have the data.
But there are many cases where a neural network overfits because
of attributes in the image that it may not
have previously seen.
But using something called image augmentation,
we might be able to fix this.
This might seem a little vague.
So let's show an example.
On the left is a cat.
And on the right is a person.
Computer vision works by isolating features.
So you might have features like the pointy ears at the top
being indicative of a cat or the two legs at the bottom
being indicative of a human.
But in the case of a cat, for example, the image on the right
is a cat.
You and I know that.
But a computer may not, because if it was only
trained on images like the one on the left, then
it's looking for triangular shapes oriented upwards
near the top of the image.
But the ears on the cat on the right don't look like that.
So what if on our training set, we could rotate the images?
So we trained with a labeled image
of a cat that looks like this.
Then the ears are the same as the cat on the right.
So by using a transform like a rotation,
we're effectively creating new data to be able to train on.
The good news is you have all the tools you need already
to get started on this using image
augmentation to artificially extend your data
sets to provide new information for the training
of your neural network.
And this can really help you with overfitting issues.
To get started, you can use the image data generator.
You've already seen this for image manipulation
with the rescaling that you've been
doing to normalize the image.
And it supports more parameters.
So let's explore some examples.
So here's an extended image data generator constructor
where I've added a bunch more parameters,
as well as the rescale one that you've already seen.
You've already seen rotation range.
This will randomly rotate each image
by up to the parameterized amount, plus or minus.
So here you can see it's 40 degrees.
So the image will be rotated left or right by that amount.
Width and height shift range will move your image around
in the frame by that amount.
So here, it can be moved by up to 20% in width or height,
which can give an effect like this
where the image is width-shifted and the subject
moves to the left.
Sharing can give an excellent effect.
And you set it up with this parameter.
The value's from 0 to 1.
So with 0.2 like this, you can shear by up to 20%.
To see when it can be useful, consider these images.
The image on the left is from the Horses or Humans data set.
And the person is standing with his hands raised.
The image on the right has nothing like it in the data
set.
And as such, it might fail to be classified as a human.
We're overfitting to standing poses, for example.
But if the left image was sheared,
it can suddenly look a lot more like the image on the right.
The network while being trained could
be trained to recognize an image like the one on the right
and would be no longer overfit just for standing poses.
Another example is the zoom range.
And this parameter gives us a value from 0 to 1 to zoom by.
This will give us a random zoom of 0% to 20% of the image size.
Again, let's see an example of why this is useful.
The image on the left here is a woman
from the training set from Horses or Humans.
The woman on the right is quite similar.
But because of how the image is framed, we can't see her legs.
And again, the neural network might
be overfit to seeing full bodies that include legs.
The lady on the right is obviously human.
But the neural network may not recognize that.
But if we were to zoom in on the image on the left,
it now has far more similarities to the woman on the right.
And we may be able to recognize the woman
on the right as a human if we train with data
like that on the left.
Another example is horizontal flip.
If this is set to true, then images
will be randomly flipped.
And let's take a look at an example
of how this could be effective.
The image on the left is from the Horses and Humans data set.
As you can see, she has her right hand up.
The woman on the right is quite similar.
But she has her left hand raised.
If the training set only has images
like the one on the left, we could overfit for humans
with their left hand raised.
But if we randomly flip, we could effectively
extend our data set to include left hand raisers.
Fill mode equals nearest is used to fill
in parts of the image that could be
lost in some operations like skewing.
So that's a tour of some of the operations
that are available with image augmentation--
a really cool tool to help you avoid overfitting
in image-based data sets.
But let's put it into action.
Next, I'm going to show you a screencast of exploring image
augmentation with a reduced version of the Cats and Dogs
data set that has only 2,000 images.
And we'll see the impact.
So here's the Cats versus Dogs example.
And what I'm going to do is use a filtered version
of the data set that you can download from here
that only has 2,000 images.
Because it has less images, it's going
to be more prone to overfitting.
We'll take a look at the impact of using image augmentation
on that overfitting.
But first of all, I'm going to create one that does not
use image augmentation.
So I'm just going to train a neural network with it--
going to use exactly the same parameters later on one
that does use image augmentation so we can see how it all works.
So you can see here, it's downloaded the images.
It's setting everything up.
And it's going to start training the model.
It's going to train for 100 epochs.
So we can see the training is beginning now.
And then I'll speed it up and see how it works later.
OK, now, we can see that it has finished training.
And the accuracy on the training set
was 99.85%, so very, very high.
But on the validation set, it's a bit lower at 75%.
So that's clearly showing some kind of overfitting.
In particular, we're expecting that because we're
using a smaller data set.
And if I plot it out, the training versus validation
accuracy, we can see that it actually flattened out
at around about 30 epochs.
And the validation accuracy flattened out
at around the same.
But you can see our loss drastically increasing
on the validation set, clearly, again,
showing that we're overfitting terribly.
So what can we do about this?
So one of the things that we spoke about in the video
was the fact that we can do image augmentation.
So here's an example of using the image data generator.
Rotation range up to 40 degrees in either direction.
Width and height shifting 20% in either direction, shearing 20%,
zooming 20% and then randomly horizontally flipping the image
just like we had seen earlier.
So let's now rerun the code.
And this code is exactly the same
with exactly the same model, except in this case,
we're going to run the image data generator code to do
the image augmentation.
So if I start running this, it's downloaded the cats and dogs.
It's giving us 2,000 images in two classes and 1,000
validation images.
And it will start training.
One thing you'll notice is that the training
is a little bit slower this time because it's
doing the image augmentation on the thousands of images.
So while earlier if I go back, we'll see our epochs
were taking about 9 seconds.
We'll see this time round, even though we
have 100 epochs-- it's the same-- it's going
to be a little bit slower.
It's taking about 19 seconds.
So we've about doubled the training time--
a little bit over double the training time--
because of the fact that we're doing
all of that image augmentation.
So now, as we can see, we finished training.
We ended up with 85% accuracy on the training set
and 83% accuracy on the validation set.
Now, while this may not look as good as your 99-plus % that you
had earlier on, the point is here is that it's not
overfitting.
We have 85% on training and 83% on validation data
that it hadn't previously seen.
So this model realistically is 85%
accurate at predicting cats versus dogs.
And when we look at our training and validation accuracy,
we can see the curves are fit together very nicely.
And the training and validation loss
also fit together quite nicely.
Maybe here in some of the later epochs,
they're beginning to diverge so that it's
showing that's beginning to overfit at that point.
But we've probably overtrained the network at that point.
We might well only want to go for about 60 epochs
or something like that so that we can be sure.
But again, this looks very, very nice
now with the accuracy and validation
being very close to each other.
And we got that by using the image augmentation.
Another little trick for avoiding overfitting
is to add drop out.
Now, we didn't cover that in the videos.
But I'll just show a little sample of it
here where the only thing that I've done differently
is I'm adding this dropout layer to the model after everything.
And I'm going to be pretty aggressive in dropping out.
And that's dropping out half of the neurons.
So in this case, it's to prevent what's
called a proximity bias when neurons that
are close to each other can be trained
with very similar values.
If you drop out a bunch of them, you can get rid of that.
And again, you can prevent overfitting.
So let's run this one.
And it's going to do the same augmentation
that we had earlier on.
It's downloading the data and setting it up.
It's going to start training it.
The training here maybe a little bit slower.
Let's take a look.
Well, again, it was the same.
It was about 19 seconds.
So the dropout wasn't impacting it in any way.
But let's train this now for 100 epochs.
And then after 100 epochs, we'll take a look
at what the validation versus accuracy curves look like.
Now, that we can see we finished recording, we're at about 82%
accuracy and about a little over 82%, almost 83% accuracy
on the validation set.
So we might actually even be underfitting slightly.
But, again, getting accuracy and validation accuracy
as close as possible to each other
is M sign of a really strong model.
And you can see we have that here.
We're getting some wild swings here in the later epochs.
And that's probably because of the dropout.
And we're training a little bit too long.
But, again, using dropout we're getting our accuracy
and our validation accuracy closer to each other
to avoid that kind of overfitting.
Sometimes you can go too far, maybe underfit slightly.
I'm not sure if that's the case here,
because if we take a look at previous epochs,
usually the training accuracy is a little bit higher
than the validation one.
So that's it.
Those are some techniques, particularly augmentation,
that will help you avoid overfitting.
And I hope this has been useful for you.
Here's the URL, pause the video, and give it a try for yourself.
And that's it for this episode.
In the next one, we're going to switch gears and start
looking at natural language processing
beginning with the concept of tokenization.
I'll see you then.
And whatever you do, don't forget
to hit that Subscribe button.
[MUSIC PLAYING]