Hi this is Jeff Heaton welcome to Applications of Deep Neural Networks,
with Washington University. In this video we're going to see our first technique
for fighting against the constant enemy of the neural network programmer and
that's overfitting. We're going to see how to use early stopping rather than
trying to decide how many epochs you want to train your neural network for. You can
set aside a validation set and train the neural network until that validation set
no longer, the error on it no longer improves. For the latest on my AI course
and projects click SUBSCRIBE and the BELL next to it to be notified of every new
video. Early stopping is a method that helps to prevent overfitting.
We'll see other methods to help prevent overfitting as well, as we progress through
this class. But what early stopping really tries to do is keep you from
training the neural network too far. This is a common visualization to show what
ill effects overfitting can have as you train more and more. X-axis shows you the
error, the Y-axis shows you the training time, the dashed line is the validation
set, and the black line is the training set. So what is happening to the error of
your training values versus your validation values, as you train longer
and longer and longer. Generally, the validation and training
will be pretty similar. The validation might be a little bit worse as you begin
to train because the training set is going to be somewhat over fit, because
the neural network has those values available it's memorizing some of them,
it's fitting very well to the training set, so the validation set will usually
be a little bit worse as you're progressing then as the training set
gets better and better and better at some point the validation set will start
to almost converge away from this to a higher error value, and you'll get worse
and worse values on your validation you'll get better and better values on
your training. This is result of overfitting, at this point here
overfitting really started to occur. Now you may not recognize it till somewhere
out here, because you have to think, ok it's it's getting worse maybe it'll get
better, maybe it'll get better. Nahhh, probably not
getting not getting better. Now there are cases where you'll go out
this far and then it'll it'll come back down here. This is a value you will see
in this class called patience, how patient are you to see these values
start to move towards convergence again, rather than diverge away into validation
just overfitting, and and not getting a better value. This is what early stopping
is. You stop early, you stop at this point, and we'll see that it's important that
we capture the weights here where we had essentially the best, or the lowest
validation error, which is probably right in here, where we had the lowest
validation error and then we save those weights, so that we have them so that
even when we decided to stop maybe here or here depending how patient we were
we'd go back to here. So that's an important thing to remember too about
patience. Patience will really not hurt you on the overall best score that
you'll get at the end. You can be as patient as you want. It's just going to
take a lot longer to train because you're going to have to go much further
out if you set a gigantic patience number of epochs that you're willing to
wait. Now to do this we need to segment the original data set into several data
sets. Now often we'll just have the first two, but sometimes we'll have a holdout
set as well. You may even split it up into more than
just these three, depending really on what you're doing it also depends on how
much how much data you have if you have a lot of data then you may split this
into many training sets and really ensure that you're not overfitting that
you're not ever evaluating based on anything that you're trained on you can
make this very pure and you may even split out a separate validation set just
to be used for nothing other than early stop but every time you split these you
lose some of your training data so it's it's a trade-off between how purists do
you want to be in terms of your never crossing training and validation never
validating on anything that the neural network is
trained on how much of your training set you want to train on because your your
neural network will typically perform better at least up to a point
the bigger the training set that you give it and Carroll competitions for
example I will usually accept some small cross contamination of training and
validation meaning I might use a validation set for for early stopping
just because I know that I have that holdout set that Kaggle will eventually
give me I can't do that too much or I over fit to even cackles holdout set to
some degree so it's it's really just a trade off and we'll talk more about this
as we get into hyper parameter tuning and other things we do need to create
additional set the ultimate insurance policy though is you set that final
holdout set that you keep until the very end and you don't you don't use this for
anything during the training process until you say okay I am I'm very happy
with my model let me see what happens when I try the final final holdout set
and that's that's the error rate that you usually decide to commit to as far
as how how well-trained your model actually is so this is what it really
looks like in terms of something like an early stopping what you're going to do
is take your data set break it into maybe 80% and 20% this 20% up here
becomes the validation the 80% becomes the training and you're going to fit or
train your model on this 80% and then you're gonna validate it on this 20% so
you'll get a training error here. That is how well it was fit to your training set,
but then you take what you trained on that 80% and validate it with that
remaining 20% and that gives you greater confidence that this is truly the error
that your neural network is going to experience when it sees new data. Now
we're gonna look at early stopping. And early stopping does require a sort of
validation that now that causes your validation set
to now be used for early stopping. So now it's part of the training set so this is
where it's good to have that final holdout set that you that you might want
to ultimately evaluate it on. Just be aware of that if you're if you're truly
using your validation set as your early stopping, then you probably need another
validation set to actually commit to an error on, or you can accept that maybe
you've got some inflation in your validation set, and you can truly use
whatever score you stopped on as an estimate of what the final neural
network might do. But be aware that you have crossed at that point your early
stopping validation set to your training is what we're going to do is just
monitor this and as soon as that validation set no longer is decreasing
and our patience expires, we're going to stop. We're gonna early stop going to
show you two examples of how you will do this we'll do early stopping with
classification and early stopping with a regression. The code is mostly the same,
but this gives you some good starting point code to look at for this. So for
this one we're going to use the iris data set will also load this sepal
length, width and petal length and width. Let's go ahead and run this so that we
load the data in as soon as the star goes away it's loaded. Okay
the data is loaded and we are we ran it and you can see the results here we'll
look at the results first and then I'll show you kind of what happened. We
trained and trained and trained you can see the training of validation loss and
then it says that right around 101 we decided to early stop and we restore the
model weights of that last best epoch, and we set up our neural network largely
just like we did before. It is going to have its classification so we're using
the Y shape, reason the number of classes and softmax and categorical cross
entropy those three together tell you that this is a classification neural
network. Make sure you set it up like that we're creating a monitor now this
is the part that does the early stopping. We are telling it that we have a
patience of five, so we're going only wait five epochs for to quit
improving. Verbose is one so that'll give us that little notice that we saw down
there and we're telling to truly restore the best weights, and now we're going to
fit it, but notice we're giving it x-train and y-train, and we're passing in
validation data we haven't done this before, so the validation data of the X-test,
Y-test versus the X-train Y-train that came from up here we're splitting
into validation and training sets so we're splitting 25% is going to the test
set the remaining 75% is going to validation so you have X-train X-test,
Y-train those are the expected values the Y's and Y-test so now you have the 4
measurements split into a train and a test and you have the expected class the
type of iris that it actually was split into train and test the train of state 42
that just gives us a way to consistently split it and then the only other thing
we do on fit down. Here is where pass a callback monitor so that's how this
early stopping object that we created actually plugs into the training, or the
fit process. When we call fit as we trained so this is how you implement
early stopping for classification it's pretty much the same exact process for
regression. You create a monitor you pass it into the callbacks and you need to
provide validation data. I just give you the exact code for this so that you can
quickly make use of it. By the way, if we do want to see the accuracy, that we
got for early stopping, notice the accuracy is very good, so we didn't have
to guess at the number of epochs, it trained until it saw that we had trained
enough. So this is a way that you can estimate and not have to guess at how
many epochs you want to give your neural network. This is a way to have one fewer
hyper-parameter. Hyper-parameters are all these things you have to pick about your
neural network that will influence behavior. Like the number of epochs, how
many layers you have, we'll see how to pick layers and
things automatically later in this class. But this is one way that you can at
least use mathematical accuracy to some degree to pick your number of epochs to
train. And a simple example of early stopping with regression we use the auto
miles per gallon, this is the same setup that we've done before we're splitting
it into X and Y, so that Y is the miles per gallon that's what we're trying to
predict. X's all these other values that we're using to predict miles per gallon
and then we use exactly the same code that we used in the previous one for
classification. We split it into train and test 25% split. Now we're doing
regression though, so notice we have one output neuron. We have mean squared error
that tells you that this is a regression neural network, and this is all the same.
We're giving them we're passing the monitor N and we're giving it X and Y,
both for validation and for train so you're giving it that two sides, the
training and the validation, and now we can run it and a trains... trains... trains.
You can see the testing the the train loss and the validation loss both are
decreasing and at some point it decides to stop. It restores the final weights so
it only made it to 31 epochs, before it lost patience here. Then you can measure
the final root-mean-square, not a particularly good root-mean-square: 12.
Now remember, neural networks start with random weights so this gives me a chance
to quickly illustrate something interesting remember root mean square
was 12, training the whole thing now let's
recalculate this for notice that huge variance this is really annoying about
neural networks, but since they start with random weights they will get
sometimes different error values this makes it very hard to try to tune
something. Imagine if you had thrown in an extra hidden layer when you did have 12,
but then you throw in a hidden layer, oh my gosh I went down to 4 that's great!
No it was just the normal variance you have in neural networks we'll see. In the
next module that's something bootstrapping will you will use your
just literally going to have to run this a few times an average together is how
you handle that and things like drop out help mitigate this at least a little bit
this is just one of the facts of life dealing with neural networks, they give a
great deal of variance in the in the score that you that you get from the
output. Thank you for watching this video on early stopping, in the next part we're
going to see how to actually extract the weights from a saved Keras neural
network and put those weights into a mathematical equation and actually
calculate the values. This removes all the magic from the process and you can
see that it's simply mathematical calculations. This content changes often
so subscribe to the channel to stay up to date on this course and other topics
and artificial intelligence.