In this video, we're going to discuss back propagation.
In particular, we'll do a toy example of back propagation, and we'll discuss the vanishing
gradient.
So there's a lot of math in this section, and I don't want you to worry about that,
all I want you to take away from this section is that back propagation reduces a number
of computations involved in calculating the derivative; and the problem of the sigmoid
function i.e., the vanishing gradient.
So it turns out that neural networks are functions of another function.
So the function a, is applied to X and then the function Z, is applied to the function a.
So to help illustrate the process we're going to use these set of gears.
And we have the gear X, and the gear X rotates at a certain rate, and the gear X affects
the gear Z.
But there's also an intermediate gear and we'll denote it by a.
So it turns out that the rate of Z is affected by a, and then by X.
So that when we take a derivative of a function it actually takes it into account.
So the derivative Z with respect to X is the derivative of X with respect to a, multiplied
by the derivative of a with respect to Z.
And we call this the chain rule.
So let's take a look at this simple toy example of a neural network with one hidden layer
and one output layer.
And you want to note the derivative of the hidden layer and the output layer; so we can
perform gradient descent on these two terms.
So let's calculate the derivative.
We have a loss function and let's just use a square error in this case.
So the derivative the output layer is the derivative the loss function.
And the loss function is a function of the sigmoid function.
So we'll have to take a derivative of the sigmoid function with respect the linear part
of the output layer.
And then we'll take the derivative of our parameter with respect to the output layer.
Things get a little more complicated when we want to calculate the derivative of the
parameter in the hidden layer.
So let's do that.
So, the derivative of, with respect to this parameter for the loss function, is the derivative
of the loss with respect to the output, which is a function of the sigmoid function.
And the sigmoid function is a function of the first linear layer.
And then the linear part of the output layer is a function of the first activation.
So we have to derive that with respect to the activation.
Then we have, then we have the initial sigmoid function, which produces the activation.
And just to note, it looks something like that.
And then we have the final parameter.
So that looks pretty complicated.
So what back propagation does, is it uses the derivative of the first parameter in the
output layer to help us calculate the parameter of the next layer.
And that's all it is.
So here's our two equations; and we're not interested in the actual equations; we're
just trying to understand the concept.
So we'll color them accordingly.
Now as you can see these two blue terms are present in the output layer and in the hidden
layer.
So let's just replace the two blue terms with this red term.
And let's see what happens to the parameter in the hidden layer.
As you can see, we can replace the two blue terms with this red term, and we get a computational
saving.
So let's see what happens when we have a deeper network; and we'll use our toy example again.
We'll have one neuron per layer.
So the derivative of the output layer looks something like this.
So the derivative of the layer preceding that layer looks something like this.
And the derivative of the layer preceding that looks like that.
And then the derivative of the first hidden layer looks like something like that.
So as you can see we have these two blue terms over here which represents derivatives.
Now we're going to replace those two blue terms with this light blue term.
So then we calculate this term over here, we simply replace these two blue terms with
this term.
And what we'll do is we will place these two yellow terms with this ugly maroon color term.
Now for this layer, we'll repeat the same process and we'll replace the blue terms with
the term we calculated over here.
The two other terms with this term.
And we'll replace these two green terms with the lighter green term.
So when we get to our final layer, we'll use all the terms we calculated before, and then
we'll also save these two red terms over here for the next layer.
So that's all that back propagation really does.
And as you can see from the final layer we reduced the amount of computations required
because you simply reuse the terms we calculated before.
So it turns out a neural network has multiple inputs, and the chain rule also applies, but
it gets a lot more complicated.
So, again we're not concerned with the actual math.
We're just concerned with the concept of what's going on.
So let's just replace these terms with colors.
So let's say we're going to calculate the derivative of the Awl'th parameter with respect
to the layer above it.
And we get all these terms.
Similar for the next neuron, we do the same thing.
And we repeat this process for the awl neuron.
So instead of layer L, let's say we're on layer L minus 1, and the derivative looks
like that.
And we can use the same reasoning and instead of calculating all those terms independently,
we can use the terms from the previous calculation and ... we can greatly reduce the amount
of computation.
So we're also going to talk about the vanishing gradient.
So let's say we're on the first hidden layer of our neural network.
So it turns out if you look at the derivative for the sigmoid activation function it looks
like that.
So what happened is we make it a bunch of very small terms and what happens is the derivative...
the gradient with respect to these terms at the beginning of the neural network become
very close to zero.
If we're multiplying the small number times another small number or if any of these terms
are zero, the gradient will equal zero.
And what happens when the gradient equals zero to our update equation, nothing happens.
So this is the vanishing gradient and there's been a lot of ways to solve it; I'm only going
to focus on a few.
So that's it!
And don't worry about the math for this section; just try to understand the basic concept.