>> So we are going to need a learning algorithm that is more robust to
non-linear separability or linear non-separability. Does that sound right?
>> Non-linear separability. Non linear separability.
>> Non?
>> Yeah think of it.
>> Left parenthesis, linear sep, spreadability left parenthesis.
>> There we go, that's right, negating the whole phrase, very good.
So and, Gradient descent is going to give us an algorithm for doing
exactly that. So, what we're going to do now is
think of things this way. So what we did before
was we had a summation over all the different input features
of the activation on that input feature times the weight,
w, for that input feature. And we sum all
those up and we get an activation. And then we
have our estimated output as whether or not that activation
is greater than or equal to zero. So let's imagine
that the output is not thresholded when we're doing the training,
and what we're going to do instead is try to figure out
the weight so that the Not thresholded value is, as close
to the target as we can. So this actually kind of brings
us back to the regression story. We can define an error
metric on the weight vector w. And the form of that's
going to be one half, times the sum over all the data
in the dataset, of what the target was supposed to be for
that particular example minus what the
activation actually was. Right? The activation
being the dot product between the weights and the input and we're
going to square that. We're going to square that error and we want to try
to now minimize that. > Hey Michael, can I ask you a question?
>> Sure.
>> Why one half of that?
>> Mm. Yes. It turns out that it turn, in terms of
minimizing the error this is just a constant and it doesn't matter.
So why do we stick in a half there? Let's get back to that.
>> Okay.
>> Just like in the regression case we're going
to fall back to calculus. Right, calculus is going to
tell us how we can push around these weights,
to try to push this error down. Right, so we
would like to know. How does changing the weight
change the error, and let's push the weight in the
direction that causes the error to go down. So
we're going to take the partial derivative of the, this error metric
with respect to each of the individual weights, so that we'll
know for each weight which way we should push it a
little bit to move in the direction of the gradient. So
that's the partial dif, dif, [INAUDIBLE] So that's the partial derivitive
with respect to weight wi, of exactly this error measure. So
to take this partial derivitive we just use the chain rule
as we always do. And what is it to take the
derivitive of something like this, if you have this quantity here.
We take the power, move it to the front, keep this
thing, and then take the derivitive of this thing. But that,
so this now answers your question, Charles. Why do we put
a half in there? Because down the line, it's going to be
really convenient that two and the half canceled out. So, it's
just going to mean that our partial derivative is going to look simpler,
even though our error measure looked a little bit more complicated.
So so what we're left with then, is exactly what I said,
the sum over all these data points of what
was inside this. Quantity here times the derivative of
that, and here I expanded the a to be,
the definition of the a. Now, we need to take
the partial derivative with respect to weight w i
of this sum that involves a bunch of the ws
in it. So, when don't match the w i,
that derivative is going to be zero because the, you know,
changing the weight won't have any impact on it. The only
place where this, changing this weight has any impact is at
x of i. So that's what we end up carrying down.
This summation disappears. And all that's left is just the one term
that matches the weight that we care about. So this is
what we're left with. Now the derivative of the error with
respect to any weight w sub i. Is exactly this, this
sum. The sum of the difference between the activation and the target
output times the activation on that input unit
>> You know? That looks exactly like, almost exactly
like the rule that we use with the rule that we used perceptron before.
>> It does indeed! What's the
difference? Well, actually let's Let's write this
down. This is now just a derivative, but let's actually write down what
our weight update is going to be because we're going to take a little step
in the direction of this derivative and it's going to involve a learning rate.