In this segment, we're going to talk about linear
regression, and we're going to derive the expression
for the linear regression model, so we're going to talk about derivation.
Now, what regression is all about is that you are trying to best fit some
given data. So let me write this down first, and then
I will show you graphically. So basically it is
that you are given certain data points, let's suppose x1, y1,
x2, y2, all the way up to xn, yn. So you're given, let's suppose n
data points, and what you want to be able to do is that you want to best fit
a straight line to it, and that straight line, let's suppose
we call it y is equal to a0, plus a1 x, that's
the general form of a straight line, y is equal to a0, plus a1 x, where a0 is the
intercept and a1 is the slope of that straight line. So what we're trying to do
is that somebody's giving us these data points, and what they want us to do is they want us
to best fit a straight line to this data point . . . to these data points.
So let's suppose somebody says, hey, this is the straight line which you
are drawing. So the equation of the straight line which we have is y is equal to
a0, plus a1 x, and this is, let's suppose, x1,
comma, y1, and this is let's suppose some xi, comma, yi, some ith
data point, let's suppose, and basically you are drawing all . . . showing all those n
data points on this graph here, and then you are trying to
draw a straight line to best fit those data points. Eventually what you want
to do is you want to minimize the amount of difference which you have between these
observed values, which are the ones which are given to you,
and what the straight line will predict, because you want to minimize the
amount of difference between, but at the same time you have the differences at several points,
at all the data points. You'll have a difference here, you have a difference here,
you've got a difference here, so everywhere there's a difference between what you are
observing and what you are predicting. So the cross
is the one which you are predicting, and the dot
is the one which you are observing. And now this data point here, or
not the data point, but the predicted value at that particular point will be simply
a0 . . . it'll be simply a0, plus a1 times xi, because that's a straight line
which you are going to find, and then if you put the value of x equal to
xi, that's what you're going to get for the predicted value, and this is the
observed value. So if you look at the amount of residual,
that's what it's called, some people call it error, so there's a residual
at each and every data point which you have, which will be the observed value, which is
yi, minus the predicted value, which is the predicted
value from the straight line, so you've got yi, minus a0, minus a1 xi. That
is the amount of residual which you are having
at each data point. Now, what you want to be able to do is you want to be
able to somehow make these residuals to be small everywhere, and one of the criteria
used is to simply take the summation of all the errors,
square it, and then you want to add all those errors,
the square of all the errors, and which is called the Sr, sometimes . . .
and the Sr stands for the sum of the square of residuals.
And that's what you want to try to make as small as
possible, because eventually what the goal is to minimize the amount of residuals, and
one of the criteria which is used is called the least squares
method of finding the regression model. And least squares
means that you are squaring the errors, and you are trying
to add all them, and you're trying to minimize them, and that's why it's called least squares
regression. So that means that it's summation of yi, minus
a0, minus a1 xi, squared, and that's what you are getting
from there, i is equal to 1 to n. So you want to minimize
the summation there. Now, you can understand that when somebody's telling you to do regression,
you have these as the observed values, and these are
the . . . this is what you're predicting, and the only control which you have is on
these constants of the regression model, so these are called
the constants of the regression model. And depending on what kind of regression model
you are drawing, in this case we are doing a linear regression, so a0
and a1 are the two constants of the regression model. You want to be able to find those.
Those are something which are in your control, and you
want to control them in such a fashion that this summation, this whole
summation becomes as small as possible. You cannot make it exactly equal 0. If you
make it exactly equal to 0, that means that it has to, the straight line has to go through
all the data points, which is not going to be
the case when you are doing regression. So you want to be able to make this as small
as possible. So what that means is that we have to derivatives
with respect to a0 and a1, put those equal to 0, going back to your differential
calculus class, to be able to minimize this expression there, minimize the summation here.
So what I'll have to do is take the derivative of the sum of the square of the residuals
with respect to a0, put that equal to 0, take the derivative Sr with respect to a1,
and put that equal to 0. So that's what I will have to do, and once
I put those equal to 0, I'll get two equations . . . two equations,
and two unknowns, and once I get two equations, two unknowns,
I'll be able to find what a0 and a1 are.So in order to be able to do that,
I will have to take the derivative of this Sr with respect
to a0. So del Sr respect to a0 will be equal to summation, 2 times yi, minus a0, minus
a1 xi, times -1 equal to 0. You are simply using the chain rule, so it's
2 times whatever is in the brackets, multiplied by the derivative of this expression with
respect to a0, it will be just -1, and then del Sr divided by . . . with respect
to a1, summation, i is equal to 1 to n, summation
is equal to 1 to n, so it'll be 2 times yi, minus a0, minus a1 xi, times the
derivative of this particular quantity here, which will be minus . . .
minus xi equal to 0, because what you are doing is that you are
finding out what the derivative of this quantity in the bracket is with respect to
a1, and that'll be just minus xi. So we have set up two equations, two unknowns,
so we need to be able to figure it out, how we're going to solve these.
So let's suppose if I call this 1, and then call this to be 2.
So 1, what does it give me? It gives me . . . let me just expand this summation, expand
this summation separately. I'll get -2 summation, yi, i is equal to 1
to n, plus 2 summation,
a0, i is equal to 1 to n, plus 2 summation, a1 xi equal to 0, that's what I
get from the first equation. Now what do I get from the second equation?
Second equation, again, I get -2 summation, i is equal to 1
to n, x times . . . xi times yi, because I have yi here,
and xi here, so that's why I get that, then plus 2 summation,
a0 xi, plus 2 summation, i is equal to 1 to n, I'll get
a1 times xi squared, because I'll have xi here, xi here, and that's what I will
get from there. Now, I can see that 2 is common in here, so
I can get rid of 2 here, I can divide both sides
by 2, that's what I am doing, I'm going to divide both sides by 2, and now,
since I see that here, this is a known quantity
here, and this is also a known quantity here, but a0 and a1 are not . . . so
a1 is right here, a0 and a1 are not known, a0 is not known here,
a1 is not known there, a0 is not known there, a1 is there,
so I'm going to keep those on the left-hand side of my equations,
and this I will transfer to the right-hand side of the equations. So what I will get
from there will be, the first equation will turn
out to be i is equal to 1 to n, a0, then the second equation will turn out
to be i is equal to 1 to n, a1 xi equal to 0, and the second equation, so that's
our first equation, the second . . . not 0, it'll be summation of yi.
The second equation will be summation, i is equal to 1
to n, a0 xi, plus summation, i is equal to 1 to n, a1 xi squared
equal to summation, i is equal to 1 to n, xi
yi. So again now, I've got to realize that a0 and a1 are now constants,
because I have found those by taking the derivative of the sum of the square
of residuals with respect to a0 and a1, so I can take those out, and you can realize
what is this? This is adding a0 n times, because you're doing a0, plus a0, plus
a0, plus a0, so you're going to get simply this to be n times a0. So the first
equation, again, can be written as n times a0, plus, I
I can take this a1 outside, because it is a constant, so I can take a1 outside,
so I'm going to write it like that, and then summation, i is equal to 1 to n, yi. Then
again, here, I can take a0 outside, so I'm going to take
a0 outside, I get that, and then, again, a1 can be taken
outside, because it's a constant now, i is equal to 1 to n, xi squared,
and them summation, i is equal to 1 to n, xi yi.
So basically I have two equations, two unknowns now, and
I'm going to write them in the matrix form so that it is clear what the coefficient matrix
is, what the unknowns are, and what the right-hand side vector is,
and also it will help you to symbolically solve these two equations,
two unknowns, whether you're going to use Cramer's rule,
or whether you're going to use any kind of Gaussian elimination
symbolically. So all of those things can be done,
or you can just use your high school algebra to solve these two equations,
two unknowns. In many of the books, you'll find out that many people
drop this summation lower . . . these limits of summation from
their books. The reason why they do that is because all the summations are the same,
so far as the limits are concerned, but I try to keep them,
because I don't want you to lose focus of that this is, let's suppose you're looking
at this quantity, that this is a summation of the xis, all the
xis are being added together, that's what that stands for. So I keep the
summations, but if you are looking at a book, please don't think that I'm doing something
different, I'm just doing something which is more complete,
showing you all the limits of the . . . limits of the summations. So once
you solve these two equations, two unknowns, which are written in the matrix form, this
is what you're going to get, you're going to get a1 is equal to n times
summation, i is equal to 1 to n, xi yi, where n is simply
the number of data points which you have, summation, i is
equal to 1 to n, xi, summation, i is equal to 1 to n, yi, and then
you're going to divide by n times summation, xi squared,
i is equal to 1 to n, minus summation, xi,
i is equal to 1 to n, squared. And then a0 will be . . . you will get it to be summation,
yi, i is equal to 1 to n, divided by n, minus a1, summation i equal to 1 to n, xi,
divided by n. So once you have found a1, you can find a0
by simply by this simple formula here, but what is this? This is nothing but the average
value of the ys, so you get y-bar, minus a1, and this
is nothing but the average value of the xs, you get x-bar. So that's how you will
be able to find a0 and a1 by doing linear regression. Now, one of
the things which you've got to understand where people make a lot of mistakes is this
and this. If you look at this expression where you are summing the square of the numbers,
square of the x values, and here, where you are squaring the summation
of the x values, so these two are two different quantities. This is where you have to square
each x value first, and then add them all up, this one is where you add
all the values of x, and whatever you as the summation, then you square it. So that's
something to be thought about when you are applying these formulas to develop the linear
regression formula of y is equal to a0, plus a1 x. And that's the end of
this segment.