We learned in the last video that there are a number of ways
in which we can calculate the intercept and the slope for the regression model.
Given this large number of options, one should ask:
"Why is least squares still the most popular way to fit these models?"
The answer is quite simple.
The least squares objective function is very fast on computers.
Because it is a differentiable function,
it's very easy to make mathematical proofs and derive properties from it.
It's also an intuitive function, easy to explain to someone who is not familiar with the process.
The other forms of fitting the linear model potentially have multiple solutions.
No one wants to get two different least squares answers from the same data set!
Other methods can be unstable and sometimes not converge, and the solutions,
the estimates for b_0 and b_1 can have high variance.
Mathematical proofs regarding some of these other methods are difficult to show.
And lastly, there's an important statistical reason.
The estimated parameters from the model, b_0 for beta_0 and b_1 for beta_1,
have the lowest variance possible when certain assumptions are met.
We'll see that coming up later on again in this module.
So I'm going to work with you next on solving the least squares model.
The important point to realize is that it is actually an optimization problem
that we are trying to solve, where we are aiming to minimize the sum of all the squared errors.
We do this over lowercase "n" values of error that we have.
We have n data points, and therefore we have n errors.
Square them, sum them up, and that's our objective that we're trying to minimize.
We can vary two things in order to minimize that sum of squares of errors.
The intercept estimate, b_0, and the slope estimate, b_1.
Now let's expand the equation for error.
The error, e_i, is defined as the observed value minus the predicted value.
If we substitute that in here, we modify what our objective function looks like.
Now I'd like to go interpret this with an example we have looked at before.
Let's go back to that gas cylinder example where the pressure was related
to the temperature using a least squares model with intercept beta_0 and a slope beta_1.
We know that beta_0 is zero from theoretical principles.
When the temperature is zero Kelvin there is no pressure.
Well how can we find beta_1 in the context of this example of solving an optimization problem?
Let me simplify the objective function here, then you notice,
our only search variable remaining is b_1.
We are trying to estimate beta_1 by varying this b_1 value.
Well to start off this optimization we need to have an initial guess.
And we have one.
We can go look at that formula and recognize that beta_1 is the number of moles
of gas multiplied by the gas constant, divided by volume.
If we sub in the values we get a number of 5.9.
That's a good initial guess for solving this optimization problem.
Let's go construct equally spaced points between 5 and 6.5, and then for every one
of these guessed b_1 values we go sub into the objective function over here.
Let's go look at that objection function as b_1 changes on our evenly spaced grid.
We will observe that as b_1 starts out here on the left with a low value of 5 and moves up
and up, that our sum of squares of errors starts to decrease.
If we keep increasing b_1, however, that sum of squares of errors goes up again.
We have a minimum somewhere around 5.8 and 5.9, which is pretty close to our initial guess.
That is the principle of minimizing with a single variable.
But a general least squares model has two variables,
b_0 and a b_1, and the same approach applies.
Except now we don't have a quadratic function anymore, we have a quadratic bowl shape.
The minimum will be at the bottom of that bowl.
You might not have taken a course in optimization, but you will recall from calculus
that this objective function is what's called convex.
And even if you don't remember that from your calculus course, what you should know is
that a convex function will always have a minimum.
And furthermore, that minimum is guaranteed to be unique, and therefore a global minimum.
The implication of that mathematics is
that for least squares regression the model will have only one solution.
Any data set that you try fit a least squares model for will,
you will only find a single value of b_0 and a single value of b_1 that minimizes
that objective function uniquely.
That's good to have.
We don't ever want different values of b_0 and b_1 for the same dataset.
Now how do computers then find where that minimum is?
The mathematical derivation is quite simple.
We know that for a minimum the first derivative of that function "f",
has got to be equal to zero at the optimum.
If you look at any bowl-shaped function and then the slope of that function the bottom
of the bowl, where that minimum occurs has a slope of zero.The plane that is tangential
to the bowl is horizontal, implying a slope of zero.
So what we go do then is take the partial derivative of F with respect to b_0
and the partial derivative with respect to b_1, and we set those equal to zero.
You can readily show that you get these summation equations.
These represent two equations in two unknowns.
In fact, these are two LINEAR equations in two unknowns.
And right there you see why least squares are so widely used.
Computers work very fast with linear equations.
This set of linear equations can be solved very quickly,
and in fact, you can even solve it by hand.
It is not hard to prove that b_0 is equal
to the average Y-value minus b_1 times the average value of X. But we need a value for b_1.
b_1 is equal to the summation here in the numerator: (x minus x-bar) multiplied
by (y minus y-bar), and in the denominator we have the sum of (x minus x-bar) squared.
Now if something about that numerator and denominator is starting to look familiar
to you, that shouldn't be a surprise.
They look a lot like the variance and covariance terms you saw earlier on in this course.
And we'll come back to that later on.
We can also go write these summations in matrix form to be a bit more compact.
The matrix form for the estimate for b_1 can be written
as (x minus x-bar) transpose times (y minus y-bar).
And in the denominator we have (x minus x-bar) transpose times (x minus x-bar).
We will also come back to the matrix form in a few videos from now.
There are a number of important properties
that we can derive just from just these two equations.
The first is that the units of b_1, the slope, are equal to the units of Y divided by the units
of X. The other important point that we notice is by a small rearrangement
of this first equation we can quickly prove that the least squares model always passes
through the point x-bar and y-bar with no error.
That artificial point, x-bar and y-bar,
will lie exactly on the least squares line right in the middle of the data.
The other thing that's interesting about the least squares model is
that we actually don't really care about the objective function value is,
the sum of squares of the errors.
We never really look back at what it is numerically.
But one number we are concerned about is the sum of the errors, or more specifically,
the sum of the errors divided by N. In other words, what is the average error?
The average error can be proved to be equal to zero.
This small proof here on the screen demonstrates that.
Pause the video and make sure that you can follow all the lines shown over there,
especially advanced students in this course.
One other final point to make is that it's quite clear that the estimate
of b_0 depends on the estimate of b_1.
Any estimation error in the slope b_1 propagates and affects the estimate of the intercept.
We say that these estimates are correlated.
Now one other final important property is this.
Let me ask you: you've probably built many least squares models.
Have you ever had the software crash when you build that model and fail
to give you a result for b_0 and b_1?
Probably never, and here's the reason for it.
If you look at the formula for b_0 and for b_1 you can see that b_0 can always be calculated
as long as we have a value of b_1.
And b_1 can always be calculated as long as this denominator is not equal to zero.
So what would it take for that denominator to be zero.
What would your data have to look like for that situation?
Pause the video and think about it before I give you the answer.
Hopefully you came to the conclusion that as long as there is variation in the X data,
as long as the variance of X is nonzero you will always obtain a solution
from the least squares model.
As long as there is variability away from the mean then you can always calculate b_1.
To end off this video,let's go through a practice problem.
Calculate the least squares estimates for b_0
and b_1 using the two formulas you've just learned.
Here are some raw data, and I've calculated some summary numbers from them to help you out.
Did you get the answers shown here on the screen?
If not, make sure you can confirm them.