To help us work with these multiple linear regression models efficiently we need
to use some matrix notation.
We will look at that in this video.
The first step here is that we are going to remove the intercept from consideration.
This is not as bad as it sounds.
The intercept is seldom of interest, and we can always recover it if we want to know it.
We remove it by first stating the regular regression model.
Then, in the second line here we restate the regression model,
at the point which we know the model will pass through: the averages.
Thirdly, we subtract the second line from the first line.
This creates a new regression model where our variables are in their centered form.
In other words, instead of x_i, we use x_i minus the mean of all x values.
Instead of y_i, we use y_i minus the mean of the y's.
Recall that centering is a process that simply shifts the variable,
but maintains all of the relationships that it has.
When we build our model with these centered x and y variables, or,
as we sometimes say "deviation variables", we now have a model where the intercept is 0
and does not need to be estimated.
Notice from the second line over here, we can always recover that intercept value back,
if we were really interested in it.
Now sometimes you will see me build regression models with the intercept in it,
especially in the Designed Experiments part of the course.
So you really don't have to do this step, but it does help for interpretation later on.
So now we now have a model without an intercept, but have k parameters, beta_1, beta_2,
up to beta_k, and each of them are multiplied by their corresponding x variable.
Right at the end, we have an error, epsilon_i, and all of that is added
up in a linear combination to give you a value of y_i.
We can express this observation as a linear combination in vector form.
Here we have a vector "x" and a vector "beta", and this is for the i-th observation.
So we collect our vector x, the input data, and the vector beta, and we multiply them,
using regular matrix multiplication.
Add the epsilon_i value to the end and then we get our y_i value.
There are k values of x, multiplied by their corresponding "k" coefficients of beta,
and we add those up to give our single y-value.
And we will call that vector x_i, to emphasize that that is the row vector
for the i-th observation of x. Now if we have multiple observations of x,
then we can stack those vectors in a matrix,
and form capital matrix X. Row 1 is the first observation, x_1,
row two for the next observation x_2, and so on, up to the last row, x_n.
I've also replaced the beta's now with b's to indicate that we are going
to estimate the b values rather than knowing what the population model is.
So we have a matrix of x observations, n rows, and k columns.
There are n rows for observation and k columns for the k input variables.
Multiply that here by our vector, b_1, b_2, up to b_k.
So "k" entries of b, and then we add a vector of errors at the end.
I've emphasize the dimensions of each of the vectors and matrices over here.
Now the objective function for multiple linear regression is exactly the same
as regular least squares: minimize the sums of squares of the errors.
If we take that vector of errors over here, lowercase "e", the sum of squares
of the errors can be easily shown to be "e" transpose "e".
Well, the vector e can be written as y minus matrix X times vector b. If we sub
that in, we get the second row over here.
And if we expand that matrix vector transpose, we can use some basic rules
of matrix-vector algebra to get this revised objective function.
We've seen before that we can minimize this objective function
by setting the partial derivative of it, with respect to the search variables, to zero.
That means the partial derivatives with respect to each
of the entries in vector b are set equal to 0.
That will give us k equations in k unknowns.
We have k partial derivative equations, and there are k values of b: b_1, b_2, up to b_k.
Now these "k" linear equations in k unknowns can be solved very compactly using what is probably
the best-known formula in least squares.
That the entries in vector b are equal to X transpose X inverse multiplied
by X transpose y. Don't forget that formula.
You are going to see it reoccurring several times in this module as well
as in the next module on Designed Experiments.
And it's time for an example now: where we've got two vectors,
x_1 and x_2, shown here on the screen.
Let's use these to predict the vector y. The first step we are going to have to take is
to center the vectors X and the vector y. We'll do that, and the results are shown here.
But before we go ahead and even solve this equation.
What do you expect the slope to be for x_1 as it relates to y?
And what do you expect the slope to be for x_2, as it relates to y. Examine the data and write
down your answer; keep it aside, and we will verify it later.
Now let's go solve this mathematically, and try to verify our intuition.
The first we need to do is assemble those two individual vectors into a matrix,
capital letter X. The first column will represent the entries for variable x_1.
The second column in matrix X represents the second variable, x_2.
Notice that we have "n" observations, six in this case.
So there must be 6 rows and 2 columns, one for each of the input variables.
X is always an n-by-k matrix.
And the y vector is always an n-by-1 vector of the output values.
These are assembled now as shown here on the screen.
So we are ready to go calculate vector b, the two coefficients of our regression model.
X transpose X inverse multiplied by X transpose y. Let's go calculate the first part:
X transpose X, that's shown over here.
Then we can go calculate X transpose y separately, shown over here.
You're welcome to use any software tool to do this,
but here on the screen are the R instructions, in case you wanted to try it over there.
But we actually need the inverse of X transpose X, and you can also do
that in R with this line of code.
So now we have X transpose X inverse and we can go multiply it by X transpose y. I want to point
out that X transpose X will always be a k-by-k matrix,
k being the number of variables in our model.
In this case, k=2.
It is going to be an important point later on that the entries
in X transpose X represent the covariance of the X variables with each other.
For example, the entry in the 1,2 position in this matrix is the same
as the entry in the 2,1 position.
That is because the covariance of variable x_1 with variable x_2 is the same as the covariance
of variable x_2 with variable x_1.
You will recall from an earlier video, and that that should be the same number.
The entries on the diagonal are also covariances, but they are covariances
of the variables with themselves, so we simply call that variance.
In this case, the 55.5 over here is directly proportional
to the variance of the x_1 variable.
The 62 is proportional to the variance of the x_2 variable.
Because the diagonal entries in X transpose X are variances, they should always be positive.
Variances are, by definition, always positive values.
And the off-diagonal elements in X transpose X will always be symmetrical.
Once you've calculated the upper diagonal part,
you can just copy the values over to the lower diagonal part.
Now here's a question I want to leave for advanced students.
What will the X transpose X matrix look like when the variables x_1 and x_2,
and so on are entirely uncorrelated with each other?
What will you observe in those off-diagonal entries?
You will see the answer to that coming up in the next module,
but you should be able to guess what it is now.
Now if X transpose X represents variances and covariances,
what do you think the entries in X transpose y represent?
If you guessed that it is the covariance of the entries of X with y, you are right.
The first entry in the X transpose y result is the covariance of variable x_1 with y,
and the second entry is the covariance of x_2 with y. Now take a look at the signs:
do they match your expectations what you wrote down earlier?
Look back at the raw data and make sure at least
that the sign information matches your expectation.
Remember right at the start of the section on linear models, we said for covariances,
at the very least you can interpret the sign of that result.
The magnitude of the value isn't so useful.
And those results that we see there, the 36.5 and then -36, seem to make sense:
the positive sign for the first entry indicates
that the first x variable is positively correlated with the y-variables.
And we see that in the raw data.
As that first column increases, the column of y also increases.
The second entry in X transpose y is negative, indicating that that column
in x has a negative correlation with y, and that's also observed in the raw data there.
Now one final thing to mention before we end this video is
that you should never calculate your least squares model using this X transpose X inverse
formula multiplied with X transpose y for real, general data.
It works in some specific cases very well, and that's why we will use it later on.
But in general, there is very poor numerical conditioning
with calculating the inverse in this way.
Sophisticated least-squares packages, such as R,
will calculate the linear model using the QR decomposition instead.
It is a similar type of linear algebra step
and interested students should go investigate that in their own time.
The R software package uses the QR decomposition behind the scenes and not this inverse.
And since we are talking about R, let's go see how R solves this example problem.
Use the "lm" command and say y is regressed on two variables: x_1 plus x_2.
Try that out and use the summary command on that model result, to see if it matches the output
that you calculated using the X transpose X inverse, X transpose y equation.
There's one last entry we need to go calculate, and that is the standard error.
Standard error, you will recall, is equal to the objective function of the least squares,
divided by n-k, and then we take the square root.
So in other words, standard error is calculated as e transpose e, divided by n-k,
and then we take the square root.
Once we have that error vector, it is easy to go calculate the standard error.
In the next video we are going to look at the important topic
of interpreting the multiple linear regression coefficients.