- [Dr. Nuttelman] In this screencast,
we're going to review multiple linear regression.
And then I'm gonna show you how we can use
something known as a matrix approach
to solve for the coefficients
in a multiple linear regression model.
Towards the end of this screencast,
we're gonna be working through a problem
in the back of your book.
We wish to develop a multilinear regression model
that relates the inputs;
we have age, the severity of the illness,
whether or not they had a medical procedure and anxiety;
to the output, which is satisfaction.
Note that the third column here
is something known as a categorical regressor.
It's either a zero or a one.
In regression models, in addition to continuous variables,
we can include categorical regressor variables.
In a multiple linear regression model,
we have multiple inputs.
We have the four inputs here:
x1 is age, x2 is severity,
x3 is whether or not they had a surgical
or a medical procedure,
anxiety is x4,
and satisfaction is our output.
Here, I've just set up a simple
multilinear regression model,
where we have the output as just some function
of each of those variables, x1 through x4,
multiplied by coefficients.
We have an intercept, beta 0,
and then we always have an error term on the end.
The error term, epsilon,
just represents random variation in our data.
One of the most basic multiple linear regression models
is shown here,
where, again, we just take coefficients
and multiply 'em by our variables.
We don't have any polynomial terms,
we don't have any interaction terms
where we're multiplying x1 and x2,
although you can certainly add those terms
into a model if you want.
This variable, k,
just represents the number of regressor variables.
In the example we're gonna be solving,
we have x1 through x4,
so k represents the number of regressor variables.
In addition to beta 1 through beta k,
we also have beta 0,
and so we can always add one to our value of k
in order to give us p,
which is the number of parameters in the model.
So if you have k regressor variables,
then you have k + 1 equals p parameters in your model.
By the way, polynomial models are considered in your text
to be multiple linear regression models.
We have a third-order polynomial model here
in a single regressor variable, x,
but your textbook basically says if we let x1 equal x,
x2 equal x squared,
and x3 equals x cubed, as just one example,
then we can turn Y into a multiple linear regression model.
So just keep that in mind.
Everything we're doing
with multiple linear regression models
you can apply easily to polynomial models.
It's quite common to rewrite our regression model
in matrix form, y = X times beta plus epsilon.
Here, y is a vector of our observations.
So these are our experimentally determined observations.
X is our model matrix.
This represents all of the inputs
that go into each of our observations.
You notice here,
and this is really important,
the first column of X is just a bunch of y's.
And the reason for that is,
if you take a look at our model parameter vector
which starts with beta 0,
when we multiply array X by vector beta,
the first element of each of our equations is beta 0,
and so the coefficient that we wanna multiply by beta 0
in our X array is simply one.
So it's really important to include this column of all ones
in your model matrix, X.
And then we also have a bunch of random errors,
epsilon one through epsilon n.
And n here is the number of observations.
Similar to what we did using simple linear aggression,
we're gonna calculate a sum of squares, L,
and we want to try to minimize that.
The least squares estimator, beta hat,
and beta hat here is a vector
that contains all of our estimates
for the parameters in our model,
is the solution for beta in the equations
where we take the partial derivative of our sum of squares
with respect to each of our beta terms.
And we wanna set those equal to zero.
So essentially, we've got a bunch of equations
that we're setting equal to zero.
Through a bit of calculus,
we can come up with something known as the normal equations,
shown here in vector form.
By the way, sometimes you'll see me write this
as X transposed, that's the same as X prime,
times X times beta hat equals X transposed
instead of X prime.
So these two are equivalent.
If you solve the normal equations for beta hat...
Remember, beta hat is our estimate
for all the parameters in our model,
and beta hat is a vector that equals X transposed X inverse,
we multiply that by X transposed times y.
And so this is how we can calculate
our regression coefficients in our model.
And again, you might see me write this
a little bit differently
in some of the workshops and presentations in this class.
This matrix equation provides the values
of the model parameters that minimizes the least squares.
In other words, this gives us the best model
for our experimental data.
So let's go revisit problem 12-9 in your textbook.
Again, we're trying to create
this simple multilinear regression model,
where we have four regressor variables, x1 through x4,
and we have 25 observations.
In a starter file called "Satisfaction Data - starter 1,"
I've got a bunch of our data here
and I've got some equations that we're gonna be using.
In the next screencast,
we're gonna use these other equations.
In this screencast, we're only gonna use
one of these equations, other than the model equation.
In a tab down here, I have the regression output.
So I've already performed this multilinear regression model
in Excel's regression tool,
and we'll compare what we get using the vector math
to our coefficients here.
So we're basically going to be calculating beta hat.
And I'm just gonna refer to this as beta,
the coefficient vector that leads to the best model.
In order to do this,
we first need to calculate X transposed,
that's what X prime is.
So I'm gonna go ahead and put in an Xt here, just a label.
Before we do that,
it's really important to do a couple of things.
First, I'm gonna name this output data,
I'm gonna name that y.
I'm also going to add in,
this is really important
if you're doing this using matrix math in Excel,
we have to insert a column
and we're gonna have just a bunch of ones.
So I drag that down.
And now what I can do is I can take that column of ones
combined with all of our input variables,
and this represents our X array,
so I'm gonna name that X.
Now I'm ready to calculate the transpose of X.
If you have Excel 2019 or Office 365,
it's really easy because you can just type in transpose of X
and you can press Enter.
Otherwise, if you have an earlier version,
you have to outline here a 4 X 25 region
and then you have to type in equal transpose of X
Ctrl + Shift + Enter.
And you're gonna have to do that
for all of the other calculations too.
But with Excel 2019 or Office 365,
they've made a couple of improvements
and you can just type in a single formula in a single cell.
So now what I can do, I'm gonna go ahead and name this,
Ctrl + Shift, right,
I'm gonna go up here and name this Xt,
that's X transposed.
Now, we need to calculate XtX, X transposed times X,
that's this little bit here.
And I can use the MMULT function in Excel,
Xt multiplied by X, press Enter.
So that's our XtX,
and I'm gonna go ahead and name this in the name box, XtX.
We now need to take the inverse of XtX,
so I'm gonna call this XtXinv.
That's simply the MINVERSE, matrix inverse, of XtX
and we get our inverse there.
Now what I need to do
is I need to calculate X transposed times y,
that's the right most part of our beta hat calculation.
I'm gonna put that over here,
so this is Xty.
Let's first go ahead and name this XtXinv.
Xty is just the MMULT, we're gonna multiply Xt with y.
I can press Enter.
And finally, I can calculate beta.
And this really is beta hat,
but I'm just gonna put it in as beta.
Let's first name this vector Xty.
Beta is simply the product,
so I'll do MMULT of XtXinv with Xty.
And when I press Enter, we get our coefficients.
So essentially what this means
is that's beta 0, that's our intercept;
beta 1, that's the coefficient here of our x1 term,
or our age;
and then we have beta 2 is negative .59;
beta 3 is .41;
and beta 4 is 1.31.
Let's compare these values.
So I'm just gonna go ahead and copy.
Let's go over to our regression sheet,
and I'll just paste them over here on the right,
and you see that they align nicely.
So this is how Excel's regression tool
is calculating the coefficients for a multilinear model.
In the next screencast,
I'm gonna show you how we can perform statistics
on these different parameters
so we can get a standard error for each of our beta terms.
And then later on,
you're gonna learn how you can use those standard errors
in calculating confidence intervals
and performing hypothesis tests.