Over the next two videos we are going to investigate the linear model.
The reason why we need to do this is after building the model we may have violated some
of the assumptions implicit in the interpreting confidence intervals.
We should examine whether there is evidence in the data
that those assumptions may have been violated.
We should also examine the data for outliers.
Outliers are often the most interesting data we have in our data set.
We can learn a tremendous amount from them so we should look out if we can detect these.
I will give more details about outliers in a coming video.
But I would like to say now that many of the assumptions
that are violated, are because of outliers.
In this video we will consider 2 assumptions and then 2 more in the next video.
In both these videos we will first consider how to detect when that assumption is violated;
secondly how to deal with it; and finally,
how to know that when we have dealt with it adequately.
One of the main assumptions is that our errors are normally distributed.
The standard error interpretation as well
as the confidence interval interpretation relies heavily on this assumption.
It is very easy to detect whether residuals are normal or not with a q-q plot.
We've shown before in the course, do not believe a histogram and do not plot the residuals
and try to pick this up with your eye to see if they are normally distributed.
A q-q plot is the only way to reliably detect normality.
If you detect non-normality in the residuals, we should try and correct it.
Let's take a look at a few options now.
Here is a regression model where we've tried to predict the price of a secondhand vehicle based
on the number of kilometres or mileage driven.
You would expect, that the higher the number
of kilometres driven, the lower the price of the car.
And so, the slope coefficient should have a negative sign at the very least.
And this is confirmed in the results from the model.
The standard error is quite high: $9,800.
This a very large prediction interval if we're trying to predict the price
of the vehicle based only on its mileage.
Now when we examine the residuals, we see strong evidence of non-normality here.
Some of the main outliers that we see are here in this cluster of points on the q-q plot.
Now as I've said, outliers are the most interesting data in the dataset.
When we examine these more closely, we notice that these cars
up there are all Cadillac convertibles.
The fact that they cluster here at such large positive residual error,
indicates that when we predict the price of those Cadillacs based only
on the kilometres driven, we are underestimating the resale price
and therefore getting a large positive residual.
There is clearly something about these convertibles
that indicates its price should be higher
than just the price predicted by the kilometres driven.
There's a premium price for this type of second-hand vehicle, and the residuals indicate
that the car price should be predicted in some other way.
Now after removing those outlier cars, we can rebuild the model.
This time our confidence interval is reduced for the slope coefficient,
indicating that we have a better estimate of it.
And our standard error has also decreased.
The residuals are still definitely not normal but they're more normal than they were prior.
We will see in a later section in the course that what we need to do here
in this instance is have a multiple regression model.
Intuitively we know that it's not only the mileage of the car
that is a good predictor of the price.
Other qualities, such as the number of doors, the number of cylinders, the sound system,
and whether it is a convertible or not;
all of these parameters will influence the price of the vehicle.
We will need multiple regression to do this, where we have multiple inputs into our model
in order to predict this output of price.
That will have to wait for a few more videos from now.
So we have seen one way to deal
with non-normality is to exclude outlier observations.
Severe non-normality in the residuals gives us an indication
that these residuals contain other information.
If there's no information left in our residuals we expect roughly normally distributed values.
The fact that we have this severe curvature in the q-q plot indicates there's more information
in the residuals and we should go try and extract that with our regression model.
So as we will see later, we will add terms to our model to get those residuals
as normally distributed as possible.
One other way we can make our residuals slightly more normal is
by applying transformations to the data.
A square root transformation, for example, on the y-value, might be useful to try
and skew the data to appear more normal looking and therefore fit the model better.
These transformations such as taking a square root or a log of the y-value are often done
to satisfy another assumption: that our model correctly matches what we expect from theory.
I will talk about that in the next video.
But here in this example we see evidence in the residuals, in the q-q plot,
that applying a transformation has made the residuals more normally distributed.
So those are the two ways to deal with non-normality: either by excluding data
from the model, or by taking transformations of the data.
Let's look at this next assumption of non-constant error variance.
This is the assumption that the residuals are normally distributed with the same variance
at all values of x. The easiest way to detect it is to look for a fan shape in our data
when we plot the residuals either against x or plot the residuals against the predicted value
of y. The reason why we look for a fan shape is the residuals is
because we expect the same variability at all levels of x, but if that variability increases,
for example, at high values of x, we will observe residuals as shown here
on the right hand-side, where the spread of the residuals is much greater at higher values
of the predicted value of y. And the reason why we can plot on the horizontal axis either the x
or the predicted value of y is because these two numbers are related
by the least squares formula.
And I've confirmed that here with this plot of the same data
where now I've changed my horizontal axis to be values of x.
So that's how we detect non-constant error variance.
We can deal with it, by using what is called "weighted least squares," where instead
of the usual objective function where each residual is weighted equally,
rather then we can go provide heavier weights for some of the residuals
and lighter weights for some of the others.
This will give more emphasis to some of the residuals and therefore will pull them all back
so they can have roughly the same contribution to the objective function.
A natural choice for the weights are those that are inversely proportional to the variance.
Residuals with large variability will get smaller weights,
so that they do not allow those residuals to dominate the least squares objective function.
R has some tools built-in that will do this work for you.
Now fortunately, the least squares model is fairly robust
to violations of this particular assumption.
We don't need to use weighted least squares every time we see a slight violation
of this assumption.
When the residuals show no visible structure in this plot of x against the residuals or y-hat
against the residuals then we know that we have adequately dealt with the problem.
In the next video we are going to consider 2 more assumptions: that of independence
in the data and the assumption of model linearity.