This video is all about checking whether data are actually normally distributed or not.
In the prior video we saw the many things you can calculate from a normal distribution.
So we always have to check if our data are normally distributed.
Here are 3 time-series plot, and below them are their histograms.
The question is, which of these plots are normally distributed?
We will come and answer that question at the end of this video.
To answer this question we need the cumulative and inverse cumulative distribution.
The cumulative curve was found using the pnorm(...)
function in R, when we know the z-value, and want to find the area.
The inverse cumulative distribution is the opposite.
We know the area, but we want to find the z-value.
This uses the qnorm(function).
The approach for testing for normality is easy.
We are going to create data that really does come from the normal distribution.
We are going to compare properties of the created data against our actual data set.
So two datasets are involved.
If our raw data has N observations, go and create N observations
from the normal distribution as well.
I am going to use this plot to help explain.
Create N evenly spaced points along the vertical axis here.
We don't start at zero or end at 1, because remember,
the vertical axis is undefined at those locations.
Start just a little bit over and end a little bit under.
Now take these N evenly-spaced points and move them across the plot and down again,
to locate where they are on the z-axis.
We can use the qnorm(...)
function to do that.
So now we have N values of z, from a pure normal distribution.
In this example shown here on the screen I have 10 values evenly spaced on the vertical axis.
And then we get our 10 z-values: -1.64, and so on, all the way up to +1.64.
Let's go find the z-values now, calculated from our raw data.
In the usual way, go subtract the sample mean, and divide by the sample standard deviation.
Since our raw data are not in any specific order, let's also go re-order our numbers.
Now we are ready to compare the z-values from our actual data
with the z-values from the normal distribution.
If our original data were really normally distributed, we should get calculated z-values
that are roughly the same as the z-values from the normal distribution.
We expect some noise and error, of course, so let's compare them graphically.
We should have the numbers lying mostly around a 45 degree diagonal line.
And in this example, they seem to be so.
The horizontal axis is the theoretical z-value and the vertical axis is the calculated z-value.
We can also show the results as here on the right.
It is the same information, just presented differently.
The first thing to notice is that the vertical axis of z-values have been unscaled
and uncentered, back to their original form.
That doesn't change the intention of the plot.
Secondly, we add a diagonal line, and the points should fall roughly along that line.
Lastly, with some extra theoretical work behind the scenes,
we can find bounds around that diagonal line.
These bounds are 95% confidence bounds, indicating that 19
out of every 20 points should fall within in the limits.
Even truly normally distributed data would have some points lying outside the bounds.
I recommend using the "car" library in R to create these plots for you.
Review the software tutorial on how to install a library and expand the capability
of R. Once you have the car library loaded, you can use the qqPlot(...)
function from that library.
That function does all the work we have described in the video,
and superimposes those calculated bounds.
So let's come back to the question we posed at the start:
which of these distributions are normal?
When we look at the q-q plot it's clear the first one is: all the points lie
within the bounds and are mostly close to the line.
The next plot can be considered normal, most of the data lie within the bounds.
However, we do notice that there is a deviation from the line at the upper and lower ends.
When we notice these patterns we say that the tails are heavy: there are more observations
in the tails than would be expected from a normal distribution.
When we go back to the histogram we can see that these bars look a little bit taller
than they would be from a regular normal histogram.
This final histogram might make you think that the data are non-normal,
based on the longer tail at the right.
However, the q-q plot shows that that part of the distribution is actually normal.
It the heavier left side that that deviates from normality.
That is the heavier end of the distribution, and we can start to see
that in the histogram once we have observed it in the q-q plot.
So we have have learned two things from this: firstly, a systematic way to check
for normally distributed data, and secondly, that our eyes do not tell us the truth.
We have to always verify our intuition with calculations,
especially in the area of data analysis.
There is no room for gut-feelings.