- [Charlie] In this screencast
you're gonna learn all about probability plots.
In exploring the nature of a set of data,
an early question arises as to whether the data
appeared to have come from a population
that is described by a particular distribution.
For example, is the normal distribution appropriate
for your data or is some other model more appropriate,
like the gamma distribution
or the exponential distribution
or all of these other distributions that we've talked about?
Which distribution does my data likely fit?
In this class in the last couple weeks,
we've already seen a couple examples
of how we can fit data to a distribution.
Here's an example where we had a thousand
different measurements collected and we fit the data
to a nice gamma distribution.
We also did something similar
with the male and female heights in the class,
we made a histogram.
But as you see, as the number of samples
starts to diminish here we have 48 female measurements
and 58 males, the number of bins that you use
in histogram starts to decrease.
And so as N decreases,
you can't really see the distribution as much
and you don't really know, you're not as confident
as to which distribution your data follows.
Today we're gonna be talking
about some measurements of brown trout in the Grand Canyon.
And here I've just created a quick histogram
and we only have 10 measurements here.
Sometimes your sample size is quite limited
and I just have a histogram here with only 10 samples.
There's no way that you can make any sort of inferences
on the type of distribution simply because
you only have three bins here.
So for large samples,
a histogram can be used to fit
various distributions with ease.
But for smaller samples, this becomes quite difficult.
Although there are different types
of quantitative calculation methods
for addressing this question,
the typical approach is to use something known
as a probability plot.
So today I'm gonna explain what a probability plot is,
how you can generate them,
and then the easiest way to generate them
is in a software tool such as Minitab.
So let's look at some data.
We have 10 samples, the length of brown trout
in the Grand Canyon.
I've got a friend who is an ichthyologist,
that's a fish scientist in the Grand Canyon,
and he basically just goes around and collects fish
and does studies on them.
He sent me some data.
This is the observed length in millimeters.
We can compute some sample statistics,
the average and the standard deviation.
And in order to figure out what type of distribution
your data follows, you have to start out with a hypothesis.
So we're gonna start with the hypothesis
that the length of brown trout
is a normally distributed variable,
and then we're gonna see how well these data
fit a normally distributed variable.
If this is true, so if the data follow a normal distribution
then our probability density function
should be the normal distribution
centered about 152.6
with a standard deviation of 24.9
as shown here in this plot.
Now, if we were to take 10 samples
from this population
that are characterized by this distribution,
the question arises approximately where should they lie?
And you guys all know,
because of how these probability density functions work,
if I had 10 samples, each of these regions here
separated by the dot lines represents a probability of 10%.
So this first region here is 10%,
this second region here is 10%, and so on.
If we have 10 samples, we would then expect to see
one fish in each of these 10 different categories.
If we calculated the cumulative probabilities
of each of these red lines,
now note that I only have nine red lines
because we're splitting this into 10 equal area sections
so I've got the cumulative probability
of all the dotted lines.
In fact, if I wanted to put 1.0,
that basically doesn't exist because it's infinite
and you would have to go all the way down this curve
on the normal distribution to get exactly
a 100% by definition.
So these are the approximate X values
that are corresponding to those red lines.
If we plotted this on a cumulative frequency diagram,
capital F of X,
we can see kind of the same sort of thing
where we have equal increments going on the Y axis here
and these are the X values corresponding to that.
Now, let's revisit the fact
that we can't have 1.0 cause that's really infinite.
What's typical is to sort of think
of these different increments on the Y axis as bins
and we sorta wanna take the bin center.
So we're gonna look at these red markings here.
For us, that would be 0.95, 0.85, 0.75 and so on.
That's what we actually do is we sort of back off
and we calculate the centers of those quote unquote "bins"
represented here by these blue lines,
and from those we can then calculate adjusted X values
that correspond to our predicted lengths in millimeters.
And these are known
as the expected cumulative probabilities.
Again, instead of using 0.1, 0.2 and so on,
we back off and we kinda do a bin center,
and that means we get 0.05 all the way up to 0.95
using the NORM.DIST function
we can calculate these X values in Excel.
How do we calculate these expected
cumulative probabilities in general?
The cumulative probability expected
is gonna be i minus 0.5.
I is just the index going from one up to n,
and then we divide that quantity by n.
We saw that when n equals 10 we get the following,
but for example, if we had n equals 23,
then the expected cumulative probability
would be as shown here.
And these are just calculated from this simple formula
over here in the middle.
The old way of doing these probability plots
was on something known as probability paper.
So you could actually buy these pads of probability paper
and this is just showing one example.
The Y axis over here is not linear.
It's almost like a log plot
but it actually goes up and then comes back down.
So the top and bottom of the Y scale here
has very small increments
and then they increase towards the middle
and then decrease as you get down to the bottom.
The old way of doing this
would be to first sort your lengths.
So we would first have to sort the lengths
in ascending order,
and then you would plot these on probability paper.
So over here on the right I have sorted our links
from smallest to largest,
and those correspond to our cumulative probabilities,
0.05 up to 0.95.
On normal probability paper, we would then plot,
we would plot these pairs.
For example, we have 105 that occurs
at about 0.05 frequency.
We have 134 that corresponds to 0.15,
and then we can keep going.
Now on probability paper
if your data appear to be linear like they do here,
the conclusion that can be made is that
the data, these 10 points that we have,
are normally distributed, all right?
So it's a somewhat qualitative method
where you just kinda look to see if these data are linear
on normal probability paper.
There's also exponential probability paper,
there's Weibull probability paper,
there's all sorts of probability paper
for different distributions.
If we plotted our data on exponential probability paper,
you can see that it might look a little linear
in the middle here.
In general, it's not very linear
because we have this point down here and this point up here.
So this appears to be very non-linear
on exponential probability paper.
The conclusion that can be made here
is that these data do not follow a exponential distribution.