We concluded our lesson about
z-tests with the sad realization that the tests rarely get used in the real world.
In fact, they've been almost entirely replaced.
We will need to take a philosophical journey to get there, so we will begin like an Irish pilgrim to Santiago de Compostela at
St. James's Gate in Dublin, Ireland.
Since Abraham de Moivre's discovery
of the normal curve in the late 1600s to it's statistical
popularization by Sir Francis Galton and others,
statisticians have created and used tables of probabilities. These are the z tables that we studied previously.
When comparing the mean of a sample to the mean of a population,
18th century data analysts were comfortable using the normal distribution and a test that we would today call
the z-test, and truth be told the z-test works just fine as long as
you have large enough samples.
But not everyone has the luxury of using large samples.
Imagine that you were a brewer at Guiness Brewing Company studying the sugar levels in barrels of barley malt extract.
You don't want the malt to have too much sugar
because that would raise the alcohol content and with it the amount of excise tax paid on alcoholic beverages.
But you don't want the sugar to be too low because then the beer would not have enough alcohol,
dismaying your customers, and
most important of all, you don't want the alcohol content to vary batch by batch
so that neither your brewers nor your customers are sure about what they will get in their pint of Guinness.
You know that when you measure sugar in
degrees saccharin per barrel you can vary by half a degree
high or low and still be, shall we say, in the sweet spot.
But every ounce of malt extract that is used for testing is an ounce that is not available to sell.
So you don't really want to pull 30 samples to have a large enough sample size.
Sometimes you have to make decisions based on sample sizes as small as two.
Now this was the challenge facing William
Sealy Gosset, a master brewer and scientist at the brewery of Arthur Guinness and son in Dublin, Ireland.
Before I tell you how he solved it, let me explain the problem of using the normal distribution and the z-test.
Now this is the formula for a z-test. The problem was this error term, sigma.
This is a standard deviation of the population.
if you don't know the standard deviation of the population
you could specify it such as using Gossett's acceptable range of variability or
you could estimate it using the standard deviation of the sample.
Take what you know from the sample and apply it to the population no problem, right?
Well, it's not a problem when your sample size is reasonably large
because with reasonably large sample sizes the standard deviation of the sample is a fairly close
approximation to the standard deviation of the population,
however, Gossett was
calculating statistics on samples as small as two.
With a small sample size
the amount of error in your approximation goes way up and that increased error, as
Gossett observed, can have a noticeable influence on your calculations.
Our goal is to estimate the mean.
We're going to do this by collecting random samples making sure that each sample is the same size.
Of course, the mean of each sample is going to be off a little bit from its population mean.
So if we had plenty of time we could measure
thousands of samples each of the same size and then compute the typical amount that a sample mean is off from the
population mean, and this is called the standard error of the mean, what Gossett called the probable error.
We want the standard error of the mean to be as small as reasonably possible,
but instead of tediously carrying out this process thousands of times to estimate the standard error of the mean directly
we could use the same fundamental fact that we used with the z-test
that the standard error of the mean is determined by only two things:
the standard deviation in the population and
the sample size.
Let's see what happens to the average error when we consider the sample size.
So we have a population with a standard deviation of 10 and
this is the formula for the standard error of the mean.
We will consider all possible sample sizes from 1 to 100.
Now, divide the sigma of ten by the square root of each of these samples.
And now I'm going to create a plot of these data.
What we can see is that for a sample size of 1,
the average amount of error is the same as the standard deviation in the population, 10.
The graph clearly shows us that as sample size increases the
standard error of the mean
continues to decrease.
It decreases dramatically when sample sizes are small,
but then begins to flatten out after a certain number of subjects, somewhere around a sample size of 30.
So adding more subjects after 30
doesn't decrease the error all that much.
So to minimize the error of our estimations we should increase sample size.
Gossett wrote, "the greater the number of observations of which means are taken," in other words the larger the sample size,
"the smaller the error,"
referring to standard error.
So here is your first takeaway:
When it comes to sample size bigger is better in terms of the precision of our estimate.
But how big is big enough?
Well we know a sample size of 30 gives us minimal error without too many subjects,
but what if you do not have the luxury of increasing your sample size?
And this is where William Sealy Gosset saves the day.
We see that estimating with small samples
increases standard error.
Error is the difference between the observed value of a statistic
and its true value of the parameter. It's being used to estimate, in other words, it's inaccuracy.
The standard error of the mean is an estimate of the amount of noise in the test statistic.
The noise occurs because we're measuring a sample
instead of the entire population, and each sample has some random error.
Random error is the bouncing around that happens just due to chance factors that determine what
observations are in the sample.
However, error that consistently goes in one direction
is not random error. It's called bias.
Data analysts in Gosset's time knew that the absolute value of the z statistic
was biased when the sample size was small
because when sample sizes are really small then even our estimate of the
standard error has noise in it, and
this bias changes the shape of the distribution and with
it the standard that we are using to determine statistical significance.
You see when we do a z test using the normal curve we set a fence to the right of z = +1.96,
but Gossett's distribution is flatter, more spread out than the standard normal z-distribution.
So if we want to find cutoff scores that fence off the top 2.5% of Gossett's
distribution, then we will need to push this fence
farther out from the center.
The new critical value is greater than 1.96.
Whenever we adjust our estimates using sample size, or more accurately, degrees of freedom like Gossett did,
we will call this a chi-distribution now. I'll tell you more about degrees of freedom and t-distributions in a minute,
but first, here is your second takeaway:
the critical value cutoff for a t-distribution is always larger than +/- 1.96.
Unless, of course, the sample size is the same as the population.
The problem is that the distribution of the z-test statistics does not have a normal z-distribution
when the standard error of the mean is estimated based on a small sample size.
The solution? Gossett wrote a paper in 1904
describing the saccharin estimation problem.
It was called "The Application of the Law of Error to the Work of the Brewery".
By outlining a new family of distributions
which were flatter and more spread out than the z-distribution and by tying the amount of additional
variability to the sample size,
Gossett could eliminate the bias in the test statistic.
He did not change the calculation of the test statistic,
but instead came up with the proper distribution to model the statistic when the sample size was small, so
Gossett wrote up his initial work with some rudimentary statistical tables and initial explanations of his adjustments of
population parameters by sample size, and what was an
internal laboratory report at Guiness. This was 1904.
To better work out the math, however, in
1905 he bicycles to the summer home of Karl Pearson,
publisher of the journal
Biometrika, for a meeting that eventually led him to studying in Pearson's lab for a year.
By 1907
Gossett had worked out all of his tables concerning the probable error of the mean, but we now call the standard error.
Pearson reviewed Gossett's paper and agreed to publish it, but Guinness
Initially would not allow it.
In the end, however,
Gossett's article "The Probable Error of the Mean" did appear in Biometrika, but under a pseudonym
Student, and there are a variety of
urban legends surrounding why Guinness was reluctant to have its scientists publishing in academic journals.
One version says that Guinness did not want to acknowledge that there was any variability between their batches of beer.
This is unlikely because
the variability that Gossett was measuring was actually in barley malt extract.
Another version holds that Guinness did not want its competitors to know that they were using science and statistics and the t-test to
test the quality of their raw materials.
However, Guinness was a leader in applying science to brewing, so it's unlikely that many other brewers,
especially smaller breweries, or even using the scientific method, or would know what to do with a t-table.
Some versions even have Gossett publishing secretly to get around the prohibition, and add that Guinness never found out
until after Gossett's death.
So if Gossett published anonymously,
how do we know Gossett's name now?
According to an article by
statistician Harold Hotelling published back in 1930
answering the question, "Who was Student?",
"A large Dublin brewery," noticed he did not mention Guinness by name,
"adopted a rule forbidding its chemists to publish their findings.
Gossett pleaded that his mathematical and philosophical conclusions were of no possible practical used to competing brewers and
finally was allowed to publish them, but under a pseudonym," and here is the real reason, "to avoid
difficulties with the rest of the staff." So, yes,
Guinness did know about Gossett's publication.
And, in fact, encouraged it only in a way that did not cause other problems for them.
Now Pearson originally called this Student's z Distribution,
reflecting Gossett's adjustments to the normal curve,
but in 1912 a letter arrived from a mathematics student at Cambridge named R.A. Fisher
inquiring about a mistake in Gossett's 1908 paper about the probable error of the mean.
In his letter Fisher demonstrated that the total sample size should be reduced by 1 in
what would come to be called
degrees of freedom.
Now although Karl Pearson would later remark that "it made little difference
whether the standard error was divided by n, or the
rigorously correct n-1"
Gossett encouraged Pearson to publish young Fishers correction in
Biometrika. It was and with the revision to Gossett's
Student's z-tables using degrees of freedom, the new tables were called Student's t Distribution Tables
We now use
Student's t Distribution when we test hypotheses about population means using small sample sizes, in
fact, the t distribution and the accompanying t-test have become so ubiquitous
that tests of means are typically called t-tests
regardless of the sample size.
Student's t Distribution table also called the t-table
can be found in your course notes and
in any statistics textbook.
Although these days you should report precise
p-values instead of cut off criteria when you publish.
So here is your third takeaway:
t tests are conducted using n - 1 degrees of freedom.
Fisher would go on to write a definitive statistics text called
Statistical Methods for Research workers in
1925, in which he would dismiss "the traditional machinery of
statistical processes" for using infinitely large samples that were not practical in the laboratory.
In this he was tweaking Carl Pearson
who had earlier waved off the importance of Fisher's n - 1 correction saying that it made little difference
because only naughty brewers take n so small that the difference is not on the order of the probable error.