Rick asked me about a chi-square that he conducted he described his findings as having a significance
of .052 (5.2%). He said "the manual that we received from our university says the following:
p < .05 = demonstrable difference between the population and the sample.
p > .05 = no demonstrable difference between the population of the sample.
When I asked my teacher for an explanation, she said if you strictly assume maximum 5%,
then at 5.2% there is no significant difference. If you are less strict and accept up to 10 percent
(or round to whole numbers) then there is a significant difference. In the meantime,
I don't understand anything anymore! Is my 5.2 significant or not? I understand your frustration,
Rick. Let's sort out this question of just what makes a finding statistically significant.
The confusion about judging when a finding is significant goes all the way back to an
early disagreement in the history of statistics. The answer depends upon whether you are doing
Ronald Fisher's significance testing or Jerzey Neyman and Egon Pearson's hypothesis testing.
The most famous of Sir Ronald Fisher's statistical illustrations was about a lady tasting tea.
A group of scientists had gathered one afternoon for tea. One of them, Dr. Muriel Bristol, insisted
that she could taste whether the tea was prepared adding the milk first and then adding the tea
or the tea first...
...and then adding the milk. Fisher, among the many who are skeptical that there really
was a difference in the taste of the tea, put Dr. Bristol's claim to the test. He asked the
kitchen to bring eight cups of tea: four in which the milk was added first, and four in which the
milk was added last. The tea would be presented to the taster in a random order. If there really was
no difference in the taste of the tea, then Dr. Bristol could be expected to correctly guess 50%
of the cups just by chance, so simply gets guessing 50% isn't good enough. So then,
here is the question: what percentage should she be expected to get correct before
you would trust that she could, in fact, taste a difference in the taste of the tea? 80% correct?
90% correct? 95% correct? Now, personally, I would say that if she got 90% of the cups correct,
that's good enough evidence for me that she can, in fact, taste a difference.
It would be really hard to get 90% correct just by chance. But how about you? How much evidence
do YOU require before you are convinced to reject Fisher's skepticism that there really
is no difference in the taste of the tea? If she correctly identified 95% of the teas,
would you be convinced? I would. How about if she correctly identified only 94.8% of the teas?
Would you now say that she had failed? I wouldn't. I'd say that she had performed the same.
If you're doing Sir Ronald Fisher's significance testing, using probabilities to establish the
amount of evidence against the null hypothesis, then p=.052 provides exactly the same amount of
evidence against the null hypothesis as p=.05 and p=.048. The only reason for the fluctuations
around that five percent cut off are random error and sample size. The underlying effect that you
are measuring is exactly the same. So, Rick, your .052 is just as convincing evidence against the
null hypothesis as .05... unless you're doing hypothesis testing. Let's pretend that we did
this tea tasting experiment with thousands of tasters and hundreds of thousands of cups of tea,
over hundreds of experiments, across multiple years. Now we want to identify which tasters can
taste the difference in the tea and separate them into one group based upon their effectiveness.
Now our null hypothesis is that none of the tasters can taste a difference between the teas.
We need to stipulate from the outset that we will only count as accurate those who
correctly identify 95% of the cups of tea. We will abandon our assumption that there is no difference
in the tea tasting ability only if the tea tasters correctly identify 95% of the teas
that they taste. We want to set a really high criterion for deciding. It would be a big deal if
I used 90 percent versus 95 percent because that 5 percent or 10 percent criterion for significance
is also our Type I error rate. When doing hypothesis testing, trying to determine whether
there is a significant evidence to reject the null hypothesis and accept the alternative hypothesis,
we may examine multiple studies perhaps thousands of data points. At 95 percent (p=.05) conceivably
we could falsely identify as many as five percent of our tea tasters as accurate just because they
guessed correctly by random chance. With a large enough sample, some people are going to get lucky
at guessing, even if they could not really taste a difference, and that would be a mistake,
a Type I error, saying that there is a difference when there is truly not, a false positive;
however, by strictly adhering to a significance criterion of 95 percent, we are limiting our error
rate to only 5 percent, and this is why we need to stick tightly to that 95 percent criteria.
The difference between .05 and .052 could be considerable over a long term hypothesis testing.
Certainly the difference between a 5% error and a 10% error could be substantial, perhaps
catastrophic, in the right setting. When we are doing what Jerzey Neyman and Egon Pearson called
hypothesis testing, we should strictly adhere to our established criterion of significance
and refuse to reject the null hypothesis on studies that do not exceed our .05 criteria.
So, Rick, your p-value of .052 is NOT statistically significant,
and there is not significant evidence to reject the null hypothesis in your study...
unless of course you were doing significance testing.
The fundamental reason for your confusion, Rick, is that the field of statistics
has conflated these two approaches from Fisher and from Neyman and Pearson into a hybrid,
"frankenstats" approach that is called NHST or null hypothesis significance testing
what Jacob Cohen called Statistical Hypothesis Inference Testing... make your own acronym.
NHST adheres to Neyman's strict criteria for significance in hypothesis testing when doing
Fisher's significance testing; furthermore, NHST largely ignores the effect of sample
size on significance, such that exactly the same effect size will be significant in a large sample,
even though it is non-significant in a small sample, leading to researchers concluding that
.052 is a failure and .048 is a success, when in fact, it's the same outcome. It leads people to
think that that .05 is somehow a magical criterion distinguishing truth from falsehood. It doesn't.
We are scientists, not magicians. So it's not your fault, Rick, that you're confused.
This statistical hypothesis inference testing is confusing. Your solution therefore is twofold:
number 1, always consider power when you design your test. Do a power analysis before collecting
data. How many subjects will you reasonably need in order to find statistical significance for your
effect size? And B, always report the effect size of your study when you report a p-value
for a test. I would suggest reporting the .05 to significance along with your Cramer's V effect
size for your chi-square. Follow those two rules and you will be doing statistics well.
Oh, and by the way, the real Dr. Muriel Bristol correctly identified all eight cups of tea
all right and we're out