As we learn about hypothesis testing, I want to warn you about a
misunderstanding that is dreadfully easy to tumble into. I want to be sure that
you do not make this common mistake. Very simply, here is what you need to know.
There is nothing magical about p = 0.05. I have known even Ph.D. level
researchers who labor under this misapprehension, and I want to disabuse
you of it before we go any further into learning about null hypothesis
statistical testing. As we work through our five steps of hypothesis testing, we
set an alpha level at step three, and I have told you to use a two-tailed test
with an alpha of 0.05 unless you are instructed otherwise. If we calculate a
t-value that exceeds our (p = 0.05) critical value then we reject the null
hypothesis. But you would be wise to ask Why do we use .05 as our
significance value? Let's start by thinking about what .05
really means. We sometimes talk about tests being significant, but what we
really mean is that the measurements are statistically significantly different.
But you know, there are different kinds of different and if you think about it,
everything is different even things that we think are the same, are different.
Two seemingly duplicate light bulbs are different. Identical twins are different.
Two spark plugs are actually different. Two cars that roll off the assembly line
right after each other are still different, and the means of two samples
drawn from the same population, like identical twins, are different.
Go out to enough decimal places and everything is different. On the other
hand, many of these differences are too small to matter in the real world, so
knowing that two samples of the same size drawn from the same population are
going to be different, we want to know - are they really different? Are the
differences greater than what would be expected by chance or measurement error?
Are they statistically significantly different?
So how different do the means need to be for us to conclude that they were
statistically significantly different? Ah, that is the question. As a field, we have
settled on a significance level of 0.05 or 5%, or 1 in 20, or 19 to 1 odds.
Alpha equals .05 means that only 5 times out of 100 would we find this outcome if the
null hypothesis was true. Remember, we start with an assumption, the null
hypothesis, that the differences between two means is zero. But if we do a t-test,
and the results that we observed were so unlikely, that only 1 time out of 20
would we find these differences by chance, then we say that the differences
are statistically significantly different. The probability of getting a
difference as extreme or more extreme than the difference that we observed is
5%. So here are some things that you need to understand about the meaning of
significance. Number one: significance is dichotomous. Your alpha level is a fence
that you set before you begin an experiment when you say to yourself: "If I
find differences that cross this threshold, then I am going to believe
that those differences were not due to chance." The fence is your alpha level.
Inside the fence is non-significance, only differences that cross that fence
are significant. So if you are in a field inside of a fence with a bull the only
safe place to be is outside of the fence it doesn't matter how close to the fence
you are, if you are still inside the fence with the bull, you are still in
danger. Only when you get over the fence, outside of the fence, are you safe.
Outside of the fence is statistical significance. If your (p = 0.06),
you are still inside the fence and still non-significant. Only when
(p < 0.05) is the test significant. Significance is dichotomous,
all-or-nothing. Number two: there is no such thing as more significant.
There is no qualifier for significance, just like you can't be mostly pregnant
or partly dead. Differences are either significant, or they are not, so imagine
that you are comparing two tests. The first test is significant, at a z of 2.01,
p < 0.05, two-tailed test. The second is significant at z of 8.37, p < .0001,
significant at z of 8.37, p < 0.0001, two-tail test. Is the second test more significant than the first?
No. It is only a less likely result. There is no such thing as more significant, or almost significant, or marginally significant.
Number three: Statistical significance is not comparable.
Imagine that you were comparing two treatments to a control group.
The first treatment did not produce a statistically significant difference, p = .08.
The second treatment did produce a statistically significant difference, p = .02.
Should you conclude that the second treatment was more effective than the
first treatment because it was significant and the first one was not?
No. The difference between significant and non-significant is not itself
statistically significant. We should also avoid something called p
hacking or data mining where a researcher runs a huge number of
statistical tests, and then looks for the ones that came out significant only
those with p < .05 get reported, even though the statistically
significant results might not be statistically significantly better than
their non-significant brethren. Number four: there is no such thing as almost
significant. Think of null hypothesis statistical testing like a horse race.
You pay your money and you place your bets, and you bet on p = .05.
Then you run your experiment and only probability smaller than .05
are significant. That is how the game is played. If your p = .06,
the difference is non-significant. Not marginally significant.
But don't be too disappointed. If you think that p = .06 is
pretty close to significant, in fact, here are some descriptions of findings from
actual peer-reviewed journals: "barely missed the common acceptable
significance level", "at the cusp of significance", "approaching significance", "on
the very fringes of significance" "trending toward significance". There is no
such thing as marginally significant. It is like saying that your favorite team
lost by one point, so they almost won, they were marginally winners. Or you
finished fourth place so you were almost an Olympic medalist. Almost means
that it didn't happen. Almost significant means that it was not significant.
There is no almost. Remember, you are not required to use .05. There are
no statistics cops hiding behind a probability speed limit sign saying do
not exceed .05. Most researchers use .05, but you could set an
alpha level to some other level. If you had a small sample size, and wanted to
use p = .10, make a case for that in your methods section. You
should probably tie it to a power analysis. You can still get published.
These were all published studies, but set your critical value a priori before you
run the experiment for statistical or theoretical reasons and then stick with
it. Don't be the researcher standing at the
finish line with a losing racing form saying forlornly, "My horse almost won, he
was approaching a win." Well at this point, maybe you're beginning to feel a little
queasy about this rigid approach to statistical testing, I mean, if the t-test
is p =.06 and the direction of change was the direction
you've predicted, and if you'd only added a few more subjects then the test would
have been p < .05, doesn't that mean something? Well if you're thinking
like this, good for you that is exactly what you should be considering, and here
is why. The null hypothesis is always a lie.
The null hypothesis is our starting assumption. the null hypothesis says that
group means are the same, and the difference between them is 0, or the
correlation in the population is 0 or something like that. But that is never
true. We already know that everything is different. We know that the null
hypothesis is wrong from the outset. Pick any two samples, measure them finally
enough, go out to enough decimal places, and those two means will differ. When we
do null hypothesis statistical testing, we are essentially saying "I'm gonna
pretend that something is true even though I know that it's not and then ask
how likely is it that I would find this difference if this assumption that I
know is false was actually true." p equals .05 is arbitrary. Have you ever wondered
just where we got this P equals 0.05 standard in the first place? It was
suggested by Ronald Fisher who wrote, "It is usual and convenient for
experimenters to take 5% as a standard level of significance. . .". So when we are
doing statistics we need some agreed-upon standards for comparison.
The alpha of .05 is a convenient standard but even Fisher wasn't adamant about it.
The value for which P equals .05 or 1 in 20 is 1.96, or nearly 2. It is
convenient to take this point as a limit in judging whether a deviation is to be
considered significant or not. Deviations exceeding twice the
standard deviation are thus formerly regarded as significant.
Non-significant results can be very important. When differences are not
statistically significant, it doesn't necessarily mean that the treatment
didn't work, only that we cannot rule out chance as an explanation with 95%
confidence. Statistically non-significant results can still be very important in
the real world, especially if they were gained using a small sample size. This
happens a lot in medical research, where you have a very limited number of
patients, or a rare disease. If most of the patients that you are treating with
an experimental drug are improving, and no one is getting worse, or has failed to
respond, but the t-test was not statistically significant, so what! The
drug is working. p < .05 doesn't matter.
Reality is telling you this drug works. Leave the statistics. Take the cannoli.
avoid what John Tukey called statistical sanctification, that is, using statistics
to prove something that can already be seen with the naked eye. This is a poster
called "the use of an absorbent soft silicone self-adhesive bordered foam
dressing to decrease the incidence of sacral h apu and hip fracture patients."
It just means that applying a special pad to patients backsides eliminated bed
sores. Now, I love this poster because there are no statistics and they don't
need any. Everything you need to know is right here before the study six of 130
patients develop sacral h apu. During the study, in which all patients receive the
special sacral padding, how many developed bed sores? None. Zero. And three
months after using the same protocol? Still zero. You don't need statistics. You
can see that this is working. We used to have a problem with bed sores, then we
used this dressing, and now we don't. Statistically significant results can be
unimportant. Statistical significance is not the same as practical significance.
Significance does not tell you whether a difference was large or small, important
or meaningful. A difference can be statistically significant and yet
practically trivial. So a gas station across town is selling gas for two cents
a gallon less. That is a real difference. It is not due to chance, but is it enough
to get you to drive all the way across town to fill up? Of course not. But if the
gas was 50 cents per gallon less, that might get you motivated! Both of the
differences are real, but only one is big enough to matter in the real world. When
we consider differences, we should also consider effect size, practical
significance. And so love your p = .06.
Maybe that p of .06 really does have something useful to tell us.
Maybe the direction of change also counts for something, and we should
demand to know precise p values, and not be satisfied with p less than 0.05. We
want to see confidence intervals, and the effect size as well, if we are to be
persuaded. Pay attention to the real-world implications of research and
do not be dogmatic about p equals .05, or as Rosnow and Rosenthal said it,
surely God loves the .06 nearly as much as the .05.,
arguing for viewing the p-value as a continuous measure of the strength of
evidence for or against the null hypothesis, and you know who would agree
with them? Ronald Fisher, the guy who gave us that .05 standard. Fisher
thought that the p-value could be interpreted as a continuous measure of
evidence against the null hypothesis. He never presented 5% as some magical fixed
value at which the results suddenly become significant. A p of .049
and .051 constitute an identical amount of
evidence against the null hypothesis. So remember: statistics don't prove anything
statistics provide evidence. We are scientists and science is not a religion
We have no need for dogmatism. We are seeking clarity and understanding of the
world around us, and null hypothesis significance testing is just one of
several tools that can help us to figure out how our world works.