It is very important that we understand what is meant by the phrase
"statistically significant difference." We know that significance is indicated by a
p-value less than .05, but before you go getting all excited about the
size of your outcome's p-value, let's be clear about what significance really
means. When you find a statistically significant difference, does that mean...
that there is a ninety-five percent chance that the alternative hypothesis
is true? That this finding will replicate ninety-five percent of the
time? That if the study was repeated, the null hypothesis would be rejected
ninety-five percent of the time?
No. It means none of those things.
So, what does p < .05 really mean? The p-value is the probability of
finding these results if the null hypothesis is true. A statistically
significant difference p < .05 means that the outcome that
we observed was unlikely to have occurred by chance.
What p < .05 actually tells us is that the results were
statistically significant, but it does not tell us whether the results are
practically significant. In other words, do these results really matter in the
real world?
Statistical significance tells us that the treatment had an effect...most likely.
But it does not tell us how large that effect was. And finding a p-value that is
very small, say less than . 001, does not make the results "more
significant." There is no such thing as being "more significant." The results are
either statistically
significant, or they are not. In order to know how large the effect was, we need a
measure of effect size. We may have found in effect, but the effect was
really small and of no real-world importance. Measures of effect size will
be things like Cohen's d, r squared, or eta squared. Statistical significance tells
us to reject the null hypothesis, but it does not prove that the alternative
hypothesis is correct.
Whenever we make a decision, we could make a mistake. As we learn more about
hypothesis testing, it will be important that we not rely exclusively on p-values,
but also consider things like effect size, confidence intervals, and the power
of our tests. We need to think about what Jacob Cohen wrote in an article titled
"The Earth is Round, p < .05"
What you want to know when you do hypothesis testing is
the probability that the alternative hypothesis is true given the evidence.
Did the treatment have an effect? What you get, however, is the probability of
obtaining this evidence, assuming that the null hypothesis is true. If the
treatment really did work, then how likely is it that we would have found
these results?
Well, the big problem with this approach is that in reality, the treatment always
works.
It always does something, even if that something is trivial. And means are never
truly equal. If you go out enough decimal places, they will diverge. So the null
hypothesis is never actually true.
Furthermore, improbable things happen all the time. In fact, if you do enough tests
with the same ineffective treatment, some of those means will, in fact, be
statistically significantly different.
p < .05 means that five times out of 100, one time out of 20,
we will make a mistake. We will mistakenly identify an effect as
significant when, in reality, it's not. We make a Type I error.
This is also called "the dance of the p-values". And finally, statistical
significance is a function of sample size. With enough people, every comparison
will be statistically significant, even if it is practically meaningless. We must
consider the power of the tests that we are running. Statistical significance
tells us that the differences that we found are unlikely to be due to chance,
or luck...but possibly not.
And even if the differences are real, they may not mean anything in the real
world. And this is why, whenever we run a statistical test, we should also compute
a measure of effect size.