Your takeaways from our discussion about what statistical significance really
means should be that (1) there is nothing magical about alpha equals .05,
(2) a p-value does not tell us enough, and (C), we should report effect sizes and
confidence intervals with our findings. An effect size is a standardized measure
of the size of an effect. It measures how large the effect was, allowing for the
objective evaluation of the size of the effect, so effect size helps us answer
the question of whether a treatment had any practical usefulness - how big was the
effect? Cohen's D is the most commonly used measure of effect size for T tests,
So how is effect size different than significance? Well statistical
significance tells us that differences between means were not due to chance or
at least probably not due to chance. Statistical significance measures the
probability that we would find the results that we found if the null
hypothesis was true - which we know that it never truly is - and making decisions
based on p-values can be very misleading. Consider the following example from Rosnow and Rosenthal. Smith conducts an experiment for the sample size of 80
comparing the effects of two leadership styles on productivity. Smith finds that
style "A" is better than style "B". Jones is skeptical because Jones created style B.
So he replicates Smith's study with a sample size of 20. Smith's original study
had a T of 2.21, 78 degrees of freedom, (P < .05).
Jones reports a failure to replicate. His T value was 1.06, 18 degrees of freedom, (P > .30).
It is true that Jones did not replicate Smith's p-value,
but let's consider the Cohen's D effect size. Smith's d was a 0.49.
Jones's d was a 0.47. The magnitude of the effect obtained by
Jones was almost identical to the effect
obtained by Smith. Jones had found exactly what Smith had
found, even though the two p values were not very close. Because of using a small
sample size, Jones's study was not very powerful, but the non-significant results
implied that there was no real effect. Not true!
Smith's power because of his sample size of 80 was more than three times greater,
and the effect size shows us that each study found the same effect regardless
of p values. Jacob Cohen gives us conventions for interpreting Cohen's D
effect size: small, medium, and large. And already you may be questioning where do
these numbers come from? I mean, didn't Fisher use 0.5 because it was
convenient? Why should we accept these suspiciously round numbered conventions
from Cohen? Well if you think these conventions are arbitrary, Jacob Cohen
would agree with you. He wrote, "All conventions are arbitrary.
One can only demand of them that they not be unreasonable."
And then Cohen went on to explain why these conventions are reasonable, or at
least, useful. They are based on probabilities. Consider that we are
comparing two distributions representing "Group One" and "Group Two". When the mean
difference is 0, and therefore d = 0.00, there is
100% overlap in the distributions. Each mean is higher than 50% of its own
distribution. That is what the mean is - the point that is higher than 50% of the
lower half of the distribution. So because there is 100% overlap, each mean
is also higher than 50% of the other distribution, but as one mean starts to
diverge, the mean difference - the difference between those upright bars -
becomes greater, and so d becomes larger. Remember that d is the standardized
difference between the means. So as the means diverge, there is also less overlap
of the distributions. So for example, when d = .01, there is a 7.7% non-overlap of the two distributions.
They overlap except for about 8%.
Small effect sizes can be measured, but their differences are not
visible to the naked eye. So when d equals 0.02,
14.7% non-overlap, and mean Two is higher than
57.9% of the first distribution; remember it started at 50%.
Cohen says that in new fields of study, effects are likely to be small, because
the measurements are imprecise. So if you are researching something that is not
well understood, assume that you are looking for small effect sizes on the
magnitude of 0.2 or 0.3. Cohen says that there is a small effect size difference
between the average heights of 15 and 16 year-old girls, so if you were to look at
a classroom of 15 and 16 year-old girls mingled together it would be very
difficult to see who is who.
A medium effect size is large enough to be seen with the naked eye, so when d
equals .05, there is a 33% non-overlap. Mean "Two" is higher than
69.1% of the first distribution there is a medium effect size for height
difference between 14 and 18 year-old girls so if you have a classroom of 14
and 18 year-old girls mingled together you should be relatively easily able to
tell who is 14 and who is 18 by their height. Cohen says that there is a medium
effect size difference in IQ between clerical and semi-skilled workers, or
between professional versus managerial occupations.
A large effect size is easily seen. When d equals 0.08, there is a 47.4%
non-overlap, almost 50%, and mean "Two" is higher than 78.8%
of the first distribution. There is a large effect of height difference
between 13 and 18 year-old girls. Look at a classroom of seventh graders mixed
with high school seniors, and you should have no trouble separating them. Cohen
describes a large effect as the difference in I.Q. between Ph.D. holders and
typical college freshmen. If there is a large effect size in your data you
should be able to see it by looking at the raw data. Another researcher expanded
Cohen's conventions to include "very small" at .10, "very large" at
1.20, and "huge" at 2.0. I hope that you can see that Cohen's
conventions, while still arbitrary, are not at all unreasonable, and don't get
hung up on categorizing a difference of d of 0.78 as medium or large. It is what
it is, and it can be directly interpreted. When we know the effect size, we can
figure out how many people that we would need in our experiment to find an effect
of that size, and this is called a power analysis. It helps us choose a large
enough sample that we are likely to find any differences that exist, and to not
miss effects and to waste time and money by having too few participants. Recall
the Smith and Jones leadership study. If we do not know the effect size, we could
use our literature review to find a similar study and calculate the effect
size, or we can make an educated guess. Cohen says that this is the advantage of
using d for power analysis, that it need not be calculated. It can be posited
directly. This is actually the strength of using d.
What do you anticipate the differences to be? If you think that the differences
will be small, then pick a d around 0.2 or 0.3 if you think the differences will
be large, pick a d near 0.8 for medium go with 0.5. Then you can conduct your power
analysis and discover the power of your test to find effects. So why should we
report effect size? Well several reasons, first, and certainly not the least
important, is that the APA recommends it. The APA recommends that all published
statistical reports include a measure of effect size, even for statistically
non-significant findings. Well why else? In the video about what statistical
significance really means, I showed an example of a research project using a
certain kind of padding for hip replacement patients, and how it reduced
the incidence of bedsores to zero. I pointed out that statistics were not
necessary to interpret this result. We can all see that the incidence has
dropped to zero. So in cases like this, where you're not trying to generalize
from a sample to a population, then null hypothesis statistical testing is
unnecessary. If the research is exploratory or focused on your
particular project, don't bother with p-values, just report
the effect size. Now when the sample size is small, null hypothesis statistical
testing confuses non-significance with the absence of an effect.
Remember how Jones suggested that failure to replicate Smith study implied
that Smith had been wrong in his criticism of Jones's program, and
statistically non-significant results can be very important in the real world. When
sample size is small, and the results are non-significant, that does not
necessarily mean that a treatment was wholly ineffective, especially when the
direction of change is in the direction that you predicted. In
other words, if a drug is halting deterioration in patients but the t-test
is not statistically significant, the drug is still promising. Report the
effect size so that we know what kinds of improvement the drug is making. And
finally, statistical significance is a function of sample size. In fact, simply
increasing sample size can make non-significant findings shift to
statistical significance. Effect size helps us to interpret probabilities in
light of sample size. So if you are doing a research project with 2,000 middle
school students about their attitudes on some educational program they have just
received, every test will be statistically significant, even if the
effect size is trivial, because you have an over-powered study. Because even the
tiny inconsequential shifts will be statistically significant, report the
effect size so that we know how the education program is actually working. In
short, there is no reason why you should fail to report an effect size.