Hello, I am Tom Reader, and in this video I am going to explore the difference between
effect size and statistical significance. We're going to look at how to interpret
the results of a study designed to test an interesting hypothesis: you have collected
some data, run a suitable statistical test, and now you want to tell the world what you've
found. It's all about the p-value, right? That
tells us if the effect we are looking for is significant. It tells us whether or not
to reject the null hypothesis. So tell the world your p-value, and your work is done?
Well, no, it's not quite that simple... I can't really imagine what it would be
like to be bald but, for your benefit, in this video I am going to make that huge mental
leap. Let's just imagine that I'm bald, and I am looking for a cure.
I have to the side of me two potential miracle cures for baldness. Here's the first one -
Furball Express: get hairy fast! Scientifically proven to give you back your luscious locks.
It looks good. And the second one - Toupee Away: say goodbye
to wigs and hairpieces, say hello to hirsute happiness. Clinical trials show a miraculous
improvement in hair growth. Wow, that looks pretty fantastic too.
But I'm no sucker. I know not to trust the marketing spiel. I want to see hard evidence
- data from an experiment - before I believe the hype.
So let's read the fine print. You know there's normally an asterisk next to any claim about
a treatment that is "proven" to have the desired effect, referring you to some evidence,
buried somewhere in a scientific report. Let's imagine we manage to get our hands on the
reports for these two "miracle cures". For Furball Express, the report looks promising.
The headline says: Volunteers treated with Furball Express were on average 50% hairier
than a control group after just two weeks.
For Toupee Away, the report says A clinical
trial with 1000 balding volunteers showed a statistically significant increase in hair
growth after six months' treatment with Toupee Away.
These both sound quite compelling, but which is better?
The report for Furball Express
suggests I am going to see a huge increase in hair growth in just two weeks. But when
I read on, I find that it says: We chose 4 people at random, from a group
of 8 balding male volunteers, for daily treatment with Furball Express for two weeks.
A control group of 4 men received daily treatment with an equivalent lotion, but lacking the active
ingredient. At the end of the trial, we measured the length of the hair of each volunteer at
a random point on the head in cm using a ruler.
Now that actually sounds a bit rubbish really
- they've only got four people in each group, and the measurements are going to vary
hugely depending on where on the head they selected to measure. And a ruler doesn't
seem like a very precise tool for measuring hair growth.
And they didn't even measure
the length of the hair of the men before the trial started, so all the variation they see
at the end - well, that could just be caused by random differences between individuals
in how hairy they were to start with! Crucially, there's no statistical test here.
We need to calculate the probability that the difference they have observed (50% longer
hair in the treatment group) might have arisen just by chance. It sounds like a big effect,
but if we have got no p-value to evaluate the likelihood that it could just be coincidence,
are we sure it's real.
Ok, so what about Toupee Away? Here's what
the report says: Half of the 1000 volunteers were randomly
selected for daily treatment with Toupee Away for six months, while the other half received
daily treatment with a control substance. 10 hairs were taken at random from the crown
of the head before and after treatment, and measured under a microscope to the nearest
micron. The increase in average hair length for volunteers given Toupee Away was 5 microns
greater than in the control group. A t-test showed that this difference was statistically
significant (p < 0.01). That sounds more convincing doesn't it?
They've got a nice big sample, they've taken really careful measurements before and
after treatment, and then they've done a statistical test which gives a clear conclusion:
the probability of seeing these results by chance is tiny (less than one in a hundred),
so Toupee Away significantly improves hair growth.
Hang on a minute, and this is a big BUT: hair growth only improved by an average of 5 microns
in six months - that's 5 one-thousandths of a millimetre! That's a very small effect,
even if it is significant. At that rate, if I was bald (and frankly I do find it a bit
hard to imagine what it must be like) I would have to use Toupee Away for 200 years for
my hair to be 1mm longer than it would be if I didn't bother.
So, I think that the correct conclusion here would be to steer clear of these two "miracle
cures" altogether. We have no idea whether the supposed benefits of Furball Express are
real, and we have really good data showing that Toupee Away doesn't improve hair growth
meaningfully.
These examples show that to make sense of
a scientific study which seeks to test a hypothesis, a p-value is not enough. A p-value tells you
whether there is a significant effect, but it doesn't tell you how large or important
that effect is. A large, powerful study can detect really
small effects. A significant effect which is small might be really important - a 2%
reduction in deaths from malaria would save thousands of lives around the world - but
it might be essentially irrelevant, as it would be in the case of Toupee Away.
So if you're reporting the results of a study, what should you do? Well, you need
a proper test of your hypothesis, and you need to report the p-value. But you also need
to describe the size of any effect you have detected. You can do this in various ways.
For a difference between two groups, you might express the difference between the means as
a percentage or proportion of the control or reference group mean. For a linear relationship
between two continuous variables - x and y - you might use something called "r-squared",
which tells you what proportion of the variation in the dependent variable (y) that is explained
by variation in the independent variable (x). Either way, it's your job to make sure that
the world understands what your results can and cannot say about the effect you're interested
in.