Hi! I'm Tom Reader, and I am going to tell you about p-values, with the help of this
bag of chocolates.
Urg! That was a raisin. Bad luck, I suppose.
I love these chocolates, but not all of them. There are three flavours: peanut, raisin and
solid chocolate. I love the peanuts and the solid ones, but frankly I think the raisins
are disgusting. The problem is you can't really tell the difference, so every time
you reach into the bag, you're risking disaster...
Oh bother. Raisin again. Double bad luck.
Oh well, best just dive back in.
Raisin again? Come on - so unfair!
What's going on here?! If I had a suspicious mind, I would start to think that someone
had tampered with my chocolates. But I'm not like that. I know that bad luck just happens,
right? This time, it will be a good one.
That's it! I've had enough with this.
I reckon someone has deliberately tampered with these. I did leave them lying around
while I was giving that stats lecture. Perhaps someone swapped all the nice ones for raisins?
Or maybe I am just having a really unlucky day?
So this kind of thing happens to people (and especially to researchers) all the time.
You collect some data - in my case the flavours of some chocolates - and you spot a pattern.
You ask yourself: does that mean something, or is it just the result of random chance?
Did I get five horrible chocolates in a row because I am unlucky, or is there something more
sinister going on? I could go on gut instinct, but surely there
is a more objective way of deciding what's going on? What we need is a way of quantifying
the chance that the pattern I have seen is just a coincidence - just the result of
random events. If we know that a pattern is extremely unlikely
to be a chance event, then that tells us something else is probably causing the pattern, and
that's when as researchers we tend to get particularly excited.
So, what is the chance of getting five rubbish chocolates in a row; if it's just an ordinary
bag of chocolates, well how do we calculate the chance? That depends on our null hypothesis.
In other words, it depends on what we consider "ordinary" to be.
Now, my chocolates here normally come in three flavours, only one of which I don't like.
And because we don't know any better, let's just assume that all three types are equally
common, and that each bag contains a random mix. That effectively is my null hypothesis.
So if I dip into a normal bag of chocolates, which hasn't been tampered with, the chance
of me getting a raisin should be one-in-three, or one third, or 33%. Statisticians might
say that the probability of getting a raisin by chance under the null hypothesis is one
third. Ok, but what if I get two raisins in a row?
What are the chances of that happening, if the bag is just a normal bag of chocolates?
Well, that's one third times one third, which is one ninth - or about 11%. That
means if I pulled two chocolates out of a bag one hundred times, I should get a pair
of raisins about 11 times, and the other 89 times, I should get at least one nice chocolate.
Ok, so in the real world, if you got two raisins in a row, you'd probably not be too freaked
out. And that's because you have a pretty good intuition for the fact that it's fairly
likely to happen by chance - 11.11% to be precise!
So what are these probabilities I'm talking about? There was the 33% chance of getting
one raisin, and then the 11% chance of getting two in a row. These values are what we call
p-values. P stands for probability; and specifically in this case we mean the probability of observing
a particular pattern in your data by chance, if the null hypothesis is true - in other
words, if nothing strange or interesting is going on (normally from a scientists point
of view). Ok, so what if you get five horrible raisins
in a row? Well, the chances of that happening under the null hypothesis are much lower.
One third times one third times one third times one third times one third is about 0.004,
or 0.4%. So p is 0.004. And that's less than a one-in-a-hundred chance of the event
happening just by chance if the null hypothesis is true. So yes it could happen, but it's
a pretty rare event. In fact, the pattern we've observed is so
unlikely to arise by chance that we might conclude that the null hypothesis is probably
wrong, and therefore that the chocolates are probably not "normal" chocolates. If it's
not random chance, the pattern we have observed must indicate something else is going on.
We need an alternative hypothesis. In my case, a possible explanation is that
someone tampered with my chocolates. Maybe they swapped all the nice ones for raisins,
or simply eaten all the nice ones and left the raisins behind. Or maybe some problem
with the manufacturing process happened and we've ended up with a bag full of only raisins.
It doesn't really matter. Lots of possibilities. Ok, so how is all this chocolate stuff and
probability relevant to the real world and to real research? Well, in all subject areas,
we want to be as objective as possible in the conclusions we reach from the patterns
that we observe. So if I am trying to decide if a new drug
is curing my patients of a disease, I need to know how likely it is that the patients
I've treated would get better anyway, just by chance. Would you really want us to spend
all that money giving an expensive drug to people if it turned out that the positive
effects that I had observed previously were simply the result of random chance?
What about if I want to know if the government's new policy to give everyone free chocolate
is improving happiness levels? I might survey a randomly selected group of people before
the policy was introduced, and then another group afterwards. I need to consider how likely
it is that the second sample of people I surveyed just happened to be of a sunnier underlying
disposition. All that talk of chocolate reminds me...