The p-value
In publications, the statistical significance of a study is usually expressed as the p-value. It’s
the result of a study that’s interpretable, as we’ll describe later on in this episode.
Even if you haven't heard much about the p-value, you probably know that smaller
p-values are better than larger ones. Let's find out why this is the case:
The p-value indicates the probability of finding the observed or more extreme events when the null
hypothesis is true. So what could such an event be? Let’s use an example to demonstrate this:
Assume you roll a die five times and get the number 4 on three occasions. Now, you’d like to
check whether the die is normal or whether it’s biased so that the number 4 occurs more often.
For this test, the p-value wouldn’t only indicate the probability of rolling 4
three times but also of rolling a 4 on four or five occasions. Both events are more extreme
than rolling the number four three times. A smaller p-value implies that the observed
outcome is unlikely. A larger p-value indicates that the outcome is not as unlikely. So what
implications does it have for our study: Let’s go back to our episode on the null hypothesis.
We learned to initially assume that the null hypothesis is true in a study. Our
study data should then ideally refute the null hypothesis. Plotting the possible study
results yields a Bell curve in which we mark our previously determined result as a dashed line.
In this context, we define a significance level on whose basis we can reject the null hypothesis:
Beyond this value, we consider the probability of committing a type 1 error too high, so the null
hypothesis is accepted. As a result, there are two rejection regions in the curve. So, from examining
the significance level in the last episode, we know that our result isn’t in the rejection
region, and the study is not significant. Let's still examine the p-value of the study as well:
So, according to our definition, where would the p-value appear in this curve?
At the beginning of this episode, we stated that the p-value indicates the probability
of observed and more extreme events occurring. Because we’re performing a two-tailed test, the
p-value is spread over both sides of the curve. The event to be examined is the mean change in
the FEV1 value by 4 milliliters. So, we’re looking at the areas on both sides of the
curve that represent this change as well as the more extreme results. The sum of the areas
is the p-value. In practice, the p-value is calculated using statistical software
and published with the results of the study. For our example, the p-value is 0.11.
This value is now compared with the significance level, that is, alpha.
In episode 10, we set a significance level of 0.05. This significance level means that the
probability of falsely rejecting a correct null hypothesis is 5%, which is the probability of
error. In court, the equivalent would be that 95 of 100 convicted individuals are indeed guilty,
while five are actually innocent. In fact, it’s debatable whether a probability of error of
0.05 would be adequate in this context. Depending on what’s being investigated,
a more stringent significance level can be set in advance. A level of 1% is also common.
The p-value should then be equal or below 0.01 for the study to be significant.
A significance level of 0.1% is also possible, the maximum p-value here should be 0.001.
However, the significance level is usually not set higher than 5% for quality assurance. So,
a more stringent threshold results in a greater certainty that a false hypothesis
isn’t accidentally accepted. But it also results in smaller rejection regions, making a
significant result more difficult to obtain. In our study, the p-value of 0.11, or 11%,
is higher than the significance level. So, as we already know, the study is not significant.
Let's imagine that the change in the FEV1 value after administering the drug had been twice
as large and averaged 8 milliliters. In that case, the study result would
fall in the rejection region. The p-value would’ve been lower than the alpha value,
and, therefore, the study would be significant. So, in summary: A low p-value means that the
observed effect is unlikely to occur when the null hypothesis is true. We conclude
that it’s not true, reject it, and accept the alternative hypothesis. In contrast, a higher
p-value means that the observed effect is not very unlikely when the null hypothesis is true.
Therefore, we accept it. However, it does NOT mean that this test and the large p-value have
proven the null hypothesis. We’ll take a look at why this is the case on the next slide.
Why can't we say that the null hypothesis has been proven when the study isn’t significant?
Now, a study might not be significant because it doesn’t have sufficient data.
In this case, there isn’t enough evidence to come to a conclusion.
This scenario is often seen in court: Imagine that there’s no evidence that exonerates the accused,
that is, there’s nothing that proves the person's innocence. However, there’s also no evidence
that the person is guilty. In this case, the person is acquitted due to a lack of evidence.
This is according to the principle that every person accused of a crime is considered innocent
until proven guilty. So, a conclusion can’t be drawn about this person’s guilt or innocence.
The same applies to the null hypothesis: If a study is not significant,
that is, there’s no credible evidence to show that the null hypothesis is false,
we can’t conclude that it’s correct on this basis. To prevent another popular misunderstanding:
The probability of the validity of the alternative hypothesis can’t be calculated
from the p-value for the null hypothesis. Just like the p-value doesn’t specify the
validity of the null hypothesis, the difference between 100% and the p-value doesn’t equal the
validity of the alternative hypothesis. So, that rounds it off on statistical significance.
In the last episode, you may have wondered about the 4 milliliter difference in the FEV1
value measured: If a study were statistically significant or not, it wouldn’t say anything
about the medical relevance of the results. If we’d measured a mean increase in the FEV1
value of 8 milliliters, then the study would’ve been significant. The next question to ask would
have been whether this change is of medical relevance. Is it large enough that patients
with asthma would really benefit from taking the drug? With a difference of 8 milliliters,
which is less than 2% of the average adult's tidal volume, this question would’ve been quickly
answered: This improvement certainly wouldn’t make a difference to patients with asthma.
Now, would you like to check if you’ve understood the most important information in this episode?
Why not try the quiz on the next slide.