What are type 1 and type 2 errors?
What is meant by “the study is significant”? To understand this statement, let's take a
step back and first look at what a type 1 error is. Because a type 1 error differentiates
between significant and non-significant findings. Let's illustrate this using the example from
our last episode: A fictional study analyzed a new drug for patients with asthma. To assess
its efficacy in managing bronchodilation, the patient's FEV1 value was measured by pulmonary
function testing before and 30 minutes after administration of the drug. From the difference
between the two values, the change in the volume of forcibly expired air in one second
was determined, allowing to extrapolate the efficacy of the bronchodilating drug. The
null hypothesis, H0, to be rejected was: "In test subjects with asthma, the amount of air
expired in one second does not change 30 minutes after administration of the drug". The alternative
hypothesis, H1, which is mutually exclusive to the null hypothesis, was: "In test subjects
with asthma, the amount of air expired in one second changes 30 minutes after administration
of the drug". As we’re only speaking of a general change, the hypothesis includes
both positive effects as an improvement of respiration and negative effects, such as
a deterioration. Such hypotheses are non-directional or two-tailed, because we’re statistically
testing both directions for a potential change. Had we excluded negative effects right from
the start, we could’ve also tested the following hypothesis: "In test subjects with asthma,
the amount of air expired in one second increases 30 minutes after administration of the drug".
Then we’d only look at the range of improved values and perform a one-tailed test. Such
a hypothesis is also called a directional hypothesis. However, in medical statistics,
analysis is usually two-tailed, so we’ll show a two-tailed example here. But our fictitious
example isn’t a randomized controlled trial, which is usually the gold standard in clinical
research. In such studies, two groups are compared: an intervention group and a comparison
group. But to keep things simple here, we’d like to show the statistical evaluation of
a fictitious study through a simple example: So, our fictitious study is testing the validity
of the null hypothesis. The null hypothesis relates to all individuals with asthma, that
is, the entire population of asthmatics. However, the study obviously can’t examine everyone
with asthma. Therefore, a representative group is tested, known as the sample. In our example,
the group comprises one hundred individuals with asthma who are receiving the new drug.
Let’s assume that the diagram shown here depicts the results of the study. They differ
significantly for the individual subjects. For example, in some participants, the FEV1
value decreases despite the intervention. However, in the majority of participants,
the FEV1 value improves. The data distribution can be more or less described with a Bell
curve. In other words, the dataset here is approximately normally distributed. But that's
not always the case. The mean FEV1 value improves by approximately 4 mL in all participants
with asthma. This corresponds to less than 1% of a normal tidal volume in adults. Based
on this result, can the null hypothesis be rejected in favor of the alternative hypothesis?
Can we state that the new drug is effective in patients with asthma?
If the study included the entire population of individuals with asthma, then the results
would be clear and conclusive despite the small improvement: The null hypothesis could
be rejected, and the alternative hypothesis accepted. But since the study only examined
an exemplary representative sample of 100 participants with asthma, we need to expect
room for error in the interpretation. So the question that arises is: How likely is it
that the study result is due to chance? Or in other terms: How likely is this observed
difference the result of chance when the participants on average don’t actually benefit from the
new drug? This question is important in assessing how
certain our study result represents the real relationship. Let’s suppose that we reject
the null hypothesis based on our observations, even though it’s actually true and the observed
results were indeed due to chance. We’d then accept a false alternative hypothesis.
This is a type 1 error. In contrast, if we reject the alternative hypothesis even though
it’s true, that is, the null hypothesis is accepted when it’s actually false, this
is known as a type 2 error. A type 1 error would have large-scale implications
and must be avoided. In our example, the patients with asthma would then be treated with an
ineffective drug and exposed to potentially adverse effects. Also, they would miss the
opportunity to be treated with another effective drug instead. However, a type 2 error would
also have large-scale implications. In our example, the type 2 error would lead to an
effective drug not being launched onto the market and remain unavailable to patients
with asthma.
How does the knowledge of the two errors help us to interpret study data? Also, how can
we assess whether the data is due to chance or reflect a true difference? Let's take another
look at our fictitious example:
The average effect of the drug on bronchodilation would always vary slightly when the study
is conducted with different subject groups, regardless of whether the null hypothesis
is true or false. To determine the extent of the deviation for the entire population
of asthmatics, we’d need to calculate a statistical mean from the experimental mean
values of the different sample groups. Assuming that the null hypothesis is true, these mean
values are distributed around zero. But is 4 mL considered close to zero or distant enough
to be considered a significant difference? The calculated mean value can help to examine
the extent to which the drug’s effect in the study reflects the mean for the entire
population. The distribution of the mean values for the
entire population would produce a Bell curve. In practice, repeating studies is too time-consuming
and expensive. So, from the mean and scatter of the study data, we can theoretically determine
how large the probability is to reject a true null hypothesis and, therefore, commit a type
1 error. An acceptable probability level of the type
1 error is defined during the study design. In medical research, the type 1 error rate,
also called the significance level or simply denoted with alpha, is usually set to 5%.
If a one-tailed test is performed, this 5% lie on the side of the curve whose range of
values is examined. In contrast, if a two-tailed test is performed, the 5% are split between
both tails of the curve so that the error range equals 2.5% on each side. Let’s mark
the error range here in blue. These shaded areas are known as the rejection region. If
the test value falls in these regions, it’s unlikely that the null hypothesis is true.
Therefore, it’s rejected. Now, let’s plot the result of the study.
As you can see, the value isn’t located in the rejection region, meaning that the
study couldn’t prove a difference in the FEV1 value before and after administration
of the new drug. Therefore, the null hypothesis is accepted. The study couldn’t provide
evidence that the null hypothesis is false and the alternative hypothesis is true.
However, the rejection region isn’t the only instrument of assessing whether a null
hypothesis should be rejected or accepted. Would you like to know how the p-value can
be used to achieve this? Then stay tuned for part 11 of our Chalk Talk series on statistics.