News alert!
You check your phone and see a notification announcing a new study that shows that eating
an apple a day keeps the doctor away.
But when you open the link, you discover that the CROP trial – a large trial examining
the effect of daily consumption of fruit of all kinds on the number of primary care visits
– actually showed that consuming fruit regularly had no impact on number of primary care visits.
That headline was based on a subgroup analysis specifically for apple consumption.
So should you believe the results of the subgroup analysis, toss out the other fruit that you
love, and eat only apples?
What are some of the key issues we need to think through when it comes to making clinical decisions
based on subgroup analyses?
Well-designed randomized controlled trials can generate reliable data to answer clinical
questions.
But they're time-intensive, and often come with hefty price tags.
To learn as much as possible about who might benefit from a treatment, investigators often
examine treatment effects in different subsets of trial participants.
Perhaps a given treatment has a different effect based on the age of the participant,
or their baseline blood pressure.
Or even the type of fruit consumed daily!
But these subgroup analyses should be approached with caution.
Let’s talk about a few specific reasons why.
First, selecting a smaller sample from a larger group diminishes the effect of randomization.
Confounding factors that were previously balanced may be reintroduced.
Imagine that the participants who ate apples as compared to other fruit were more
likely to work as teachers.
They might then be younger, with fewer chronic health conditions – and therefore might have
fewer primary care visit – than others in the trial.
Second, subgroup analyses require us to reconsider statistical significance.
The more subgroup analyses performed, the more statistical testing involved, which increases
the likelihood of getting a positive result from chance alone.
A subgroup analysis may therefore detect an association between apple-eaters and number
of primary care visits, when in fact no association exists.
Third, we have to think about statistical power, which is the probability of finding
an effect if there is one to be found.
For example, daily orange consumption is known to increase
vitamin C levels.
If the difference in vitamin C levels between orange eaters and non-orange eaters is large,
this effect can be detected, even with a small sample size.
But if the effect is small, there could be too few participants to meaningfully detect
the difference in vitamin C levels between these groups, therefore leading to a higher
probability of a false negative, or Type 2 error.
A similarly underpowered analysis for daily apple consumption and overall health could
suffer from the same error.
So, is there ever a good reason to perform a subgroup analysis?
Subgroup analyses can be useful for generating hypotheses if unexpected associations are
detected that investigators want to explore in future studies.
Being able to reproduce a finding from a subgroup analysis in a prospective trial is an
important confirmation of validity.
But there are some important rules to avoid the bad apples: subgroup analyses should be
pre-specified in the study protocol and statistical analysis plan.
Analyses should be restricted to a few clinically important questions, and results of all analyses
should be reported – not simply those with positive results.
If subgroup effects are expected, trials should be appropriately powered to detect them.
So you want to plan a trial to see if an apple a day really leads to fewer primary care visits?
I’ll bite!