Welcome to Impact Factor, your weekly commentary on a new medical study.
I’m Dr. F. Perry Wilson of the Yale School of Medicine.
Despite COVID-19 death rates holding relatively steady, there is a palpable sense of pandemic
optimism in the US and abroad.
The Omicron wave – the largest of the pandemic – has, in most of the world peaked, and
is now receding.
There is broad hope that the immunity engendered by that wave, combined with that engendered
by vaccines, may bring us to the end of the pandemic.
And with that hope, a slew of changes in governmental policies regarding COVID – with even liberal
coastal governors like my own Ned Lamont in Connecticut starting to rescind mask mandates
and other mitigation policies.
As the pandemic ebbs, it is natural and appropriate to look back on the decisions we made along
the way and ask – were they correct?
Did we do the right thing?
A new “working paper” made a huge splash last week by claiming that, when it comes
to lockdowns – we made a terrible mistake.
The paper comes from three Johns Hopkins Economists and is a meta-analysis of other studies examining
the effect of lockdown policies on mortality.
Meta-analyses are useful tools – combining the estimates of various studies into one
overall estimate – allowing the vagaries of study design and the vicissitudes of chance
to wash out via averaging and, ideally, produce an estimate of effect that is something closer
to the truth.
So is their overall finding – that lockdowns do essentially nothing to reduce deaths from
COVID – close to the truth?
It was good enough for the media.
Fox News ran multiple segments on the study, on the news shows as well as on Hannity.
It garnered an Editorial in the New York Post and Newsweek as well.
If true, the implications are two-fold.
First, the individuals who were pushing hard against government restrictions during COVID
would get to drink a big glass of I-told-you-so, but more importantly, when the next pandemic
occurs – and it will – we can handle it differently.
But is it true?
Is it possible that government restrictions have no effect on COVID mortality?
That would go against the whole pandemic playbook – not just this pandemic – the sort of
basic tenets of infectious disease public health.
So there may be a problem here.
But before we get too deep, we need to talk about this term – “lockdown”.
What does that even mean?
At first blush, when I think “lockdown”, I think “shelter in place” – all but
essential businesses closed and all of the population told to remain in their homes.
This is not what the authors mean when they use the term lockdown.
They state “Lockdowns are defined as the imposition of at least one compulsory, non-pharmaceutical
intervention”.
So – closing schools?
Lockdown.
International travel ban?
Lockdown.
Mask mandate?
Lockdown.
Seems a bit hyperbolic to me, but there it is.
OK back to why this meta-analysis might not be telling us the whole truth.
The magic of a meta-analysis arises from the studies you put into it, so let’s start
with the included studies.
Figuring out the impact of lockdowns on covid mortality is not straight-forward.
There are a myriad of approaches to try to answer the question, but they fall into a
few categories.
First, you can do a modeling study.
The idea here is to take the epidemic curve you observe before lockdown and predict where
it would go assuming the conditions on the ground didn’t change.
Then you observe where the curve actually did go after the lockdown was instituted.
The difference between predicted and observed is an estimate of lockdown effectiveness.
This study in Nature showing the effect of lockdowns during the first Wave in Europe
used that approach.
You can see here how the predicted deaths assuming no lockdown (in blue) was substantially
higher than those observed once lockdowns were put in place.
This study does not appear in the meta-analysis.
In fact, all studies that used a modeling approach were excluded.
You can try to get at the lockdown : mortality effect by using a before-after study.
Here, you simply look at the death rate before lockdown and after lockdown and compare them.
Of course, you need to account for other stuff that changes with time, including the epidemic
curve, but good epidemiologists are pretty good at that.
This is a PLOS Medicine paper that looks at COVID cases and mortality before and after
lockdowns by state in the US.
Even after adjusting for things like the size of the pandemic when the restrictions were
put in place, the results show a significant decrease in death rates once lockdown occurs.
This study does not appear in the meta-analysis.
In fact, all studies using a before/after approach were excluded.
You can take what’s known as a difference-in-difference approach.
Look at two, hopefully similar, locations over time where one institutes some sort of
lockdown and the other doesn’t.
Then watch their respective death rates.
This is the only type of study allowed in this meta-analysis.
There are some good difference-in-difference studies, and some bad ones.
Epidemiology is hard.
In any case – after poring through 18,590 potentially eligible studies, the authors
narrow the field to 34 studies for review.
24 of those were included in the meta-analysis.
Of those 24, 12 are pre-prints.
Some of the studies looked at the effect of shelter-in-place orders – the only thing
that really says “lockdown” to me.
Overall, the authors estimate that shelter in place orders decreased COVID mortality
by 2.9%.
That works out to around 27,000 people in the US, which isn’t nothing, but would not
be described as a pandemic game changer.
The problem is that there are plenty of studies that estimate that shelter in place orders
have a much stronger effect.
The meta-analysis contains studies, like the Sears study here, that show a greater than
30% reduction in deaths, but they are given very little weight in the analysis compared
to one study – Alderman - which shows a 1% reduction in death rate.
The authors describe how they create the weights for each study but they don’t seem appropriate
to me.
In fact, the Sears study, which showed the 30% reduction in deaths, is an analysis of
data from all 50 states.
So is the Alderman study.
Why is the Alderman study given 28 times the weight in the meta-analysis?
You see the same effect of weighting in another analysis looking at studies that used the
Oxford “stringency index”.
The stringency index is an ordinal scale where higher values mean tighter restrictions – from
recommendations to social distance, to full shelter-in-place orders.
One study in that analysis, from the CDC, showed that higher stringency was associated
with lower death rates.
The authors report that the CDC study implies a 35% reduction in mortality when comparing
a stringency level similar to what the US had with a lower level of stringency more
consistent with advice to be careful.
Now compare the weight of that CDC study, 11, with the weight of this study by Dr. Carolyn
Chisadza – 7390.
That huge weight, assigned to one study, drives the entire stringency score-based analysis.
Given that many of these studies use national and even international databases, it seems
unlikely that one study should be given so much higher importance.
And, just to mention it, the author of that study – apparently – disagreed with the
meta-analysts interpretation saying:
“They think that lockdown had no effect on mortality, and that’s what they set out
to show in their paper” Turning back to the shelter-in-place order
analysis, studies in the meta-analysis are also given a quality score.
The authors show that higher-quality studies show less effect of shelter-in-place on mortality.
For example, the four studies with only 2 out of 4 quality points show a 34% reduction
in deaths, whereas the 4 studies that got all 4 quality points show just a 1% decrease
in deaths.
There are several good instruments for scoring the quality of medical studies.
The four point system the meta-analysis authors used is not one of them.
Rather, you get a point if your paper was peer-reviewed, a point if your study ended
after May 31st, a point if your study shows no effect on mortality in the first 14 days,
and a point if your study is written by a social scientist as opposed to an epidemiologist
or other scientist.
What?
With the exception of the peer-review thing, these quality metrics do not make sense.
In fact, they seem designed to penalize studies that show a mortality effect of lockdown.
Now I’ve been focusing a lot on shelter-in-place orders, but remember the authors define lockdown
as any non-pharmacological mandate.
And they do provide an analysis of mortality prevention stratified by various mandates.
It looks like this.
As you can see, these authors find that mask mandates reduce covid deaths by more than
20%, and that closing bars and restaurants reduce deaths by more than 10%.
These findings were not reported by the media.
The narrative around this study solidified firmly around “lockdowns have no effect”.
Ok here’s the thing.
Determining the effect of government policies on COVID mortality is really hard.
There have been a lot of papers – a lot of attempts at figuring this out – so many,
in fact, that you could write a meta-analysis like the one we’re discussing today that
says there are no real effects, or like this one which argues the effects are quite profound.
Personally, I think the clearest argument for the efficacy of government mandates, at
least in terms of curbing COVID mortality, is China, which has experienced 3.5 COVID
deaths per million population compared to 2800 in the US.
Obviously, I would never accept the extremity of lockdown that China’s authoritarian regime
has imposed – and I doubt my fellow American citizens would either.
But from a purely scientific standpoint, it’s hard to argue that lockdown – in the extreme
at least – doesn’t work.
The problem in the liberal democracies is that the infection rate, government mandates,
and personal choices are all interrelated.
People might be more careful about masking or social distancing because case rates are
high.
They might be more careful because the mere fact that a government mandate exists makes
them take the virus more seriously.
And of course, they might be careful because they are forced to be careful.
It can be hard to tease that all apart.
When the next pandemic occurs, this question will come up again.
How much should governments do to curb the spread?
It’s another one of those public health outbreak tenets that you would rather do too
much than too little.
The problem, of course, is that if you do too much – no one will ever know how bad
it could have been.
And if you do too little, well, people die.
It’s a lose-lose from a PR standpoint.
Criticism is inevitable.
Of course, that’s all part of the job.
For Medscape, I'm Perry Wilson.