It’s football season, but that’s not the only grand American tradition going on right
now.
No, we also seem to be in the “evaluating old CVOID policies” season – as papers
looking at the outcomes of decisions public health officials made in 2020 and 2021 are
bursting forth like the Wolverines from Michigan Stadium.
Last week we looked at the impact of masking in public schools, and this week, we’re
looking at, appropriately enough, attendance at NFL games.
That’s right.
It’s time to do some Monday morning quarterbacking.
We’re talking about this study, appearing in JAMA Network Open which looked at the 2020-2021
football season.
The key to the study is the fact that teams did different things over the course of the
season with regard to how many people they let in the stadium.
You had teams like the Vikings and the Chargers who never let any fans in the stadium, and
teams like the Cowboys who had more than 20,000 fans every single game.
Did it matter?
Was restricting attendance a good idea?
The authors decided to see if COVID rates went up in the county where the game was held
in temporal proximity to game time.
They looked at 7-day windows after the game (which are probably too short to see an effect
of game-day transmission), as well as 14- and 21-day windows.
But hold on - what does it mean to say that COVID rates went up?
This is where the study starts to feel more like a discussion on some stock technical
analysis forum than it does like a standard medical paper.
The authors tabulated case rates on a daily basis based on publicly available records.
They used some statistical tweaks to account for the expected differences in reporting
on weekends and the occasional data dump.
They then used several different outlier detection techniques to find days where more cases than
expected happened.
The simplest technique used a moving average approach.
Here’s an example using the Buccaneers.
The black line shows the 21-day moving average case rate.
You can see how it climbs from September to January – that’s the alpha variant wave
– how quaint – and then starts to descend.
The dots reflect individual daily case rates, and, when they are significantly higher than
the moving average, they are coded as outliers – yellow in this case.
In other words, the operational definition of “more cases than we’d expect” is
based on the assumption that case-rates should rise and fall with some smoothness.
I’m not entirely confident in this approach – but I’ll get to that in a minute.
Ok, assuming we believe that those yellow dots are important signals, we can ask a straightforward
question.
Would games with high attendance have more yellow dots following the game than games
with low or no attendance.
The authors find that they do.
In both 14- and 21-day windows following attended games, there were about 30-40% higher rates
of “outlier days” compared to unattended games.
As a partial negative control, the authors report no relationship using a 7-day window
following the game, which feels like what we’d expect epidemiologically given the
incubation period of COVID but for the life of me I can’t find the actual numbers in
the manuscript or supplement.
For games that had more than 20,000 fans, spikes in covid cases doubled, and for games
with less than 5000, there was no difference in spike rates providing some sense of dose-response.
This was true for the county where the NFL game was held, as well as contiguous surrounding
counties.
I would have loved to see another negative control here looking at remote counties, for
example, but no such luck.
At face value, this study provides some data to suggest that big sporting events increase
local COVID rates.
That, of course, makes sense biologically.
You’ve got a bunch of people in close proximity screaming – it’s not exactly surprising.
But that said, most of these stadiums are outdoors – there’s a fair amount of airflow
there.
It’s not an open and shut case.
And this method of flagging outliers got me a bit concerned.
One thing I noticed was that there were no outliers in the other direction – no “surprisingly
low” case rates.
It’s not that I think massive attendance at NFL Games would reduce COVID rates, but
I’m worried that the very choppy daily COVID case data has a risk of generating “outliers”
that aren’t truly meaningful.
I mean, look at the graphs – it’s not like the outliers are all one day after another.
Why would a game lead to a spike in cases on Tuesday but not Wednesday?
Could this all be noise in the system, and if so, would negative outliers cancel out
the positive ones?
I spoke to study statistician Justin Kurland about this issue who confirmed to me that
there was no analysis of negative outliers, but that very few were detected in any case.
I’m left not entirely sure what to make of the study.
Moving forward, if there were a highly infectious disease without a known cure for which no
vaccine existed (as was the case for much of the 2020 football season), you’re still
not getting me to pile into a stadium.
But I don’t think this study, despite its clever analysis, provides the solid proof
that supports my apprehension.