On August 24, 2020, as COVID-19 cases were spiking across the state, the University of
Illinois welcomed students back to campus at Urbana Champaign for the fall semester.
The decision was informed by months of planning and research.
According to an epidemiological model developed by university professors,
there could be a relatively safe way to open campus for in-person classes.
NEWS: They’re doing 10,000 to 15,000 tests a day.
NEWS: We're testing everybody on our campus at Urbana-Champaign twice a week.
NEWS: Results can be accessed on an app.
Things didn’t go according to plan. On September first,
just a week after students returned to campus, there was an outbreak.
NEWS: Two weeks in, the University of Illinois reported more than 700 cases,
which is more than they expected for the semester.
And the University of Illinois wasn’t alone.
Tucker Carlson: At this point, we should not be surprised that the model got it wrong.
Andrew Cuomo: It sounds like I’m saying I don’t really trust the experts… because I don’t.
Why does it seem like COVID-19 models are getting it so wrong?
In reality, there’s a lot more going on here.
In order to understand how models can be useful, we first have to understand how they’re built,
what they can— and can’t—tell us, and how all of this often gets lost in translation.
Epidemiology is not rocket science—it's harder.
When scientists build an epidemiological model, what they’re really doing is writing
down a recipe for how they think the reality of an epidemic unfolds,
and adding math so that they can project that reality into different possible futures.
Typically, in a model, what you'll do is first write down
what we call the variables. These are the things that are changing over time.
And then in the next step, you’re going to try and write down how those different variables interact.
And then what you hope to do is take
some observations about the real world and help to parameterize your model.
How fast does the disease spread from one person to another?
How many contacts does a typical person of some age have with people of other ages?
Scientists use two main approaches to building models of COVID-19:
agent-based and compartmental models. Most models are a combination of both.
With an agent-based model, our goal is to think about individual agents—in
this case, people. We give them certain rules or certain behaviors, and then can simulate forward,
based on those behaviors, how they interact—and in the case of COVID,
how they spread disease and how their behaviors might change because of the disease around them.
But agent-based models don’t easily scale to very large populations. They can also
require a prohibitive amount of computational resources to run.
So, for a broader perspective, scientists also use compartmental models, or SEIR models.
Everyone in the population falls into one of these categories “S,”
for susceptible to the disease, “E,” for exposed, “I” for infectious, and “R” for
recovered or removed. You can think about these compartments as different stages of the disease.
Modelers use differential equations to simulate how people move from one
stage to the next based on data like infection rate, hospitalization rate, and fatality rate.
Scientists can use approaches like compartmental and agent-based models to approximate reality.
But reality is a complicated and unpredictable place, rife with uncertainty.
First off, we all go in assuming we're going to be wrong. We understand that
there is no way to get it right. There is no such thing as a model that will exactly
represent the world or can possibly capture every one of the dynamics in the real world.
The prediction is very sensitive to how good data is. Data is dispersed, is incomplete,
and is not taken with one uniform protocol everywhere. When you don't have good data,
you cannot have good predictions.
Small differences in the parameter are amplified as the dynamics of the system progresses.
So this is very similar to the butterfly effect. So slightly different initial conditions leads to
a very different outcome. This exponential sensitivity causes differences to amplify
exponentially fast. So you have a spreading cone of possible trajectories of your future.
But there’s an even deeper source of uncertainty that lurks in every model. That source... is us.
So when we look at human behavior and human dynamics, there's a huge
amount of uncertainty there that you can imagine.
Our daily contact patterns are so out of the norm. We’ve been scrambling to find data, to get insight
into how people are behaving during COVID and how that behavior is changing on a weekly basis.
In the absence of data on relevant human behaviors,
modelers have to make assumptions. A small error or a missed assumption can have profound
consequences on the model’s outcome. And even correct assumptions can change over time,
causing the model to veer way off course from its original projections.
This is why nobody can predict the future — not even modelers.
Models fundamentally depend on what are the assumptions that you make.
We made assumptions that people would be partying. We estimated how much that would be going on.
What we didn't take into account is we didn't expect that people would be noncompliant.
In other words, if you are told to isolate, we assumed that people would actually
do that. That assumption was one of the most erroneous ones that we made.
As a modeler, you have to try to focus on what are the important things.
And they may not be the things that you would spend most of your time on as a
scientist writing a paper for publication. But if you're doing that, but meanwhile,
you've left out the fact that, you know, a third of the people in your population don't even
believe that you should be even trying to mitigate COVID—that's a far more important source of error.
The statistician George E.P. Box once famously said, “All models are wrong, but some are useful.”
Instead of predictions, something modelers can offer us is projections.
These are glimpses into a range of possible futures that we have the power to change.
The best early models are the ones that are wrong precisely because we react to them and make them
wrong. So when we see projections about what could happen where we could do nothing at all,
the fact that those projections are invariably wrong is fantastic because
that means that we actually respond to the model's predictions and prove them wrong.
But even when a model does the job it’s supposed to do,
there’s a risk that it could be misunderstood when the worst-case scenario doesn’t come to pass.
How do we communicate when we can't predict the future, but we can provide scenario predictions,
right? This is a real challenge for scientists, for modelers,
but it's also a really important responsibility.
So early in the pandemic, one of the very first requests we received from
policymakers trying to understand what the threat was and trying to figure out what the right path
forward was a request from the White House Coronavirus Task Force. “What do your models
say about how many people are going to die from COVID-19 in 2020?” And we responded and said, you
know, we don't have a crystal ball. It depends on the policies. It depends on people's behavior. So
we can't give you a forecast, but we can make some projections under a number of different scenarios.
For example, we can make projections under the assumption
that we do not do anything to slow the spread of this virus. Or we'll consider sort of
the opposite extreme. We'll assume we're staying strict for the foreseeable future.
And then let's consider something in between, where we're sort of on a roller coaster of
trying to contain this virus. And the forecast under those scenarios looked very different.
There's a lot of misunderstanding about the application of models.
What I'd love to see the public take away from our modeling for COVID is that this is one tool
in the toolkit. And it's really important for us all to understand how the tools get used.
Nobody knew anything about epidemiological modeling
12 months ago. All of a sudden we're seeing the results of modeling on the nightly news.
I find it fascinating and really pretty awe-inspiring that there's this huge uptick in
scientific awareness and awareness of what mathematical modeling even is,
and, I think, a new appreciation for the fact that it's out there.
Pandemics will come again. We are in a relationship with nature that
favors that pathogens jump from one species to another. So for the future,
what I envision is something similar to what we currently do with the weather
forecast. We have whole institutions, we have surveillance systems with many
weather stations all over our territories that inform us of the current conditions everywhere.
And this is fed into models that continuously update their forecasts about weather. Why don't
we have that for pandemics? This is not a matter of a single country. We have to be together
in this. So when we do face a new challenge— and they will come—then we are ready just to
plug in the new data that instantly arrives and say something about how this can spread.