- So in this video, what I'd like to do is build on
what we were talking about in the last few videos
in terms of our approaches that we use
for propagating uncertainty into forecasts,
to then think about
how we can break down those uncertainties and analyze them,
in order to better understand our models and our predictions
and to hopefully improve them.
One of the concepts I'd like
to emphasize here is what I call
the Model Data Feedback Loop.
The idea that we might start from observations,
use them to calibrate a model,
propagate that uncertainty into a prediction,
and then analyze that uncertainties
in terms of breaking them down into the component parts
that go in.
So I might have a model with multiple parameters
and I wanna know the contribution
of different parameters and processes
to my overall prediction.
I might have different covariates
and I wanna understand the contribution of those covariates.
I might wanna understand the relative importance
of parameters versus better constraining
the initial condition,
the initial current state of the system
in terms of how it might reduce the forecast uncertainty.
Once I understand how different things
impact the forecast uncertainty,
that can help me set research priorities
that can direct, for example, how I might go
and attempt to synthesize
our current literature and current data,
or set up plans for how I might
make new measurements that specifically target
key sources of uncertainty.
One of things I wanna note here
is that in ecological systems,
most ecologists have an abundance of hypotheses
about how all different aspects of our system work.
So I don't view this idea of having
model data feedbacks prioritizing research
as any way in conflicting with the idea
of hypothesis driven research.
I think of it more as a way of
perhaps prioritizing the order in which we
tackle different hypotheses,
so that we can maximize the information gain
and our ability increase our predictive capacity.
To understand how uncertainty analysis works,
let's come back to
the simple analytical case
of being able to propagate uncertainties
in terms of the analytical or linear tangent approaches.
Where in that sense,
we saw that every component uncertainty
could be expressed in terms of
the uncertainties in an input,
whether that be an initial condition,
a parameter or a driver,
and then the sensitivity of the forecast to that prediction.
So, what we then see is that
the overall forecast uncertainty can be decomposed into
a sum of the direct impacts of each of these uncertainties
and then the interaction terms due to their covariances.
This essentially allows us
to say, what proportion of the predictive uncertainty
can be attributed to each of those terms.
Furthermore, it allows us to understand
when a term contributes a large amount
to the predictive uncertainty,
is it contributing a large amount
because that is an inherently sensitive
parameter or process?
Or is that contributing a lot
because that is a poorly constrained?
All of us being equal,
if we're trying to reduce uncertainties and forecasts,
we usually get more bang for our buck by targeting
things that are poorly constrained
over things that are inherently sensitive,
but may already be well constrained.
If we take a step back,
one of the things that this tells us
is that if we wanna understand the behavior of our systems,
and our, and their predictability,
we can't do that just with sensitivity analysis alone.
This is something that we'll see a lot
in the literatures that people use sensitivity analysis
to try to understand their models,
because that's only giving us 1/2 of the picture.
It's telling us what's sensitive,
but not how that interacts with uncertainties.
This flip side,
you know, often if we do a calibration exercise,
we get an estimate of what's uncertain in our model.
So what's the uncertainty in the parameters
that also by itself is insufficient.
It's really the interaction between these two things.
So if we have parameters that are very sensitive
and very uncertain,
they're gonna contribute a lot
to our predictive uncertainty.
By contrast, if we have things that are very insensitive,
and very well constrained,
they can contribute very little
to our predictive uncertainty.
And then in between, things can be important
either because they're very sensitive,
or because they're very under constrained.
If we look at this more broadly,
a complex model to make predictions may have
scores of parameters and inputs and initial state variables
that we need to put into a model to make a prediction.
So, as an example,
imagine a model had 100 different inputs
that needed to go in.
All else being equal,
we would then expect that on average,
each of those things would only contribute
1% to the predictive uncertainty.
In practice, it's very rare to see all components of a model
contribute equally to the predictive uncertainty.
But what we more often see
is there's going to be a small number of things
that contribute a lot,
and a larger number of things that contribute a little.
That is helpful because it tells us there's a,
you know, potentially large number of things
that are not going to be our priorities,
we can kind of take them as given for the time being,
and really focus our attention
on a small number of things
that often drive our forecast uncertainty
and really prioritize those.