so once we have these two different
dimensions of validation right face
validation appeared vs empirical and
micro versus macro we can combine that
with our standard notion of inputs and
outputs to think about different
standards for validation that are
important to accomplish in order to
provide some benefit to some it's that
indication that your model is validated
right so in a paper that I previously
removed rollin Ross we can kind of
discuss these four kind of standards for
validation it means suggested that you
kind of at least address each of these
standards when
developing your model so what is
microphase validation right show that
the elements of the implemented model
for the agents actually correspond to
real-world elements right making sure
that the agents in effect look something
like a real-world agent does in terms of
their properties or behaviors the next
of macrophage validation that the output
patterns that the processes by which
things happen in the model
match in some respect of real-world
patterns of behavior right the next one
is empirical input in that validation
making sure that the parameters and we
are setting up your inputs the behaviors
and purity match to some of the data
that you have drawn from the real world
or observe about the real world of
phenomena and finally and most
importantly this is the one that people
think about when they also think about
empirical output validate your sorry and
validation is empirical out with
validation showing that the actual
numbers your model produces correspond
to real-world data in facts and this can
be done a number of different ways right
you could have just stylized facts right
for instance we know the diffusion often
is the shaped curve right we've shown
that our model the diffusion model we
talked about in Chapter four are uniform
kind of correspond to that you could
have real-world data right you could
have actual numbers about the way the
patterns of the real world look and show
that your model produces output
just that and in fact that's one of the
things we're going to talk about in the
next couple of talks because that's
often considered to be an the gold
standard for validation right and then
you could also have cross validation so
imagine that you have your model and
someone else has a model they validated
right one of the things you could do is
show that your model produces similar
results to that older model right this
is often this is sometimes also called a
model docking right where you dock the
two models together and somebody to show
they're related to each other but those
are the standard methods for doing
validation in reality you should really
be checking off each of these boxes as
you go along and building your model and
showing that it's valid and
two for just that we've kind of
discussed through the first three for
the
uniform model at least the first two in
terms of empirical input validation what
one of the things we did was we ran a
set of experiments with the original
bass model where we specify the inputs
to be in the same range of empirical
inputs that have been observed
historically for the bass model and then
we look to see whether our model in the
output results correspond into
real-world data and so the way we did
that was we took
the prediction in terms of adoption rate
and because we weren't simulating
necessarily the same markets and things
like that we just looked at the
correlation between our model and the
actual empirical adoption of innovations
that are occurred in the past so for
instance air conditioning bed the bed
feeders actually is what this is coffee
makers dryers freezers lawnmowers
refrigerators and as you can see the
ancient based version the bass model
correspondent pretty well with the real
world data and in fact in some cases
actually did better than the actual bass
model and there's a couple of reasons
why that happened but the important
thing to know is that we were able to
empirically validate the output of our
model to real-world data and that's one
of the kind of the gold standards as we
mentioned of
model validation
so yeah so I'm going to talk a little
bit more in the next couple of
discussions about how to do that kind of
validation and we'll explore that in
more detail just in case that
explanation of how we did the validation
on the basket I wasn't exactly clear I
wanted to actually bring up the model
and show you so this is a very very
simple diffusion model that we had build
for the random rust paper and in it
right we just have different ages than
that connected by a network or anything
like that but they adopted a PDA to you
and M here is the number of agents in a
model right and so what we did to
validate our our model against the
real-world data was we actually ran a
behavior space experiment so I'm showing
you the one for refrigerators right now
but we ran it for all the different
versions and we set the P and Q values
actually to the same empirical P and Q
values that asked that these in this
original model now we could have
searched over these and maybe there's
actually some better fitting values for
our particular parameter but it turned
out that they generated very good
results just using the ones that vasa
originally used is original paper so we
set those parameters up we didn't ran
the model for a number of different
random seeds and then we just counted a
number of trails with adopted then what
that allowed us to do was to compare the
results of this data once we have it in
our to the actual empirical data that
Vasek collected for the number of
doctors of refrigerators every year for
a certain time period
so yeah so this worked out pretty well
in the end you're able to successfully
validate are