[SQUEAKING]
[RUSTLING]
[CLICKING]
RICHARD DE NEUFVILLE: Hello, again.
Now, the next section that we'll be talking about is this topic,
the forecast is always wrong.
Now, I put it in quotes there because it is possible
that the forecast actually matches what
happens later on in the world.
It is not as if there's some devil that if you say yes,
it will say no or vice versa.
But our experience is that the forecasts
we make about the future tend to differ substantially
from what happens.
So the objective of this exercise
is to give us a realistic assessment of our ability
to forecast accurately.
You will come away with the evidence
that, in general, the forecast that we
make for the performance of a system
five, 10, 15 years in the future, or even a year
or six months, if it's a fast turnaround kind of technology,
are different from what we anticipated.
Let's look at some evidence and, first of all, the simplest
possible case, projection of project costs.
So here is a ratio, a graph showing the ratio,
of real cost--
what it actually cost--
to the estimated cost.
And on the vertical axis here, it
is a percent of observations, going from 0% up to about 15%
here, of the whole observations on a very simple job, which
I'll explain in a moment.
So this is how many observations here.
And this is the ratio between the real cost and the estimated
cost.
So in this case, if you had actually
gotten exactly the forecast correctly,
you would have 1 being the ratio.
And what you see is that it's very
different over a range, as evidenced
by this histogram of occurrences, frequency
of occurrences.
Now, let me say a little bit about what this is.
This represents what possibly is the simplest possible
technological activity.
It represents the cost of resurfacing runways.
I told you I was interested in airports.
This is an airport example.
Now, resurfacing runways is that you basically
put down a layer of asphalt over the top of the runway.
Because after aircraft pound on it and pound on it
and pound on it through the many landings that occur,
it breaks up a bit.
So the task is you put down a determined amount of asphalt,
a depth, over the width of the air
of the runway times the length.
And the volume is pretty clear.
The technology is basically you drive up a truck,
you dump out the asphalt, you roll it down, and that it is.
It could hardly be more simple than that.
So you might ask, why is it that this ratio of the real cost,
the actual cost, is so large?
Let me emphasize how large it is.
It goes down from about 0.5 on one scale to about 2 and 1/2
at the other end of the scale.
And all this is around a median value of about 1 and 1/4.
1.25 is the highest point.
Now it'd be easy to understand why
the median, the most frequent, value is higher than 1.
Because while you're there, you may decide, oh,
I'll do a little extra taxiway, I'll do a little extra here.
And it's our common experience that their estimates
are undershooting what actually happens.
But what's interesting about this case, which is so simple,
is that we go from 1.25 as a median down to about half
of it--
0.5 or 0.6-- to about double of it.
That is, our ability to predict the actual cost
is off by a factor of 2-- half or double
what actually is the most frequent cost.
This is really remarkable.
This is the kind of thing I'd like you to keep in your mind
about how difficult it is.
Now, why does this occur?
It's not because the designers couldn't do the math,
that width times length times depth
is a certain amount of asphalt, and there's how to do it.
What happens here in this particular case,
as in many cases, is that one of the major determinants
of this distribution is the fact that energy prices--
in this case, petroleum prices, but they
tie in with other energy prices--
vary enormously.
That is, the price of oil during the last 20 years
or so has gone from about $15 to $150 per barrel,
by a factor of 8 or 9 or 10, however
you wish to talk about it, but a huge variation.
And that's what's reflected here.
That's one of the things that's reflected here.
But it's not just asphalt. If you're making cement,
if you are heating steel and creating steel,
all these are energy intensive.
Electricity prices change the same way.
So this is largely driven by factors beyond our control.
In addition to these is the factors
that their labor changes.
That is, during periods of high economic activity,
you'll have to pay overtime for the workers.
When there are recessions or bad economic times,
you'll get low bids.
And so in addition to the materials changes,
there are the economic changes.
But in short, this particular example
gives you an illustration of the way that,
even in the simplest possible situations,
that our ability to predict the actual cost
is not very good at all.
This is the kind of changes that happen.
Now, to amplify this, I have a similar kind of graph
here for all kinds of projects.
Again, this is on the vertical axis up here.
And we're talking about the average cost
compared to the predicted cost.
So the predicted cost is 1, about here.
And the averages are whatever is on here.
It goes up to about 2 and 1/2 in this particular graph.
And it's for a whole variety of projects.
This was surveyed some time ago.
And the quality of this is defined
by that little bar over the far side,
over about in here, where it talks
about the standard deviation.
What's that, the standard deviation?
The standard deviation is a measure
of how the dispersion is around the average.
So in the previous graph, we saw that there
was a very wide dispersion of about a factor of 1.
But in all these ones here, the standard deviation
shows the dispersion, which is this little bar
on the far right-hand side for you, which
is about a standard deviation of close to 1/2.
And these are for ordinary projects--
building highways, all kinds of ordinary kinds of activities.
So what we see here, the common experience
is that the projected costs in advance
are under what actually happens, and under by large amounts,
both up and down, high variations,
the standard deviation about 50%.
Plus or minus 50% is what we get.
Now, this is for the same kind of figure but for NASA.
NASA's the National Aeronautics and Space Administration.
And it shows these now for high-tech groups.
And what you observe when we're talking
about high-tech and more innovative and less usual
kinds of activities that the cost variation is very much
higher here.
And now we have--
a lot of these different projects went up to about twice
the estimated cost.
And over on the right, you see the standard deviation,
which is now over 60%.
And it's more variation.
So what we observe in each case that I've shown so far
is that we don't get it right, and we don't even
get it right within a range.
There's a large range of possibilities.
Our forecasts are, as I suggested at the beginning,
our forecasts are wrong, wrong in the sense
that what actually happens is not what you said was happening
or what the expert said was happening.
And it's not saying the experts aren't good at what they do.
It is that the world changes.
Things happen.
So we need to take this into account
as we think about our ability to estimate and manage
a set of projects.