(humming tone)
>>Hi again students.
This is our last class topic and it's one of my favorites.
We're going to explore the method of synthetic control.
This topic is not in our textbook,
but I've made a separate reading available
to go with this video.
Many times we have questions about a single policy.
For example, what was the economic cost
to West Germany reunification?
Or, what is the effect of California's high
cigarette tax on smoking?
Or, what is the effect of Oklahoma's Right to Work law
on wages or employment?
To answer these kinds of questions
we need to know what would have happened
without the policy intervention.
Synthetic control does this by creating an artificial
version of the policy in question and comparing actual
outcomes to the outcomes predicted by this
estimated synthetic control.
The technique brings a lot of credibility and rigor
to case studies and is being widely adopted in
finance, economics, and political science.
The method seeks to create a control
that mimics the behavior of the outcome being studied
in the pretreatment period
and also matches important characteristics
of the policy under study.
We will be more specific about that later.
For now, suppose we want to estimate the effect
of Oklahoma's 2001 Right to Work law
on state output by a synthetic control.
We would first choose a set of potential control units.
In our example, states without Right to Work laws
would be a good choice.
Then, we would choose a set of characteristics
that we think effect state output.
For example, education levels, population density,
and investment.
Then the method uses linear programming
to choose a weighted average of some of the potential
control units to create a synthetic Oklahoma
from say 1980 to 2000, the year before the law passed.
That weighted average is the synthetic control
and is chosen both to track actual Oklahoma state
output levels and have a similar profile
for education, population density, and investment.
To tell if we have a good control,
we can compare how well it tracks actual Oklahoma
and how closely it matches actual Oklahoma
on the characteristics we have chosen.
In a way, this is like matching
except we have two criteria and we're looking
to match over a relatively long period of time
instead of at one particular point in time.
Once we have a good synthetic control,
we simply compare the path of Oklahoma after 2001
with a path of synthetic Oklahoma generated by our control.
Assuming the only difference between the two
is due to the Right to Work law, our estimated treatment
effect of the law is simply the difference
in the two post 2001 time paths.
This is somewhat similar to a difference
in difference approach.
So to use this technique, we need a relatively long
pre-treatment period, a set of reasonable
potential control units, a set of characteristics
we think are important in determining the outcome
we are studying, and data on the outcome,
and characteristics for the treated
unit and all the potential control units.
In the pre-treatment period,
we estimate the synthetic control unit
as described above using only pre-treatment date.
Then, since the control is just weighted average
of the other units, we can use those weights
to create our counterfactual outcome
in the treatment period.
The estimated treatment effect is the difference
between the behavior of the treated unit and that of
it's synthetic control after the treatment is realized.
Let's consider an example from the literature.
In 2015, Abadie, Diamond, and Hainmueller
published paper in the American Journal of Political Science
that used synthetic control to estimate the effect
reunification had on West Germany.
Their pre-treatment period was 1960 to 1989.
Their potential control units were Australia,
Austria, Belgium, Denmark, France,
Greece, Italy, Japan, the Netherlands, New Zealand,
Norway, Portugal, Spain, Switzerland,
the United Kingdom, and the United States.
The outcome variable was real per-capita
gross domestic product.
The characteristics they chose to match on
were inflation, industry share of value added,
investment, schooling, and a measure of trade openness.
Their estimated synthetic West Germany was 42% Austria,
16% Japan, 10% Netherlands, 10% Switzerland,
and 22% the USA.
Here's a graph showing the results.
As can be seen, the pre-treatment fit is very close.
The synthetic control tracks actual West Germany
extremely well.
After reunification though, West Germany underperforms
the control by about 12% by the end of the period.
Here is the table checking for matching
on the economic characteristics.
By comparing the first two columns
it becomes clear that the synthetic control
matches West Germany quite well
on the chosen economic characteristics.
So in this paper, we see a terrific example
of what we want our synthetic control to do
which is closely track the actual outcome variable
pre-treatment and closely match on the chosen
characteristic variables.
If we have that, our work is done
and the treatment effect is what it is.
In the case of West Germany,
we see the country was around 12% poorer in 2003
than it would have been without reunification.
Like propensity score matching,
synthetic control matches on observables
meaning that time varying unobservables
could confound the results.
However, if the control is able to mimic the actual unit
over a long pre-treatment period,
then unobservables are not a big problem during that period.
Which means maybe they aren't a big problem
in the treatment period either.
As described, the method does not provide
a significance test on the treatment effect.
This isn't necessarily a problem
as case studies generally do not provide such tests.
However, we can create a significance level via
what is know as a permutation test.
To do this test, we estimate a series of placebos.
That is we pretend that the treatment occurred
in one of the control units,
create a synthetic for that unit in the pretreatment period,
and then compare the fit of the actual and synthetic
in the treatment period.
We repeat this process for each of the control units.
Obviously, since the treatment didn't happen
in the control countries, we shouldn't see an effect.
So if the bogus treatment effects are estimated
in the control units are the same size or larger
than the one we estimated for the treated units,
we would infer that the estimated treatment effect
is not really significant.
It looks just like the noise
generated by the placebo treatments.
Under the assumption that the treatment is randomly
assigned to a unit, we can use the rank of the size
of our estimated treatment effect
compared to the placebo effects
as a significance test.
In the paper on German reunification,
they show the estimated effect for West Germany
is larger than any of the estimated placebo effects.
The probability of that happening is one 1/17th.
Or 17 is equal to the total number of countries used,
namely West Germany, and the 16 controls.
1/17th is equal to about 0.06
and that could be taken as the significance level
of estimate of the cost to West Germany
of the reunification.
Case studies often lack rigor
and synthetic control is a good way to make them
more systematic and believable.
To run a good synthetic control experiment,
we need a long pre-treatment time period
to establish the control.
We need a set of potential control units
that are similar to the treated units,
and we need to ensure these potential controls
do not receive the same treatment as our treated unit.
We also need a good set of variables
that help to explain the outcome we are studying.
If the synthetic control outcome
closely matches the treated unit's outcome
during the pre-treatment period
and matches the treated unit on the values
on the predictor variables chosen,
then we have a good model and can simply take
the difference of outcomes between the synthetic
and the actual treated unit during the treatment period
as our estimated treatment effect.
If we have our heart set on having a significance test,
we can get one by performing a series of placebo test
and creating the significance level for the estimated
treatment effect based on the rank ordering
of the sizes of the estimated effects,
both actual and placebo.
Well, that's about it class.
I hope you've enjoyed and learned from this short course
on quasi experimental methods.
Always remember, friends don't let friends run
OLS regressions to estimate treatment effects.
(humming tones)