So I've been pretty loose about how exactly you compute
the treatment effect at the cut off, for regression discontinuity
design.
Well let's talk about that a little bit more.
So here in the Thesalian Camle/g paper, we've got test scores on
the horizontal axis and here we've got students scored ten, they
didn't get a certificate of merit, they're right below the cut-off
and students who scored an 11, they did get a certificate of merit,
they're right above.
So this simplest thing we can do, by analogy to how analyzing randomized
experiments work, is say what these students over here there are
treatment group, they got treated.
So let's just look at the average outcome for those students.
Well, that's exactly what's plotted on this chart because it's the
percent of the students winning scholarships and when the outcome
variable is binary, one or zero, did you win the scholarship or
not, the percentage
percentage of the units who won the scholarship is actually equal
to the average outcome.
So this variable on the vertical axis is the average outcome that
this number right here about 54% of students won scholarships.
So that's the average outcome in the treatment group.
Now let's look at the students in the control group, just people
who scored a ten and among those people, the average outcome was
about 46, that's this long solid number right here.
So by analogy that randomize experiments we compare this number
with this number and the difference here which is about, it looks
like 10% or so, that's our estimate of treatment effect.
Now you might think, why should I only compare ten or eleven?
Why do I actually compare the students who scored a nine and ten,
call that to my control group to the students who scored an 11
Who scored an 11 and a 12, call that my treatment group.
Well if I did that, then I would have some average outcome which
would be between these two numbers here in the control group.
Exactly where it lies depends on the percentage of students who
got a 9 versus 10.
Let's say it was 50-50, then it'd be halfway between these two numbers
Over here, it'd be somewhere between the outcomes for 11 and for
12, which since this is just a flat line, it'd just be up here.
So if we included 9s in our control group, and the 12s in our treatment
group, then the treatment effect would be the difference between
this number and something right in between these lines, or about
this distance right here.
So it'd actually be a little bit smaller than if we only included
the 10 and 11.
Because these people are closer, up here.
We could keep going.
I could include the 8s over here, I could include the 13s here,
and then that'd give me another difference.
I could include the 7s and
7s and then the 14s, that'd give me another difference.
So as I increase the bandwidth around the cut-off, the width of
this band of people I include, I get a different number estimated
treatment effect for each time I add somebody else on.
Now here's the problem, as I keep going or worry that we are all
of a sudden picking the confounding effect of test score increases.
Because as I already mentioned, we can't just compare average outcomes
over here with average outcomes over here, because the people over
here have higher test scores than the people over here.
In addition to the fact that the people over here got the certificate
of merit.
So that's not a clean comparison, we've got that confounding variable
test score.
We only really believe the comparison when we're doing it close
to the cut-off.
So the big question in this whole literature in this topic is, how
close is close?
Is just one away close, or is it two?
Or is it two or can I go up to three?
That's a question that people continue to work on to this day.
It's very very difficult and very important for practice because
as you change the bandwidth your aspect/g treatment effect changes.
Now why would you wanna include more and keep going out further?
One reason would be That you get more data, and the more data you
include we know that that means your confidence intervals are gonna
go down.
So if you look too close to the cut-off, then you're gonna have
very little data cuz there's not that many people who are that clean
close to the cut-off.
And so you're gonna get big confidence intervals.
So that's one benefit of using more The cost is that you worried
of that confounding bias that enters in.
So that's one thing you could do, that's the most natural thing.
Now, another thing you could do, which is what [UNKNOWN] and Campbell
have, is you do a regression on the left-hand side, that's what
this dotted line is here, then you do a regression on the right-hand
side, that's what this dot line
that's what this dotted line is here, and then you just compare
the values of these two different regression lines at this point.
So that's what this little circle here is actually, that's a continuation
of this regression line over to the other side.
And then you can compare this difference, and that gives you another
way of computing this treatment effect.
well the benefit of doing that, as opposed to this other approach
which I call the non-parametric approach to this experiment randomize
experiment analysis.
The benefit of this, this parametric linear approach is that you're
actually using all the data, even the people down here to compute
this number and even the people.
And even the people over here are being used to compute this number,
because everyone over here is used to compute this line.
So that means that your estimate of this treatment effect is actually
going to be quite precise because you are using a lot of data.
The downside is, you really have to make sure that this specification
is correct that it truly is sort of a linear relation
relationship and this linear relationship is invalid, then you might
be getting some extra bias because of mis-specification, because
of a problem with your statistical method as opposed to the analogy
experimental design where you're just comparing the people to average
outcomes in these two groups.
Okay?
So, the bottom line here is that there isn't one easy answer that
you should always do on one case and people are still working on
this.
And this is choosing this bandwidth and how exactly to contribute
the trimming effect is the most important problem and the RDD design,
regression discontinuity, so These are few ways that you can do
it, but there is no easy answer.
[MUSIC]