[MUSIC]
So earlier in the organ healthcare experiment,
we saw that
there was huge noncompliance.
So we took our people,
we randomly split them into two groups, receive
medicaid, don't receive medicaid.
But among the people assigned to enroll in medicaid,
only 30% actually
did enroll, 70% did not.
These people did not comply with their treatment assignment and
that's the problem for learning about treatment effects.
Now we talked about four ways for dealing with this.
The first is to do intention-to-treat analysis.
And that's what we did before.
We redefine our treatment of interest to simply be have any option
of enrolling in medicaid,
instead of actually receiving medicaid.
The second approach is assuming random compliance and that is the
people who enrolled in medicaid were basically the same as the
people who didn't enroll.
They just randomly decided.
And they got that form in the mail they just flipped a coin and
if it's head, they enrolled, if not they didn't.
Now that's probably not true because you think that the people who
enrolled probably were sicker,
have more health problems whereas,
people who didn't enroll might have been younger,
healthier and
thought well, I'm younger,
I don't need Medical insurance
wherever, through the form.
So we probably don't believe that assumption.
The third approach is called,
instrumental variable analysis and
we will talk about that in a later module.
Now we're gonna talk about the fourth approach called Bounds Analysis
and basically the idea behind this Bounds Analysis is that our
data is gonna tell us what happens to the people who complied when
they get medicaid but it doesn't tell us anything about this people
who didn't comply, these guys didn't get medicaid.
So we don't know anything about what happened to them.
So it's possible that they could all be at the
extremes of whatever our outcome variable is.
So we're gonna look at depression.
So it's possible that if these guys had actually complied,
and got
Medicaid, then all of them would be depressed okay,
it's possible.
We don't know because they didn't actually get Medicaid so we don't
see in our data, that's a counterfactual.
The counterfactual that they do get Medicaid,
it didn't happen.
Another extreme is that when they all get Medicaid,
none of them
are depressed.
They're all super happy.
We don't know which one of these is the case.
So we're gonna assume that all of them are depressed and see what
our analysis gets us.
Then we're gonna assume that none of them are depressed,
see what
our analysis gets us.
And those are gonna give us two bounds for the actual effect of
getting Medicaid.
And the truth is somewhere in between,
the truth is that if
all of these people enrolled in Medicaid,
some percentage would
be depressed.
It's just the problem is we don't know what that percentage is but
we do know that it's between zero
and 100% by the definition of our percentage.
Before moving on to remember what our data is let's look at the
control group, the people who were assigned not to get Medicaid.
Well fortunately there's basically no compliance problem there.
They were assigned not to get Medicaid and about 100% of them didn't
get Medicaid like they should.
Now some people did manage to somehow enroll in Medicaid,
even though
they weren't supposed to.
But it was a very small number of people,
so we'll just ignore it
for for simplicity.
So let me remind you of what our data is.
We're gonna look at the outcome variable percentage of people with
a positive depression screening.
And we looked at this before when we did the intention-to-treat
analysis.
And this is from table S1 in Oregon Healthcare paper.
So over here I've written two groups.
First the control group,
the people who were assigned to not get
Medicaid and because all of them complied,
everyone in that group
actually didn't get Medicaid.
So those people, 30% of them had a positive depression screening.
Up here I've got a very specific group,
the people who were assigned
to get Medicaid and they actually enrolled in Medicaid.
So these are the people who were assigned to and complied with getting
the treatment.
Among those people 27.79% of people had a positive depression screen.
So remember before when we did the intention-to-treat analysis,
we just looked at the difference between these two numbers and
got -2.21% as our average treatment effect of having the option
of enrolling in treatment.
And if we assumed random compliance this would also be our average
treatment effect of actually receiving Medicaid.
So remember with random compliance
the analysis doesn't change,
only the interpretation of your number
changes but we don't wanna assume that because of these reasons
I mentioned earlier.
So let's do the Bounds Analysis.
So let's start out with the control group.
So these people we know exactly what the average outcome of when
you don't get medicaid is because there was no compliance form,
it's 30%.
So if I assigned everybody out there in the population the whole
United States to not get medicaid,
then 30% of people would have
a positive depression screening,
but now remember the definition
of our average treatment effect.
It's the difference between the average outcome when everybody gets
medicaid and when everybody doesn't get medicaid.
So we know one of those numbers,
30% that's the average when nobody
gets medicaid. 134 00:05:59,944 --> 00:05:59,849 What about
the other one,
the average when everyone gets Medicaid?
Well because of random assignment,
the people who were assigned
to get Medicaid are random sample of all our people okay.
So if all of them complied,
we'd be done, but not all of them
did.
So we've got to split them into two groups.
We've got our first group 30% of people,
three over ten, those people we know
what happened when they got Medicaid because they complied.
So we know that for these people,
the percentage of people with
the depression screening was 27%.
Now let's look at the people who did not comply,
the 70% of people
who didn't get Medicaid even though they were supposed to.
We don't know what would have happened to these people if they had
gotten Medicaid..
But let's look at the extremes.
The one extreme is that they are all depressed.
So 100% of these people are depressed.
Well if that's the case,
then our overall average among the people
who were assigned to get medicaid would be about 78%.
So that's just doing this algebra Now,
let's look at the other extreme.
So here for the people who did comply,
we have the same thing.
We know exactly what happens to them when they get medicaid,
28%
[BLANK_AUDIO]
for the people who did not comply,
suppose That none
of them were depressed,
they were all very happy, okay, 0%.
It's possible, we have no idea because in our data,
they actually
did not get medicaid and we're asking what if they did?
So our data doesn't tell us anything but we know that it can't be
below 0%.
So
let's suppose it is.
If we do that, then this is just zero and so we just get three tenth times 28,
that's about 8%.
Okay, so what we see in this analysis is that if we took everyone
in the US and we force them all to get medicaid then the percentage
of people with positive depression screen,
the percentage of depressed
people would be between 78 and 80%.
That's what our experimental data reveals.
Now this is a big bound,
it's a lot of numbers between there but
the reason is there's a lot of non compliance,
a lot of people
did not get medicaid and I think about even the worst case where
nobody complied where we assigned a whole bunch of people to get
medicaid and none of them took it.
Then in our data
we would not see a single person who had Medicaid and yet here we're
asking what if they get Medicaid?
Well in that case it did 0% to a 100 because our data just wouldn't
tell us anything about what would happen if everyone got Medicaid.
But fortunately that's not our problem is not that extreme.
So we actually have these chunk of people here who are tightening
these bounds.
But there is a lot of people 70% who didn't comply.
So that's why our balance is so big.
Because that's what the data tells us okay so let's put that together
with our data from the control group and write down some balance
for the other treatment effect. So the average treatment effect
could be high as 78% in if you had medicaid minus 30% if you didn't
that's from our
control room
or, approximately 50%
[BLANK_AUDIO]
that means that if
this is the case then putting moving everyone in the Medicaid would
increase the amount of people depressed by 50% it's a huge number,
huge increase.
Alternatively, let's look at the other bound,
it could be 8% minus 30%
which is about minus 20%.
So it could be that when we put everyone into medicaid, the percentage
of people who are depressed dropped by 20%,
that's a really big
good treatment effect,
that we need to lower depression by 20%.
So that's what Bounds Analysis tells us,
that the average treatment
effect is between this two numbers about 50% and minus
20%, and it's a big range because there's all this noncompliance.
So we can't tell is the average treatment effect,
zero or not, we don't
know because the Bound include zero.
Is it positive?
Is it negative?
We have no idea because the Bound includes numbers that are positive
and it includes numbers that are negative.
So, even though we had a really nice randomized experiment,
it wasn't
perfect because there's so much noncompliance and so that's a big
constraint on our analysis.
So this is one thing you can do without making assumptions like
random compliance which we don't release or without changing your
goals like doing Intention to treat analysis.
So to compare these results,
we have -20% to 50%, intention to treat
and random compliance will both give -2.21% and later on,
see that if you do the instrumental variables analysis,
you'll
get -9.15%.
So later on we'll talk about why this number is not necessarily
what you want either because it's actually not the average treatment
effect, it's the average treatment effect for just a small set
of people, not for every body.
So unfortunately, there is no perfect answer but that's just what
happens when you have things like non compliance in the data.
So these problems arise in data analysis and there's various ways
of dealing with them and this is the kind of thing you'll see when
you go and look at the research and look at the literature.
[MUSIC]