[MUSIC] So last time we looked at one outcome, positive depression screen, and found that
there was a statistically significant effect of being in the treatment group on this outcome
variable.
Now let's look at a different one, Total Cholesterol.
So here, the average value of Total Cholesterol in the control group is 204 (about), and the
treatment effect is 0.53, positive.
So that means that the average outcome in the treatment group is about 204.53.
So these numbers are pretty close.
The average cholesterol in the treatment group and the average cholesterol in the control
group are basically the same.
The difference is only 0.53.
So again we have to worry about: do we have enough of a large enough sample to say whether
this a true causal effect?
A true statistically significant causal effect?
And not due just the fact that we had random variation in who was put in what group.
Because, remember, the whole idea behind randomized experiments is that only when you have a very
large number of people that the two groups start to look the same.
If you have a small number of people, then it might be that you just happened to put
people who have, let's say, high cholesterol in the treatment group, people who have low
cholesterol in the control group, and then your groups wouldn’t be exactly the same.
So we have to actually worry about sample size in practice.
So that's what the confidence interval is going to get us.
So here the confidence interval is -0.83 to 1.89.
So first question we ask: is zero in the interval?
Yes.
So unlike before with depression, where zero was not interval.
Here, zero is in the interval, what do we do?
Now there’s two questions we have to ask: is this a big interval or is it a small interval?
How do we determine what big means?
Well, if you are a medical doctor, you might know how to interpret cholesterol numbers
like 204 and the differences of 0.3 and you might be able to tell instantly, oh, this
is a small interval because you have a lot of familiarity with that data like this.
Let's say you've never seen these variables before, and you have to tell us big or small
just from the table.
How can you do that?
Well, that is where these numbers here come into play.
These are the standard errors in our estimate.
And remember that is a measure of variation, and variation is how different are these variables
across people.
So, not everyone has the same cholesterol in the control group.
There is some variation.
This number is going to tell us how much variation there is.
So a rough rule of thumb is that about 70% of your people are going to be within the
mean minus your standard error and the mean plus your standard error.
So that's 204 minus 34 up to 204 plus 34.
So about 70% of people have values of cholesterol within this range in your data in the control
group.
So this interval is 68 units wide because it goes down 34 and up 34.
So if we look here, this was our confidence interval for total cholesterol.
So if we look here, this interval is about three units long, which we compare to this
interval, which is about 68 units long.
So that looks pretty short compared to 68, so that means that this is a small interval.
So what we have is something called a precisely estimated zero.
So we conclude that there is no treatment effect of being in the treatment group here
on total cholesterol.
0.53 - that's basically zero.
These two groups - 204.1 in the control group and 204.53 in the treatment group -basically
the same thing, treatment didn't matter.
Now there is a third possibility that isn't illustrated here.
And that is when you have a large confidence interval that contains zero.
There the conclusion is we don't have enough data, and we can't say anything, unfortunately.
It means that the true treatment effect could be zero, or could be something large and positive,
or could be something large and negative.
We just don't know.
We don't have enough data to determine.
So those are the three ways that you can interpret confidence intervals in practice.
And every interval, every paper and every research you see will have the same interpretation.
So, based on these data that we have seen here, this statistically significant -2.21
effect of being in treatment on positive depression screen, and this precisely estimated zero
effect of being in treatment on cholesterol, the authors conclude that Medicaid had a positive
effect on depression, meaning it lowered depression.
That's a good thing, but no effect on Total Cholesterol.
Now, there are other columns here that I haven't talked about.
This third column comes from something called instrumental variables analysis that we'll
talk about later on.
Column 2, what I have been calling average treatment effects this whole time, comes from
intention-to-treat analysis.
Remember, which is there are non-compilers in this data, there are people in the treatment
group who actually didn't get Medicaid, and all of this analysis I've showed you here
actually counts those people as having Medicaid.
So that's why up here they actually write “Effect of Lottery Selection.”
So these are average treatment effects not of having Medicaid, but of being selected
to have the option of enrolling in Medicaid.
So really, the proper interpretation of this is that being given the option of enrolling
in Medicaid lowers the average depression by -2.21.
In this column here with instrumental variable analysis, they're going to get the actual
effect of having Medicaid, not just the effect of being given the option, but we're going
to do a much more complicated analysis to get to these numbers later on, and you can
see right here down in the footnotes here that's where they say “Column 3 reports
the local average treatment effect,” remember I mentioned that earlier on, and “Column
2 reports intention-to-treat estimates and their 95% confidence intervals.”
So these are the main results from that Oregon study.
We'll get back to this column later on when we talk about instrumental variable analysis.
[MUSIC].