[MUSIC] So oftentimes when there's non-response in surveys, people just ignore it, and hope
for the best.
That's probably the most common thing that people do.
Now, that's valid if you think that the people who respond are the same as the people who
don't respond, and there are just random response rates.
But if you're worried about whether the people who respond are different from the people
who don't respond, then you need to start thinking a little harder.
So in the Oregon Experiment, one thing that people can do is take values of variables
for the non-responders and compare them to the values of variables for responders.
If there's truly random response, because people randomly decided to respond or not,
then the values of these variables on average should be the same.
The responders should have the same average variables as the non-responders.
Just like for the same reason that when you randomly assign people to treatment and control,
the average values of their variables across the two groups should be the same, there should
be balance.
So that's one possibility, but of course, the problem with that is you need the average
values of variables of non-responders.
But where do you get that from because they're non-responders - they didn't respond?
But in some cases, if people have a complex survey, there may be multiple stages of the
survey, and you may have people who respond early on and then stop responding later on,
so you can compare values for the first stage with the responders and the non-responders,
but you can't compel them at some point, because at some point people stop responding.
So that's one thing you can do, if you have some data on non-responders.
If you don't have data on non-responders, then one thing you’re left to do is say,
well okay, I know that I'm not going to be able to learn anything about non-responders,
let's just see if we can learn something about treatment effects for people who respond.
Now the problem is in experiments is: are the people who respond in the treatment group
the same as people who respond in the control group?
That is: is that bias, the non-response bias, balanced across the two groups?
So that's something you can check in your data, and that's what they did here in the
Oregon Experiment.
So here's Table S7 from the appendix to this paper.
In Panel A, they just have data on response rates.
So up here we say, how many people responded to the survey?
Well in the control group, there was 73%, and in the treatment group, there's some other
number, but what they've presented here in this column is the difference between the
average value in the control group and the average value in the treatment group.
Lottery winners, those are the treatment group.
So the difference in those two means - 73 minus whatever the mean in the lottery winners
group - is 0.28.
So basically it is 73% in both groups, so there is no difference in response rates.
So that's one variable that we see that's common across the two groups.
Now, down here in Panel B, they have a lot of variables: age, female, black, other race
is Hispanic, so some race is Asian, English is the preferred language, address is P.O
Box etc, etc.
And again, they have the average value of these variables in the control group listed
here, and over here the difference between this number and the average value in the Lottery
Winners.
So in the control group, the average age is 40 years or 41 years, and in the treatment
group, the difference between this and the average age of the treatment group is 0.2.
So again, the average age looks basically the same.
So what they're going to do is compare all of these variables in the control group with
the treatment group, and see if they look the same.
So if you do that comparison, you get these numbers here, and we see that basically it
looks the same for age.
The difference is 0.2, which is very minor, in terms of age difference.
A big difference I would say, average age is 40 in the treatment group, and average
age is 10 in the treatment group - this is the control group, sorry.
That would be a big difference.
And that would be a sign that the people who responded in the control group were different
than the people who responded in the treatment group because their ages are different.
But since we see that they are not different we have a little bit of confidence that these
two groups are similar, and for whatever reason people are not responding on the survey, it's
the same across both groups.
So now I'm going to talk about some of the other numbers on this table.
So here, these numbers here are what we call the point estimates.
Those are just the averages you compute.
You get your data, you add up everybody's age and you divide by the number of people.
This number here in the parentheses is called the standard error.
And it's a measure, if you remember from our earlier modules, of the variation in age.
So it just tells us what are the largest number of oldest people in our group, or what is
the smallest, and what's the distribution of that.
So it's just a way of summarizing the data and telling us how much variation there is.
And they give that for every single one of these.
Over here in parentheses, we get something that's called a 95% confidence interval.
And again that's a way of measuring how much uncertainty there is in these numbers.
So here the 95% interval is -0.28 to 0.69.
Now remember the way you want to interpret these intervals is basically 2 things.
If they include 0, then we say that there's no difference - if we include 0 and the interval
is very, very tiny, then we say that there's no difference between the control group and
the treatment group in their age.
If they contain 0 and they're very big, say suppose it was -40 to 40, that's pretty big
compared to age, you know?
So if that's the case, then you just say we have an imprecise estimate.
We don't have enough data to figure out what the true difference is.
Here, is what's called a precisely estimated zero, because zero is in interval and the
interval is very tiny, and what I mean by tiny that has to be relative to whatever your
variable is.
So is a difference in age of 40 to 50 big, or is that tiny?
Well, that's sort of a subjective judgment call - I say that's pretty big.
Here is a difference of age of 0.2 big or small, or 0.28 or 0.69, that's pretty small,
but again, it's a little subjective.
So that's the way to interpret these intervals, they contain zero and they're very tiny then
you say there's no difference.
If they don't contain zero and they're very tiny, then you say there is a difference.
And here, fortunately for the authors, all of these intervals contain zero and all of
them are quite small, not all of them but most of them are quite small, so they're precisely
estimated.
So from this data, the authors are going to conclude that whatever the non-response bias
is, it's balanced across the two groups, and therefore we can learn the average treatment
effect of having Medicaid for the people who responded.
We would not be able to conclude that if we found that these numbers were very big.
And therefore there's a big difference in these variables between the two groups.
By the way, here's a good example of an imprecisely estimated number.
Actually no, I'm wrong.
This's where you gotta be careful.
You gotta remember the scales.
So here's median income.
So income, this is dollars, $44,000.
And so these numbers, this is dollars, $400, -400 to about positive 400.
So it's actually a pretty small number relative to overall dollars.
So if you don't look at what kind of variable it is, you might think, oh compare, look 400
to 0.3?
Wow, this must be imprecise.But actually it is pretty precise given the scale of this
number here.
So, that's one way you can check for, deal with non-response bias.
But again, unless you assume that the responders are the same as the non-responders, you're
only going to be able to get the average treatment effect for the responders.
[MUSIC].