Hey guys, in this video I'm gonna talk about the difference-in-differences experimental design.
Also there's a really famous paper in econometrics that used it, which was the Card and Krueger,
1994 paper on minimum wage and I also thought the paper was really interesting or like the
theme of the paper because I only skimmed through the actual paper but the, you know,
the conclusions that the paper got were really interesting and so I thought I would make a video
on it. In this video, I'm going to first talk about what pooled cross section data is and the
definitions of different data types because pooled cross section data is what's used in
difference-in-differences. Then I'm gonna go over how you do difference-in-differences and then I'm
going to talk about the Card and Krueger paper at the end. For those who don't know already,
this was a paper talking about how--or trying to look at the effects of raising the minimum
wage and how that affects unemployment. They found that there was no effect on unemployment,
which was controversial to say the least, like that's probably still a controversial idea even
today. It's kind of counterintuitive and a lot of economists hate this paper and so a lot of people
have made rebuttals. At the very end I'm going to talk about that and I'm just going to call it etc.
because Card and Krueger have made rebuttals to the rebuttals since then as well and they have a
lot to say about this topic. We'll get into all of that at the end. Let's first talk about what
pooled cross section data is exactly. Pooled cross section data has flavors of cross section data,
obviously, and it has also flavors of time series. Cross section data has no regards for time as a
variable, so no time variable. Everyone pretty much knows what this type of data is because I
think it's one of the--it's one of the most common data collection methods. You just can
take a survey of someone or do something to someone one time and you don't care if they
would do it differently another time. It's what cross section means, right? You just get a slice
and you don't get more than one. Time series, it's kind of the flipside of that. It's when
you collect data at different time points and kind of like periodically. Most famously the Dow Jones
you can have a stock here one time point then it goes up, then it goes down, then it goes up high,
oh then it plunges, and then it goes like that, and so this, you're following it overtime. To
combine these two, it's kind of like if you take multiple cross sections and if certain assumptions
hold, you can pool these cross sections together and kind of make a cylinder of cross sections,
if you will. And so this might be 2004 data, 2005, and again if certain assumptions hold, which I'll
talk about later, you can pull these together. The benefit of doing pooled cross section data
would be that your N increases. Your sample size increases if you can do that and that would be
beneficial for lots of reasons, like statistics and stuff. Okay so difference-in-differences is
a really simple way of doing pooled cross section data. Let's get into that now.
Difference-in-differences, it's literally if you just look at two groups at two different
time points. It's really simple. The DID method is often used for experimental or pseudo-experimental
data. In econometrics, you usually have, you know, a big policy change or some kind of natural
disaster or something that you can measure the effect changes between pre-event and post event.
The two groups that we talk about are usually a control group and a treatment group. And then we
have pre-treatment and post treatment as the two time periods. Let me give you--let me do a little
drawing and then I'll discuss like how you can look at it graphically and an important assumption
for DID. Okay so take this graphic as an example. We have a control group and a treatment group and
we took a measurement in 2014 and 2015. And as you can see here, say for example,
if you have a pill/supplement and you give it only to the treatment group and this pill supposedly
you're trying to see if it helps people grow their hair faster. And so you take measurements
of hair length in 2014 and 2015 but you tell both groups not to cut your hair and you tell only
the treatment group to take the pill. You can see here that the slope for T, or the treatment group,
is steeper than the control group although both groups had you some hair growth. Well,
if you did the DID method correctly and all the assumptions hold, then you can reach the
conclusion that the pill helped grow hair faster. One of the assumptions if you're going to do that,
you need all the assumptions that you would need for ordinary least-squares. You need the zero
conditional mean assumption, all your variables have to be exogenous, you need normality, all of
that. In addition, you need the-- what's called the parallel trend assumption. This would hold
true only if you're assuming that the treatment and control groups would otherwise grow hair at
the same rate. If for some reason your treatment group was growing hair faster for another reason,
then this would not--the pill wouldn't necessarily have made people's hair grow faster. You're
assuming basically that their hair would have--if I can draw a parallel line--it would have been.
No, it looks like I can't. You're assuming that the hair would have grown. Let me try one more
time it's so hard on this--like that. Okay, you're assuming that the treatment group would have grown
at a rate similar to the control group and this extra additional growth is therefore caused by
the pill. And so how do you prove that they're growing at the same rate? There are two ways that
I know of. You can make sure to take measurements before the two time periods that you tested to
show that the rates are both stable and that the treatment and control groups are the same rate.
You can also get a second control group and see that they also are going at a parallel rate. It
was easier for me to draw the line that time for some reason. And so if you have two control groups
and a treatment group, then you can say, yeah they were growing at the same rates. I have two
control groups. You can compare multiple groups or compare more times, basically, and that will
cover your parallel trend assumption. Now let's see the DID method written out as, you know, the
equation or the model form and you'll see one of the reasons why it's like a nice and pretty model.
You have your dependent variable. That could be hair length. It could be log wage. It could be
anything. And then the model would be B0, your intercept, plus B1 Treatment plus B2 Post. And
these are--Treatment and Post are dummy variables. And then you have an interaction term B3 TreatPost
plus your errors. This is the variable or the estimate of interest. You want to see,
is the B3 estimator nonzero and if so, your slopes are going to be different and you're
going to have that extra slope for the treatment, hopefully. And then if you do, is it significant?
That's what you want to see for this model. And again, one of the nice things about it and the
reason it's called difference-in-differences is that you can calculate this manually by hand
if you know the sample averages. There are two ways of calculating it. You basically take the
difference between pre- and post-, and treatment and control, and then you take the difference of
those two. Let me write it out and then I'll talk about it. Okay, so this is what it would
look like written out. You take a difference here and you take a difference there, and then you take
the difference of the two differences. That's why it's called difference-in-differences and so it's
a really simple, you know, formula but it's very effective, I think, as a modeling tool. Now let's
go to the third and final part of this video, which is to discuss an application of this method
which was done in the 1994 Card and Krueger paper. Okay, so this is a picture of Card and Krueger. I
don't know which ones which, actually I think this is Card, I think this is Krueger but don't quote
me on that. And this is a screenshot of the title of the paper that we're going be talking about,
so this is the 1994 paper. It's actually freely available online, I think David Card
is a professor at Berkeley and he posted the link to his paper. I'll put the link below but I think
if you just google it, you'll see it as the first link. There are PDFs of it available online. Let
me just talk about what this paper was talking about before we get into the statistics. They
basically were trying to look at the effect of raising the state level minimum wage. This is a
policy change, like pseudo experiment. And they wanted to look at the different--the effect of
it that it had when New Jersey raised the minimum wage, I think it was in 1992 the minimum wage was
raised from four dollars and twenty five cents to five dollars and five cents. The way they looked
at how this was affecting people was that they did survey data before the 1992--I don't remember what
month it was but let's say it was October 1992 that the law was passed. They did a
survey a couple months before the minimum wage was raised and a couple months after. And they
surveyed fast-food restaurants and they asked people about the prices. They asked how many
workers do you have. They measured whether a store was even still open before and after.
They called and they asked a lot of people and they said that they had--because sometimes you
might call someone previously and then the second time you try to follow up with them, you can't,
but I think they had like a 92 or something or more percent successful rate of getting the same
people to answer their surveys before and after. And all this data is in the paper that they talk
about. And actually I'm reading my notes that says there was a seven month difference. There
was a seven months separation between the two calls. Oh I should have gotten a picture for
this. I don't know how to draw. That's a bad drawing but they called basically restaurants
in eastern Pennsylvania, which that's my drawing of Pennsylvania and this is eastern Pennsylvania
and New Jersey. They basically were saying or making the argument that eastern Pennsylvania
and New Jersey are going to have similar economies and this pop--eastern Pennsylvania was acting as
like the control group for this natural experiment and New Jersey was the treatment group. And the
previous time that they picked was before the law was enacted because they knew when it was going to
be enacted and the post was a couple months after the law was enacted. The paper discussed this as
well, the law was passed I think in 1990 or like a few years before the recession was taking place
at the nation level was--at the national level there was a recession and the state congress of
New Jersey almost tried to block the minimum wage raise as a result, because they said it would be
bad for business, but it still got passed. And so they collected data. At least when I was
thinking minimum wage fast food restaurant workers, I was thinking, okay they asked a
bunch of McDonald's restaurants but it actually says what restaurants they picked and actually
McDonald's wasn't in there but I think Wendy's was one of them, Burger King was one, I think,
and they asked four different restaurants. Okay, so as you can see--oh it has the months here. It
was February and November. Okay, whatever. As you can see here, this is a little table
that you can do for difference-in-differences and this is the difference-in-difference B3
estimate in my model. Obviously, one of one of the things that they asked was about employment
and so I think these were measures of full-time employment and they actually found that there
was an increase in employment. Basically, what they argued in this paper was that the minimum
wage raised in New Jersey did not cause a decrease in full-time employment. And they're not arguing
that it increases employment, they're basically saying that it doesn't have a significant effect
because if you have, you know, an effect that's contrary to what you're thinking of,
that it's kind of like counterintuitive and you might think that there's a problem with your
statistical analysis or your experimental design but I think whether this was a significant or not,
I think they just made the argument basically that it has no significant effect on unemployment. Now
they also looked at does it affect price and I think that they did find that some restaurants
in New Jersey did increase their prices but that was not even consistent within New Jersey. I think
southern New Jersey had some price increases but North New Jersey didn't and that's basically what
the paper was saying. Now a lot of people have criticized this paper. I think when I heard about
this paper in class, one of the things my graduate student who told me about it argued was that it
failed the parallel trend assumption. He made the argument that the recession hit Pennsylvania
worse than it hit New Jersey and that's why you didn't see that New Jersey's unemployment fell
worse than Pennsylvania because Pennsylvania's unemployment was just falling at a higher rate
already. Now I think I skimmed it through the paper and they did measure--I think I remember
seeing that they looked at teenage unemployment rates prior to the two measurements that they
got. If they did do that, that would be a measure of trending rates. Now, I don't remember if that's
100% true or not but I think they did do that. To me the more obvious mistake is that they obviously
took measurements really close to when the law was enacted. They literally took it just a few
months prior and just a few months afterwards. For me that's obviously a red flag because, well if
you raise the minimum wage, you're not going to get fired--maybe you're not gonna get fired the
day the minimum wage raises or even five weeks after the minimum wage is raised, but you might
see that difference, you know, you might get fired a year later when they look at their revenues and
see that they can't employ as many people. A lot of people have made rebuttals to this paper and
said they've made that critique and they've made other critiques about the experimental
design of it. A lot of people have said that if it doesn't--if it doesn't affect unemployment,
it affects job growth rate and I saw that a lot when I was skimming through like what people
had to say about it, that if you do increase minimum wage or just wage in general, then that
will decrease job rate. And they also actually looked at, I forgot to mention in this paper,
they also looked at non-minimum wage workers and how raising the minimum wage affected people who
already were working above the minimum wage. And I think that they said that those people's wages
didn't increase. If you increase the minimum wage, that didn't mean your wage was going to increase,
you know, regardless of what your starting wage was if it was above minimum. But anyway going back
to this, I think it's pretty well documented that increasing minimum wage does decrease job growth
rate. Now it's not a one-for-one, you know, it's not proportional. I think I read somewhere that
if you increase by 30 percent the minimum wage, then that only decreases like 0.5 percent the
job growth rate or something like that, but maybe that was just in one specific state. It probably
depends but okay, so that's one argument. Now Card and Krueger have gone back and made a lot
of rebuttals since 1994 defending their results and also like finding more data about unemployment
and the effect of raising minimum wages at the federal level or state level. They in particular,
I don't remember what paper it was but I'll try to find it and maybe put the link in the description
box below if it's available, but I read briefly that they did a meta-analysis. That's basically
when you take--a lot of people study results and then you actually do statistics on the the
statistics that were done in other studies. You take the results of a bunch of studies and analyze
those and they said that basically the reason people think that there's unemployment increase
when you increase minimum wage is because of publication bias. They said that papers that
found no significant effect on unemployment were less likely to get published because there's just
a publication bias in general across science that says, okay, if you're going to get published,
you need to have significant to statistics and p-values and if you don't, you're not going to get
published because it seems like your study--you didn't really discover anything but I haven't
actually read that paper, so I don't know how they could come up with that assumption but that's what
their meta-analysis allegedly says. I'll put as much information about it below as I can.
Actually, I got this table also, by the way, from Wikipedia. I'll put the Wikipedia page link for
difference-in-differences below as well. And if anyone is interested in this econometrics question
about the effect of minimum wage, I'll try to put some resources in the box below as well.