[MUSIC] So, now let's turn to a more practical question: how do you actually read an empirical
research paper?
Well, the first secret trick is that you don't read it from beginning to end.
You do not read a paper like you'd read a book, or a newspaper, or anything else, and
for that reason, authors don't write papers like they're meant to be read from the beginning
to end, because they know that the readers aren't going to do that.
So what should you do?
Well first you want to skim the paper to find out four main things.
First: what is the point of the paper to begin with?
What's their main question?
What causal effect are they trying to get at?
Second: what's their main result or their conclusion?
What's their main finding?
Did they answer the causal question they were trying to get?
Third: what's their data source?
Where's their data coming from?
What data are they using?
And fourth: what method or identification strategy did they use to draw their conclusion?
So this first step, where you just skim the paper to find out these four main things,
can usually be done in about five minutes or so, especially if the paper is well written
and if you are very familiar with the particular question in the context, like say the effect
of preschool.
Now, if you're not familiar with the context or the methods, or the paper is not so well
written, it could take a little bit longer, and sometimes debates about the pros and cons
of papers can last for years.
And indeed trying to debate these things and figure out exactly what's going on is part
of the point of ongoing research.
So you are just trying to get a basic idea what the topic is, what the findings are in
your first skim, you don't have to understand everything.
The next thing we want to do is know what the structure is or the typical outline of
a paper.
So it's going to start with the introduction, then it'll maybe talk about the mathematical
model or just a theoretical discussion of the substantive problem they are trying to
tackle.
Next they will talk about the data sources, what data they are using, and where it comes
from.
Then they will talk about the institutional details and give you some discussion of the
particular context of the data and the policy or the treatment.
Then they will give you some summary statistics of their data; you know, some means, some
standard deviations, some mins and maxes, just to give you an idea of what the data
looks like.
Then they will talk about the empirical strategy: what are they going to use to learn about
their main effect?
Which of the identification strategies we've talked on this modules will they use?
Then their main result: what's the main finding, the main take-away from the data analysis?
After that they will give you some other results, stuff that's not as important or interesting
as the main result, but still worth writing up and talking about.
Then they will talk about some robustness checks, trying to explore different alternative
explanations for their main finding, where here really trying to convince you that they
actually found a causal fact of the intriguing variable on the outcome variable.
Finally, they will end with a conclusion, where they’ll just wrap things up and reiterate
what their main finding is, talk about other broader implications, etc.
So that's the basic outline of the paper.
Not all of the sections are always going to be present and they are not always going to
be in that order.
So personally when I pick up a paper, I usually start with the empirical strategy and their
main results, and then I'll work backwards to these other sections.
So put differently, I'm going to start by figuring out what their outcome variable is,
what variable are we trying to learn the causal effect on.
Second, what is the treatment variable?
So we want to know the effect of a treatment variable on an outcome variable.
So what are those in a particular paper?
Then I'm going to look for a table or graph that summarizes the main causal fact that
they find.
Usually, there will be just one table or graph that does that.
For example, just like all the ones we've looked at in these modules.
So once you find a that main table or graph, you can read off what their main finding is,
assuming that everything they did was correct.
So the next - now that I've got the main finding, and I know what they're talking about, I want
know how do they get this, how did they come to this conclusion, how did they get that
number?
That is, what was the identification strategy?
So which of the methods we've talked about in these modules are they using?
So here's where you want to look for keywords like natural experiments or instrumental variables
to tip you off on what method they're using.
Now here you can often stop reading and go do something else if you're not interested
in the further details, because you've got the main point in the paper.
You know their outcome variable, you know their treatment variable, you know what causal
effect they found, and that's pretty much the main point of all research in empirical
social science.
So you can often read a table by just looking at the figures and the graphs, you don't even
have to read the body text.
If you can just find that main table or graph, you're basically done.
Now, suppose you actually are interested in more details about finding or institutional
context - then you want to read on.
The next thing I'll look at is the data sources.
Where do they get their data from?
What exactly are they using?
How do they define their variables?
And then what's the relevant institutional context that I should know about - you know,
the settings about how things actually worked in practice?
This step's going to involve looking at the other tables, like the summary statistics,
and trying to build a context for the main causal effect.
So for example, if the main effect is something about the effect of a policy on students'
GPA, you want to know what's the meaningfully large increase in GPA, so that is, how much
variation is there in GPA in the data?
The summary statistics table will tell you that.
So should I be looking for a 0.2 unit increase in GPA?
Is that a large increase or is it a 1 unit increase in GPA?
I don't know.
That's what we have to find out by reading these other sections.
After you've done that, I will look at the robustness checks.
These are different analyses that are trying to look alternative explanations for the finding.
They are going to claim “I found the causal effect of a policy and an outcome,” and
you might say, “Well, couldn't it be this?
Couldn't be that?”
Maybe there actually isn't a causal effect?
Those are things you going to try to explore in this section, where their main point is
going to be to convince you that their actual main finding is correct.
So last, last but not least, we're going to look at the introduction, the conclusion and
the theory section.
So in the introduction and the conclusion, they are going to try to put their finding
in a broader context of the literature and what we know in general about the effect of
various policies in these settings.
And in the theory section they are want to try to give you a way of thinking about the
situation, thinking about how people are behaving and making choices, and how policy may actually
work in practice.
So the one reason you leave these to last is that often if you know a lot about the
situation, they're just going to be repeating things that you already know, so that's one
reason why people often just skip it.
So this section you want to read, especially if you don't know a lot about the previous
literature and about the context for this finding.
But if you're familiar with the topic you can just skip them, go straight to the main
findings.
So that's an overview of how you should actually read empirical papers.
[MUSIC].