Hi everyone, I'm Simine Vazire, I'm in the psychology department here at UC Davis and
I will be moderating the first session which is on defining the issues,
research on replication, reproducibility, and transparency.
And the goal of this first session is gonna be kinda of to set the stage for
the rest of the day And
discuss where we're at in terms of replicability in the social sciences.
So our first speaker is Garret Christensen.
And Garret Christensen is a Data Science Fellow at the Berkeley Institute for
Data Science and staff scientist with the Berkeley Initiative for
Transparency in the Social Sciences.
A program of UC Berkeley's Center for Effective Global Action.
He received his PhD in economics at UC Berkeley in 2011 and has since conducted
research on the wash benefits public health randomized trial for innovations in
poverty action at Emery University and has taught economics at Swarthmore College.
He is interested in research transparency and reproducibility and
questions of causal inference in labor and development economics,
particularly with regard to child health and
education Programs in both developed and developing country settings.
He has also run more than 50 ultra marathons and
walked from border to border across the entire United States four complete times.
So, please join me in welcoming Garret Christensen.
>> [APPLAUSE]
[INAUDIBLE]
Hi everybody.
Simine, we've got 20 minutes each,
the three of us or something like that.
So I will introduce you to sort of the problems of publication bias,
and p-hacking, and a potential solution in the form of pre-registration.
So, sort of to introduce this whole thing, I like to frame this quest, well.
One second, yeah.
So I'll introduce sort of, I like to frame this in terms of
transparent research being ethical research.
I'll talk about registration and pre-analysis plans.
I'm from BITTS, which is the UC Berkeley Initiative for
Transparency in the Social Sciences.
We're one of several organizations that have been built up in the last few
years that are trying to increase transparency in research,
particularly across the social sciences.
And we're happy to be here today.
So, I view transparency as part of being an ethical researcher.
A sociologist of science, Robert Merton,
came up in the forties with sort of agreed upon values of science.
The one that sort of Is most relevant to us.
He actually called it communism but [LAUGH].
Commonality is just this agreed upon norm of open sharing of knowledge.
And I think we could do something to get back to that.
There's Pretty good evidence for why we might want to worry about this.
If you ask researchers in the.
Okay, didn't wanna do that.
OK.
This one's blue for some reason but, so
if you ask researchers do they subscribe to these
subscribe to norms of data sharing everybody says yes but
then if you ask researchers like do you actually
practice that yourself less somewhere about 60% but
then if you ask them more what do you think other
people are doing it's below 10% so
that's probably people overestimating their own conforming to the norms.
And again, if you ask researchers, have you ever
failed to report all your dependent measures, collected more data after seeing
whether results were significant, failed to report all your conditions,
stopped data collection after achieving the desired result, rounded down P values,
Etc., down to the inner circle of hell, falsifying data.
People say yes, that they've done,
so surprisingly large amounts of people say that they've done this and
assume that a lot more people are actually doing it even more.
So And sort of just outright fraud or just poor statistical purposes which,
honestly, I'd like to believe that is not most of the problem.
Fraud.
There's just not
deliberate problems maybe, but other existing problems publication-wise.
There's evidence that publication bias exists.
Effect sizes diminish with sample size.
There's a higher fraction of rejected hypothesis tests in social compared to
hard sciences.
No results are disappearing over time across all disciplines.
And data on the complete sets of experiments round so
if you have the universe of studies that were run on a certain
topic there is evidence that 40% points more likely stronger out of 40% more
likely to be published and 60% more points more likely to be written up.
So in these publication bias there's evidence that it exists across all fields,
medicine, and social sciences in general, economics, sociology,
and political science.
So in medicine, you can, because of Registries were
people have to declare all the studies that they're doing on a certain topic.
You can go back after a while and say okay, of all these studies that we know,
this is the universe of studies, how many of those got published?
And so this is from Turner Et Al in 2008 and
they're looking at anti-depressant clinical trials and.
Basically it gets published if it's a positive result and
it doesn't get published if it's a negative result.
And this is an RCT with pretty clearly defined treatment so
might be worried about that.
This is a science paper by Franco et al.
Mention the results.
This is from the time share experiments, which is
a NSF funded, nationally represented phone survey.
And if you want, you can apply to get your set of questions Added to the survey so
that folks at Michigan who run the experiment they have the universe of every
set of questions that got put into the survey and
it's all ready passed some handle of quality to get into this survey And
then they find that again the null results don't get published.
They don't even get written up.
And the strong results, they all get written up and most of them get published.
Of course there' some endogeneity in what gets written up based on your
expectation of what's gonna get published, but still pretty concerning results.
In economics, if you,
collected z-statistics from several
years of the top journals and found this sort of disturbing
In Z statistics just below the significance cut off and
they sort of backed it out that 10 to 15% of the marginally
signficant statistics should actually be on the insignificant side.
Same is true, sort of if you do a caliper test.
If you look in a narrow bandwidth on either side of the significance cut off
in top sociological journals, and top political science journals,
you get the same Unlikely to represent the truth outcome.
So, that's publication by us that if we only know, if we only write up or
publish significant results and we have no record of the insignificant results,
we have no way to tell if our supposedly significant results are real Or
if they're just the 5% that we should expect due to noise,
how statistical testing works.
So another related problem is p-hacking,
it's not always clear what's p-hacking and what's publication bias.
But p-hacking also goes by the names of fishing researcher, degrees of freedom.
Specification searching, data mining, data-monging.
Lots of names for it.
But this is, when flexibility and data analysis
allows portrayal of almost anything as below and arbitrary.
P value threshold.
Significant so that it loses its meaning.
And this isn't something only evil people do.
This is subconscious.
There's lots of psychologists in the room who can tell you that humans are Really
good at motivated reasoning and so you're going to run a bunch of specifications and
then finally you find one that gets you the 0.4 and
that's the one that you really wanted to run all along and that one is best
clearly and you can pretty easily convince yourself of that because you're human.
Evidence of this, 538 did a great piece on this where, depending on how
you define political parties, and whether you look at the Senate or the House or
governors or the president, you can find, sort of, whatever result you want
in explaining which political party leads to The best economic growth,
whether it's just define your outcomes differently, whether it's unemployment, or
increasing GDP, you can get any statistically
significant results you want.
Joe Simmons, Leif Nelson, and
Uri Simonsohn are great psychologists working in this area.
They have this paper about undisclosed flexibility where they prove,
using a randomized trial, that listening to the Beatles song when I'm 64 actually
makes you younger, because there's just enough,
there's enough Flexibility in which specifications you run and
how you write your models that you can quote unquote prove that.
So we view a solution to this, a potential solution to this is registration.
So Registration is publicly stating the research you'll do and
what hypothesis you'll test prospectively.
This has near universal adaption in medical randomized trials.
It got that way because the top journals,
the international consortium of medical journal editors Basically saying,
we're not going to publish your RCT unless it's registered.
This has very strong adoption through clinicaltrials.gov.
It would be even better if these registries had outcomes and
some medical trials, some types of medical trials,
are required to report outcomes but the reporting.
It's a very small fraction of them and the reporting is actually not enforced so
strongly But the N.I.H., last fall, has taken steps to sort of strengthen that.
So, this same sort of registration is taking off in the social sciences.
It's relatively new but if your an economist doing a R.C.T.
you can register it at socialscienceregistry.org EGAP,
which is experiments in governance and politics.
That's the political science registry.
There's 3ie, which is the International Initiative for Impact Evaluation.
You're doing impact evaluation in developing countries.
They've got a registry.
And the Open Science Framework, built by the Center for Open Science.
Has a open format registration system.
So you can basically just upload the materials saying,
these are the hypothesis I'm going to test, hit the register button and
you can create a frozen timestamped version of your analysis plan
And share that when you're ready, but it'll be frozen and time stamped so
that you can't edit but you get a verifiable time stamp with it.
So registrations in the American Economic Association registry started in 2013, and
there's over 500 500 registrations so far.
This is still orders of magnitude below clinical trial, but
we're happy this it's continually increasing.
There's another type of registration which is designed based publication.
This is also called registered reports.
I think we sort of overused the word registered in our team but
I call this design based publication or results blind review.
That's basically we move peer review
before data gathering results and analysis.
So you design a project And then you submit it to the journal
and then you get reviewed based on the importance of the question and
the quality of the design, you get in-principle acceptance and
then if you follow through, your known result,
whatever you get Would get published, as long as you follow through on the design.
So this is, so if our twenty journals have done this, if you more or
have published special issues, basically this is what's going on.
I think that's an interesting model that's worth testing, mostly it's so
far it's in psychology journals because
Psychologist Chris Chambers is the one who developed that model.
Real quick, as part of the registration you can include a pre-analysis plan, which
is a detailed layout of the statistical analyses you're planning to include.
So this started with this is sort of expected with a medical trial,
that you're registering with the FDA.
But in social sciences you might be, we've got
We encourage sort of more detail where you'd lay out all these things,
the description of the sample, the hypothesis to be tested,
how you're going to do multiple hypothesis testing,
how you're going to group hypotheses together as families,
if you We're gonna build a formal model as economists do.
Include that model, and then remember to archive it.
So you include the statistical analysis plan in the registrations.
You write this before you do the study.
Examples of these All these slides are online,
by the way, since I'm over time already.
But, one last thing is that it prevents us from doing this.
This is a paper by Casey, Glennerster, and Miguel in
the Quarterly Journal of Economics and this is the cherry-picking table.
They did a large community driven development
project in Sierra Leone and community driven development is by nature
hard to define because you let the communities decide what projects they want
so They could have found because there are so many outcomes that they measure.
They could have found that this program that was named Golby Fo,
they could have found that it weakened institutions,
they could have found that it's strengthened institutions, but instead,
since they had written the detailed statical pre-analysis plan,
they found a big fat zero And they were under sort of no pressure to
cherry pick that into something other than what it was.
You could even do this with observational studies,
cuz most research is observational research.
There's an example from Over a decade ago from economics.
Because you know in the US that the government is going to release data
on a reliable schedule, before that data comes out you can say,
I'm going to run this specification on next year's minimum wage data.
And people will Maybe take your results a little more seriously.
So why would you wanna do this?
Because it's good science, you can get a badge.
Some journals are giving badges and because you could get $1000 if
you go through this preregistration challenge with
The Center for Open Science.
That is all. >> [APPLAUSE]