Hey, everybody.
My name is Will Sutherland, and I
am a PhD student at the University of Washington
here in Seattle.
And I'm going to talk about reproducibility.
But specifically, I'm going to talk about how reproducibility
is understood in practice through a set of data tools
and data work practices that are emerging
in the sciences around computational reproducibility.
And this is a project that I worked on
with my wonderful collaborators, Melanie Feinberg, Sarah Beth
Nelson, Mohammed Jararahi, and Arcot Rajaseker.
And it centered around a workshop
that was meant to teach these sorts of data tools
and practices to researchers in the natural sciences,
for the most part, mostly PhD students and postdocs
in fields like genomics, bioinformatics, physics,
material sciences, chemistry.
There we're also a couple of computer scientists as well.
And it included things like using the cloud with Amazon Web
Services, versioning with git, Jupyter notebooks,
and containers, which I'll talk about a little bit more.
And I engage this-- we engage this phenomenon by--
I actually participated in the workshop as a student.
So I sat through all of the class sessions.
I did the homework.
I was on a project group.
We reproduced, an analysis as a group project.
And Sarah Beth Nelson also was observing through all the class
sessions, and so we were able to compare notes.
And I would love to talk more about the sort of methods
that we used on this to get to the findings
that I'm going to show you.
But for time's sake, I'm going to leave that to questions
afterwards, if you have questions about it.
And also, please go read our paper.
So one of the things that emerged in this workshop
was an understanding of reproducibilty
as a kind of recapitulation of a finding.
And so this was really taking a computational analysis
and running it again, from start to finish, under as much as
possible the same circumstances as before.
And so this is one of the participants speaking
in this kind of mode about reproducibility.
So he's saying that another researcher should
be able to take my data, which is open source and publicly
available, run through all the analyses,
and produce the same outputs.
And this very much reflects a notion of reproducibility
that's been discussed in the literature
as direct replication.
And it's really running the exact same analysis
under the same circumstances and verifying that a result has
come out of that.
And this was very closely tied to containers, these container
technologies, because what a container will allow you to do
is to take a piece of software or a workflow and a software
analysis, and stick it in a box along with all
of its dependencies environmental factors
that need to be set.
And then you can just hand that box off to somebody else,
and they can rerun the analysis from start to finish.
And so this was an understanding of reproducibility-- really
enacted an understanding of reproducibility
as a kind of automation, of rerunning
the exact same analysis and verifying the result that
comes out at the end.
But there is another understanding
of reproducibility that emerged in the workshop that was much
more centered around reuse.
And so this was about being able to hand off
a workflow or computational analysis to another researcher
and have them use it for slightly
different circumstances, to do the same kind of analysis
again, to use it as a tool in some sense.
And this was much more closely related
to what Schmidt calls a conceptual replication as
opposed to direct replication, and what Cartwright simply
calls reproducibility as opposed to replication.
And so this understanding came very directly
out of the kinds of laboratory and research group environments
that these participants were working in.
And so this is one of our participants explaining
an understanding of reproducibility
in this sort of mode.
"We have difficulty reproducing people's work,
and that's a very standard problem, I think,
that goes across fields.
It's challenging in my lab to have
knowledge transfer between generations of students
and post docs because of the size of my lab
and because of the structure of it."
And so what he was very much concerned
about is this problem of handing off an analysis, a tool from--
across generations of grad students in a high-turnover lab
or between labs, and enabling this analysis
to be used in future work, to build off of that analysis.
So this was very much a notion of reproducibility
as a kind of reuse.
And participants in the workshop were
able to switch between these two understandings
of reproducibility fairly unproblematically.
They didn't see a whole lot of-- they
didn't really distinguish between them
very strongly at all.
And so our point here is not so much
that the term "reproducibility" can be an ambiguous term.
I think that's been observed and that there's
been a lot of work trying to delineate
what is reproducibility.
But our point here is that when we build tools and systems
for reproducibility, a lot of times
they're motivated by this notion of a reproducibility crisis.
But what we're seeing in this workshop
is that what people are really interested in these tools
in a lot of cases has much more to do
with a crisis of having to collaborate around these really
large tangled code bases that are developed by PhD students
and people who are coming and going from research labs.
And so what we want to point to here
is the way that when you build tools for reproducibility,
they enact certain understandings
of reproducibility, and they enact
certain criteria for what constitutes
good scientific work.
So when we build a tool for reproducibility,
are we trying to support a rerunnable analysis or more
a reusable analysis?
And how does our-- how do computational tools
and reproducibility enact these different understandings?
So I hope that there are a lot of questions about this.
And please, again, go read the paper.