Reproducibility is the ability of an entire experiment or study to be reproduced, either
by the researcher or by someone else working independently. It is one of the main principles
of the scientific method and relies on ceteris paribus. The result values are said to be
commensurate if they are obtained according to the same reproducible experimental description
and procedure. The basic idea can be seen in Aristotle's dictum that there is no scientific
knowledge of the individual, where the word used for individual in Greek had the connotation
of the idiosyncratic, or wholly isolated occurrence. Thus all knowledge, all science, necessarily
involves the formation of general concepts and the invocation of their corresponding
symbols in language. Aristotle′s conception about the knowledge
of the individual being considered unscientific is due to lack of the field of statistics
in his time, so he could not appeal to statistical averaging of the individual.
Reproducibility also refers to the degree of agreement between measurements or observations
conducted on replicate specimens in different locations by different people, as part of
the precision of a test method.
History
The first to stress the importance of reproducibility in science was the Irish chemist Robert Boyle,
in England in the 17th century. Boyle's air pump was designed to generate and study vacuum,
which at the time was a very controversial concept. Indeed, distinguished philosophers
such as René Descartes and Thomas Hobbs denied the very possibility of vacuum existence.
Historians of science Steven Shapin and Simon Schaffer, in their 1985 book Leviathan and
the Air-Pump, describe the debate between Boyle and Hobbes, ostensibly over the nature
of vacuum, as fundamentally an argument about how useful knowledge should be gained. Boyle,
a pioneer of the experimental method, maintained that the foundations of knowledge should be
constituted by experimentally produced facts, which can be made believable to a scientific
community by their reproducibility. By repeating the same experiment over and over again, Boyle
argued, the certainty of fact will emerge. The air pump, which in the 17th century was
a complicated and expensive apparatus to build, also led to one of the first documented disputes
over the reproducibility of a particular scientific phenomenon. In the 1660s, the Dutch scientist
Christiaan Huygens built his own air pump in Amsterdam, the first one outside the direct
management of Boyle and his assistant at the time Robert Hooke. Huygens reported an effect
he termed "anomalous suspension", in which water appeared to levitate in a glass jar
inside his air pump, but Boyle and Hooke could not replicate this phenomenon in their own
pumps. As Shapin and Schaffer describe, “it became clear that unless the phenomenon could
be produced in England with one of the two pumps available, then no one in England would
accept the claims Huygens had made, or his competence in working the pump”. Huygens
was finally invited to England in 1663, and under his personal guidance Hooke was able
to replicate anomalous suspension of water. Following this Huygens was elected a Foreign
Member of the Royal Society. However, Shapin and Schaffer also note that “the accomplishment
of replication was dependent on contingent acts of judgment. One cannot write down a
formula saying when replication was or was not achieved”.
The philosopher of science Karl Popper noted briefly in his famous 1934 book The Logic
of Scientific Discovery that “non-reproducible single occurrences are of no significance
to science”. The Statistician Ronald Fisher wrote in his 1935 book The Design of Experiments,
which set the foundations for the modern scientific practice of hypothesis testing and statistical
significance, that “we may say that a phenomenon is experimentally demonstrable when we know
how to conduct an experiment which will rarely fail to give us statistically significant
results”. Such assertions express a common dogma in modern science that reproducibility
is a necessary condition for establishing a scientific fact, and in practice for establishing
scientific authority in any field of knowledge. However, as noted above by Shapin and Schaffer,
this dogma is not well-formulated quantitatively, such as statistical significance for instance,
and therefore it is not explicitly established how many times must a fact be replicated to
be considered reproducible. Reproducible data
Reproducibility is one component of the precision of a measurement or test method. The other
component is repeatability which is the degree of agreement of tests or measurements on replicate
specimens by the same observer in the same laboratory. Both repeatability and reproducibility
are usually reported as a standard deviation. A reproducibility limit is the value below
which the difference between two test results obtained under reproducibility conditions
may be expected to occur with a probability of approximately 0.95.
Reproducibility is determined from controlled interlaboratory test programs or a Measurement
systems analysis. Reproducible research
The term reproducible research refers to the idea that the ultimate product of academic
research is the paper along with the full computational environment used to produce
the results in the paper such as the code, data, etc. that can be used to reproduce the
results and create new work based on the research. Psychology has recently seen a renewal of
internal concerns about irreproducible results. Researchers explained in a 2006 study that,
of 249 data sets from American Psychology Association empirical articles, 73% of contacted
authors did not respond with their data over a 6-month period. The first author published
a paper in 2012 suggesting researchers should publish data along with their works, releasing
a dataset alongside as a demonstration. Reproducible research is key to new discoveries
in pharmacology. A Phase I discovery will be followed by Phase II reproductions as a
drug develops towards commercial production. In recent decades Phase II success has fallen
from 28% to 18%. A 2011 study found that 65% of medical studies were inconsistent when
re-tested, and only 6% were completely reproducible. In 2012, a survey done for Nature found that
47 out of 53 medical research papers on the subject of cancer were irreproducible. John
P. A. Ioannidis writes, "While currently there is unilateral emphasis on 'first' discoveries,
there should be as much emphasis on replication of discoveries." The Nature study was itself
reproduced in the journal PLOS ONE, which confirmed that a majority of cancer researchers
surveyed had been unable to reproduce a result. Attempts to reproduce studies often strained
relationships with the laboratories that were first to publish.
Noteworthy irreproducible results Hideyo Noguchi became famous for correctly
identifying the bacterial agent of syphilis, but also claimed that he could culture this
agent in his laboratory. Nobody else has been able to produce this latter result.
In March 1989, University of Utah chemists Stanley Pons and Martin Fleischmann reported
the production of excess heat that could only be explained by a nuclear process. The report
was astounding given the simplicity of the equipment: it was essentially an electrolysis
cell containing heavy water and a palladium cathode which rapidly absorbed the deuterium
produced during electrolysis. The news media reported on the experiments widely, and it
was a front-page item on many newspapers around the world. Over the next several months others
tried to replicate the experiment, but were unsuccessful.
Nikola Tesla claimed as early as 1899 to have used a high frequency current to light gas-filled
lamps from over 25 miles away without using wires. In 1904 he built Wardenclyffe Tower
on Long Island to demonstrate means to send and receive power without connecting wires.
The facility was never fully operational and was not completed due to economic problems,
so no attempt to reproduce his first result was ever carried out.
See also References
Turner, William, History of Philosophy, Ginn and Company, Boston, MA, Etext. See especially:
"Aristotle". Definition, by International Union of Pure
and Applied Chemistry Further reading
Blow, Nathan S.. "A Simple Question of Reproducibility". From the Editor. BioTechniques 16: 8. doi:10.2144/000114117.
"It is interesting to note that at this moment of greater irreproducibility in life science,
journals continue to minimize the space given to Materials and Methods sections in articles."
Timmer, John. "Scientists on Science: Reproducibility". Arstechnica.
External links Reproducible Research in Computational Science
Guidelines for Evaluating and Expressing the Uncertainty of NIST Measurement Results; appendix
D Definition of reproducibility in the IUPAC
Gold Book Reproducible Research Planet
ReproducibleResearch.net cTuning.org - reproducible research and experimentation
in computer engineering