In 2011, a team of physicists reported a startling discovery:
neutrinos traveled faster than the speed of light
by 60 billionths of a second
in their 730 kilometer trip from Geneva to a detector in Italy.
Despite six months of double checking, the bizarre discovery refused to yield.
But rather than celebrating a physics revolution,
the researchers published a cautious paper
arguing for continued research in an effort to explain the observed anomaly.
In time, the error was tracked to a single incorrectly connected fiber optic cable.
This example reminds us that real science is more than static textbooks.
Instead, researchers around the world are continuously publishing
their latest discoveries
with each paper adding to the scientific conversation.
Published studies can motivate future research,
inspire new products,
and inform government policy.
So it's important that we have confidence in the published results.
If their conclusions are wrong,
we risk time,
resources,
and even our health in the pursuit of false leads.
When findings are significant,
they are frequently double-checked by other researchers,
either by reanalyzing the data
or by redoing the entire experiment.
For example, it took repeated investigation of the CERN data
before the timing error was tracked down.
Unfortunately, there are currently neither the resources nor professional incentives
to double check the more than 1 million scientific papers published annually.
Even when papers are challenged, the results are not reassuring.
Recent studies that examined dozens of published pharmaceutical papers
managed to replicate the results of less than 25% of them.
And similar results have been found in other scientific disciplines.
There are a variety of sources for irreproducible results.
Errors could hide in their original design, execution, or analysis of the data.
Unknown factors,
such as patients' undisclosed condition in a medical study,
can produce results that are not repeatable in new test subjects.
And sometimes, the second research group can't reproduce the original results
simply because they don't know exactly what the original group did.
However, some problems might stem from systematic decisions
in how we do science.
Researchers,
the institutions that employ them,
and the scientific journals that publish findings
are expected to produce big results frequently.
Important papers can advance careers,
generate media interest,
and secure essential funding,
so there's slim motivation for researchers to challenge their own exciting results.
In addition, little incentive exists
to publish results unsupportive of the expected hypothesis.
That results in a deluge of agreement between what was expected
and what was found.
In rare occasions, this can even lead to deliberate fabrication,
such as in 2013, when a researcher spiked rabbit blood with human blood
to give false evidence that his HIV vaccine was working.
The publish or perish mindset
can also compromise academic journals' traditional peer-review processes
which are safety checks
where experts examine submitted papers for potential shortcomings.
The current system,
which might involve only one or two reviewers,
can be woefully ineffective.
That was demonstrated in a 1998 study
where eight weaknesses were deliberately inserted into papers,
but only around 25% were caught upon review.
Many scientists are working toward improving reproducibility in their fields.
There's a push to make researchers raw data,
experimental procedures,
and analytical techniques more openly available
in order to ease replication efforts.
The peer review process can also be strengthened
to more efficiently weed out weak papers prior to publication.
And we could temper the pressure to find big results
by publishing more papers that fail to confirm the original hypothesis,
an event that happens far more than current scientific literature suggests.
Science always has, and always will, encounter some false starts
as part of the collective acquisition of new knowledge.
Finding ways to improve the reproducibility of our results
can help us weed out those false starts more effectively,
keeping us moving steadily toward exciting new discoveries.