Wow, it is hot out there!
The news says that this is the hottest it’s ever been in Arizona this late in the year
for as long as they have data.
Who knew?
Well, someone had to find out.
Behind every number that’s casually reported, there’s a researcher who spent hours if
not months or years analyzing and fact-checking.
Just for this weather anecdote, the news fact checker had to find, combine, and analyze
the local and state-wide weather records as far back as they go.
Like, before there were spreadsheets!
They even had to call the historical society to double check an old weather journal.
All of that time and work collapsed into a few sentences or even just a headline.
But there’s a ton of context that we miss.
By now, we’ve talked a lot about how to check out study results when they come to us.
But a lot happened before the data got to us to make sure research that reaches the
public is valid, reliable, and credible.
It’s time to go behind the scenes of the research world.
I’m Jessica Pucci and welcome to Study Hall Data Literacy presented by Arizona State University
and Crash Course.
Research is the process of answering questions about the world around us.
Way back when we collected our local weather data, that was research.
So was using data to pick our fantasy sports team.
And researchers are people who propose questions, make hypotheses, and then try to find evidence
that either supports or contradicts their hypothesis.
If you’ve ever wondered about something and investigated the answer, you’re a researcher.
Research questions come in all shapes and sizes on basically every topic you can imagine.
But at their core, research questions can either be answered with quantitative data
— so numbers — or qualitative data — so descriptions and symbols that aren’t numbers.
After finding an answer to our research question, our next move is to turn the results into
a paper.
But, like, why?
Publishing and sharing research papers is kinda how researchers “get credit” for
their work.
Academics and researchers care who thought of an idea or strategy.
But more than that, papers are a good way for researchers to share their work with others.
No matter what field you’re in, raw data doesn’t tell a very compelling story or
let people easily learn and build on the research question.
Anyone, but especially journalists, can start with research papers to help bring new findings
to the public.
But how long it takes the journalist to write an article and how long it took the researcher
to write the paper are usually very different timelines.
Before a research paper is formally published, it has to first be reviewed by a handful of
other researchers with knowledge about the subject.
This review process or peer review is required before an academic journal will agree to publish
the work
in the hopes of making sure it’s accurate and hasn’t already been published.
So peers from around the world make suggestions about how to improve both the presentation
and the work itself and ultimately judge whether the paper is a good fit for the journal.
Peer reviewers are volunteer professors, senior graduate students, and other expert researchers
donating their time on the side, so the process can be slow.
So if you submit a paper in 2020, it might not debut until 2021 or even 2022!
One paper usually goes through this process many times across different journals before
it finally finds a home.
It takes time for the researchers to refine their story and make it compelling to their reviewers.
Finding a journal where a research project fits in with what else is usually published
isn’t always straightforward either.
Let’s see how a researcher navigates this process in the Data Point.
It’s been a labor of love (and okay, sometimes hate), but the researcher finally hit submit
on her paper.
Two long years of data collection, statistical analyses, and revising the writing.
Now she has to wait for the journal editor to identify experts in the field who can weigh
in on her work.
Many journals have a double-blind review process, which means that the reviewers don’t know
who the researcher is and vice versa.
Hopefully this lets the reviewers be honest without being biased by any information about
the author.
Or maybe it lets them be a little too honest.
A month later the researcher gets the news.
Rejected!
But that’s okay, it’s all part of the process.
She perseveres and sends her paper to her second choice journal.
Another month goes by before she gets a response in her inbox: the paper isn’t ready for
publication… yet.
She receives a revise-and-resubmit request — which is basically what it sounds like.
Revise-and-resubmit requests mean that she has some time to consider the feedback that
the reviewers submitted,
incorporate it into the paper, and resubmit the updated draft for reconsideration.
She’s getting there!
Undeterred, she gets back to work fixing everything from small typos and color changes in the
graphs to rewrites of whole paragraphs.
Some of the reviewer comments seem reasonable.
Others she disagrees with.
Part of the revise-and-resubmit process has her create a new document or track her responses
to each comment
from a reviewer and say how she addressed the suggestion or why she doesn’t think
it’s a good idea.
This way she can defend herself where she disagrees and prove that she took the reviewers’
other suggestions to heart.
Yet another month later, this updated version of the paper gets the green light.
It’ll be published in the next edition of the journal coming out some time next year.
Time to celebrate!
Will all of us publish a paper in a research journal?
Probably not.
But knowing the lifecycle of a scientific paper helps calibrate our expectations and
change how much trust we’re willing to put into the “latest” findings.
The conclusions might not be hot off the presses since it takes so long to publish something,
but that’s usually time well spent.
Peer feedback helps researchers grow throughout their career by honing the researcher’s techniques.
And the research process and findings are ultimately vetted by other experts so hopefully
incorrect information doesn’t slip through.
But if a paper does manage to get published with an error, a journal can issue a retraction,
or remove the paper and issue a notice that a major problem was identified in the work.
Of course, retraction notifications aren’t guaranteed to reach everyone who read the
original paper and is already using the work.
But places like Retraction Watch are trying to change that by documenting retractions
more prominently and transparently.
Retractions can happen for a variety of reasons —
something small like a typo in the original data or something more dramatic like a flaw
in the data analysis or how the data was collected.
These big issues are harder to spot, especially without seeing the original data.
Catching this kind of bad data has led to a push for more transparency and including
all of the hard data work behind the formal paper in the published product.
To bring it to larger, more accessible audiences, some of this transparency work is even taking
place on “academic Twitter” where researchers share findings, responses, and even sources.
By sharing our process, other researchers can use our methods or even reproduce our
experiment to confirm or expand the results.
The idea of reproducibility has many interpretations, but it basically means that someone can follow
a study’s process and get the same results.
If a study is reproducible, this helps us trust that the results are reliable.
Not every study is reproducible though, even if it was done in good faith.
By chance our sample of mice might’ve been really good at mazes in a way that doesn’t
represent the general mice population.
Or we could fail to measure all of the conditions needed to really reproduce the result.
If it turns out humidity is really important, and we don’t report what our lab is like,
another team might have a less humid room and think nothing of it.
We can strive for reproducibility by recording every detail and decision, even if they seem unimportant,
and sharing data and the code used to analyze it so that reviewers can fully check the results
in the paper.
Transparent data also helps other researchers build on the work instead of making them first
reinvent the wheel just to make a tiny improvement.
Sharing data and results that didn’t work can be just as helpful to our fellow researchers,
but we’ll rarely see null or negative results published in formal, peer reviewed journals.
Knowing what doesn’t work is still valuable for understanding a problem and could help
change the way we all talk about failure.
But because there’s such pressure to publish, viewing negative results as not being worth
sharing creates an incentive for researchers
to tamper with their results and ultimately mislead the public.
Some journals take advantage of this pressure to make a profit.
Predatory journals charge fees for publishing papers but don’t perform good quality control.
So when we pick up a journal article or see one referenced in the news, it’s worth doing
some investigating about the journal itself.
Not having a peer review or some other clear quality control process is a big red flag.
We can also research the journal and see if there’s any bad press about it or if it
appears on any official list of publications in a field.
Or you can ask a librarian; they often have insight into what journals are considered legitimate.
Hard to find retractions, reproducibility problems, not publishing all types of results,
and predatory journals are all good reasons to shun the traditional publishing world.
And some researchers do.
Work can be posted directly on a personal webpage, or companies might post a formal
report about their research on their website, which is often called a white paper.
And since the publishing process can take so long, initial drafts of formal papers are
sometimes posted on the web to start sharing the research.
These drafts are called preprints.
The benefit of self publishing or releasing a preprint is that these approaches circulate
ideas much more quickly.
Not to mention getting published can be tricky and inherently unfair.
Like there are noticeable gender, race, and country of origin disparities in who gets published.
But the downside of a preprint is that it misses being rigorously evaluated.
If we depend on these results, there might be weakness in our work that an expert would’ve caught.
Publishing papers is also more the currency of academic researchers.
But for researchers in other careers, writing papers might not be a priority.
Instead, other checkpoints can be put into place by governments or other regulatory bodies.
For example, medical research done by pharmaceutical companies must pass certain quality control standards.
In the US, the Food and Drug Administration requires drug trials meet certain benchmarks
to make sure the drug is effective and safe.
The EU equivalent is the European Medicines Agency.
In China the equivalent is the National Medical Products Administration.
Regulatory methods are different from peer review, but they still provide a way to control
the quality of research that reaches the public.
Balancing sharing information quickly like through preprints with getting helpful feedback
from other experts during peer review is tricky,
but by knowing the stages of research we can better evaluate where a paper falls in that balance.
Always being on our guard for data done bad is a useful data literacy strategy, and knowing
what goes on in the world of research
helps give us some confidence that the research that makes it to our ears is credible.
Today we learned how researchers’ work is checked for quality, whether that’s through
a peer review or a regulatory process.
We aren’t the only ones who value transparency and reproducibility; these processes do too.
Quality controls have their own pitfalls, but staying up on paper retractions and predatory
journals helps mitigate these weaknesses.
Next time we’ll learn about cases where bad data can slip through the cracks and experts
can make mistakes.
Thanks for watching Study Hall: Data Literacy, which is produced by Arizona State University
and the Crash Course team at Complexly.
If you liked this video and want to keep learning with us here in Study Hall, be sure to subscribe.
You can learn more about ASU and the videos produced by Crash Course in the links in the description.
See you next time!