Many studies at CHI PLAY try to seek out effects that are true for an entire population.
However, by the nature of research we are usually not able to observe every person on the planet, for example.
Therefore: We use inferential statistics to infer from the sample we CAN collect to the population we seek to make a statement about.
The clear favorite of researchers – at CHI Play and in many other fields-- when it comes to inferential statistics, is Null Hypothesis Significance testing or NHST, for short.
However, despite being widely used NHST has become very controversial:
In fact, the naive use of it has been identified as one reason for the replication crisis.
Said crisis, so far, resulted in a massive loss of credibility in many fields, where one big finding after the other was discovered to stand on shaky empirical ground
As a rule scientific progress is achieved by standing on the shoulders of giants.
So if the giants keep crumbling below our feet, then any scientific progress is doomed, really.
But how is NHST related to these uncovered false results?
Well, the p-value, which NHST uses to determine how a researcher should behave in light of the data is a tool to control for making inferential errors:
falsely rejecting the null, and falsely rejecting the alternative hypothesis.
However, as p-Values depend on the random samples drawn from the population of interest, they fluctuate, and can be significant, even when there is no real effect to be observed.
When done well, a p-value will be falsely significant alpha amount of times: So 5% usually.
However, this false positive rate can be inflated by decisions made during data collection and analysis.
These practices to increase the actual alpha level are called:
Questionable Research Practices.
So what are QRPs?
Well, first of all: QRPs are not fraud!
Fraud is faking data
QRPs are the tactical or accidental use of otherwise justifiable or essential decisions during the research process in order to generate significant results.
For example excluding an outlying participants in order to create a significant effect between groups
Again, excluding participants is a normal part of research and often required to ensure data quality.
But when done delibartely, researchers can use outlier removal to create significant results where they want them.
Sadly QRPs are potent
Collecting just 10 more participants after unsuccessfully testing for significance,
leaving out a non-significant dependent variable,
post-hoc deciding to control for gender or other additional variables,
and selectively choosing between three conditions and then dropping the non-significant one
increases the actual alpha threshold from the nominal 5%, to up to 60%
As usually only successful – read significant – studies get published, performing QRPs is often beneficial for researchers.
Though, as a matter of fact, when people perform them, they usually do so without knowing that what they are doing is wrong.
All this is to say, that we are in a situation where the decisions researchers makes during the analysis impacts the probability of them reporting false positive results.
Therefore, reporting all these decisions one has to make during research -- like stopping rule, outlier removal or the dropping of measures -- is paramount for communicating results accurately.
We investigated how papers at CHI PLAY reported these decisions and found, that many papers
..didn’t
So we as readers were not able to determine if, in these papers, the chance of false positive results was increased or not.
In fact, in some cases we could easily tell that important steps to control the Type I error rates were missing.
This all leads to the uncomfortable realization, that we might not be able to trust many results as much as we would like.
So what should we do? What can we do, really?
Well, more details can be found in the paper, but here I want to talk about two improvements, we believe will noticeably increase the quality of quantitative studies at CHI Play
First,
Make sure what the goal of your study is.
Is it Confirmatory? Or Exploratory?
Often, we do not have enough knowledge about a phenomenon to really test a precise hypothesis.
In these cases, working exploratory and employing an estimation approach or even just descriptive analyses might be more beneficial.
Likewise, reviewers should be aware, that well done descriptive statistics are much more useful than badly done inferences and therefore not insist on p-Values.
Second:
Be as transparent as possible.
At the moment, only few papers at CHI Play share their data.
But sharing your data, your analysis plans and research artifacts increases trust: Trust in your results and trust in your handiwork.
At the same time, it allows others to get a deeper understanding of your work, allowing readers to draw their own conclusion and then more easily build on your foundation.
In Conclusion
Research is a long and complex process. Along the way, from idea to paper many things can happen.
As researchers, we have the best intentions on finding the truth, however, way too quickly, we end up not reporting the truth that is, but the truth we want there to be.
But we can reduce this risk:
In our paper we give you an extensive, non-technical, template pointing out many of these pitfalls and how you can avoid them in future.
These problems with NHST are neither new, nor exclusive to CHI PLAY, but we need to address them.
They say science is self-correcting, but this is only true if we as scientists actively seek to improve it.