Dear Fellow Scholars, this is Two Minute Papers with Károly Zsolnai-Fehér.
We are here to answer a simple question: what is peer review?
Well, in science, making sure that the validity of published results is beyond doubt is of
utmost importance.
To this end, many scientific journals and conferences exist where researchers can submit
their findings in the form of a science paper.
As a condition of acceptance, these papers shall undergo extensive scrutiny by typically
2 to 5 other scientists.
This refereeing process we call peer review.
Single blind reviewing means that the names of the reviewers are shrouded in mystery,
but the authors of the paper are known to them.
In double blind reviews, however, the papers are anonymized, and none of the parties know
the names of each other.
These different kinds of blind reviews were made to eliminate possible people-related
biases.
There is a lot of discussion whether they do a good job at that or not, but this is
what they are for.
After the review, if the results are found to be correct, and the reviews are favorable
enough, the paper is accepted and subsequently published in a journal and/or presented at
a conference.
Usually, the higher the prestige of a publication venue is, the higher the likelihood of rejection,
which inevitably raises a big question: how to choose the papers that are to be accepted?
As we are scientists, we have to try to ensure that the peer review is a fair and consistent
process.
To measure if this is the case, the NIPS experiment was born.
NIPS is one of the highest quality conferences in machine learning with a remarkably low
acceptance ratio, which typically hovers below 25%.
This is indeed remarkably low considering the fact that many of the best research groups
in the world submit their finest works here.
So here is the astute idea behind the NIPS experiment: a large amount of papers would
be secretly disseminated to multiple committees, they would review it without knowing about
each other, and we would have a look whether they would accept or reject the same papers.
Re-reviewing papers and see if the results is the same, if you will.
At a given prescribed acceptance ratio, there was a disagreement for 57% of the papers.
This means that one of the committees would accept the paper and the other wouldn't, and
vice versa.
Now, to put this number into perspective, the mathematical model of a random committee
was put together.
This means that the members of this committee have no idea what they are doing, and as a
review, they basically toss up a coin and accept or reject the paper based on the result.
The calculations conclude that this random committee would have this disagreement ratio
of about 77%.
This is hardly something to be proud of: the consistency of expert reviewers is significantly
closer to a coinflip than to a hypothetical perfect review process.
So, experts, 57% disagreement, Coinflip Committee, 77% disagreement.
It is not as bad as the Coinflip Committee, so the question naturally arises: where are
the differences?
Well, it seems that the top 10% of the papers are clearly accepted by both committees, the
bottom 25% of the papers are clearly rejected, this is the good news, and the bad news is
that anything between might as well be decided with a cointoss.
If the consistency of peer review is subject to maximization, we clearly have to do something
different.
Huge respect for the NIPS organizers for doing this laborious experiment, for the reviewers
who did a ton of extra work, and kudos for the fact that the organizers were willing
to release such uncomfortable results.
This is very important, and is the only way of improving our processes.
Hopefully, someday we shall have our revenge over the Coinflip Committee.
Can we do something about this?
What is a possible solution?
Well, of course, this is a large and difficult problem for which I don't pretend to have
any perfect solutions, but there is a really interesting idea by a renowned professor about
crowdsourcing reviews that I found to be spectacular.
I'll leave the blog post in the comments section both for this and the NIPS experiment, and
we shall have an entire episode about this soon.
Stay tuned!
Thanks for watching, and for your generous support, and I'll see you next time!