Hi, I'm Casey and i'm a professor in an information science department.
I make videos about academic life and advice for graduate
students and other academics and I've had some requests recently to
make a video about how to review a paper, which i actually
think is a great topic though as I was thinking about it
I realized that in order to think about how to review a
paper mechanically it's also important that you know how
the peer review process works. It might seem like it's
obvious but when i was a graduate student
parts of it were still pretty opaque to me.
The first time that I was on a program committee reviewing papers as a faculty
member it really opened up my eyes. It was like
seeing how the sausage was made. Which sometimes makes you want to not
eat sausage anymore. Also in addition to being a good paper
reviewer, understanding how the process works
has a couple of benefits in writing papers. One is that you can think about
how your paper might be received during the review process.
The other is that honestly once you understand
the degree of randomness that is involved in the peer review process
you actually don't feel as bad when your papers get rejected.
So let's talk about how the process works and where that randomness comes
from. Okay so I'm going to be talking about
both the traditional journal peer review process
and in computer science specifically our major publication venue is actually
conference proceedings and the conference peer review process
with program committees is a little bit different so
I'm going to talk about both of them as we go through this. I'm not
going to talk about law reviews which is like a whole other
bizarre thing that should be the topic of perhaps a whole other video. Also I am
talking here about real good academic publication venues,
not predatory journals. If you want to know how peer review happens for a
predatory journal... (it doesn't.) And also what I'm talking about here is of course
most common in my experience. I do a lot of reviewing
both for these computer science conferences as well as for a lot of
journals in different disciplines. I also was the associate editor of a
law related journal for a long time and so I have a little bit of experience
from that end as well. But of course lots of disciplines are
different lots of journals are different and there are lots of variations on what
I'm talking about so it may not be exactly like this for the
publication venue that you are thinking about.
All right so the first thing that happens here
is someone writes a paper and as you know
that is a lot of work and even after you've written the paper
you then have to decide where you're going to submit it.
Are there any deadlines, you probably have to change some kind of formatting
which ends up being a total pain and then you fill out a form
that's confusing and you submit your paper and then
you wait. But what is happening while you're waiting? After it gets to
the venue there is intake of some kind. Sometimes
there's an admin who is checking for desk rejects
because it's too long or you did the formatting
wrong or something egregious but at some point at a journal your
paper goes to an associate editor or someone like that
and essentially it's someone who is overseeing the review process. How
they are assigned varies widely from journal to journal. As far as I can tell
sometimes it's based on subject matter expertise,
sometimes it's random, or whoever happens to be doing all the papers that week,
but in the case when I was the associate editor for her journal I was the only
one. But in any case at a journal once it
goes to that person their job is to find reviewers. How many?
Well that varies. I would say that for me in the journal that I was working with
I would typically send out invitations to four
people, hoping to get three and sometimes would settle for two, but
it really varies from journal to journal. It is
very rare though for there only to be one reviewer for a paper.
Also as I mentioned in computer science where our main publication venues are
conference proceedings there is an entire
committee called the program committee who review
all of the papers at once. Those of you who are not in this field you're going
to think this is really weird. My understanding is that this is pretty
unique to computer science but might also happen in some
adjacent fields like maybe certain disciplines of engineering.
I'll also link below to some resources about sort of
why this is because that's a whole other video.
But in that case there's usually two or three people who are the papers chairs
of the conference who are kind of like editors in chief of a journal and then
a large number of associate chairs who make up the program committee
and of course how many there are depends on the size of the conference.
But these people are essentially playing the role of associate
editor as if they were in a journal. Sometimes all of the reviewing happens
within the program committee. So a certain number of associate chairs
are assigned to each paper and maybe one person
is designated the sort of overseer meta reviewer for that paper
or each paper is assigned to someone who finds external reviewers.
Conferences that I am most familiar with typically two
associate chairs are assigned to each paper with one of them
being the meta reviewer. So it means that it's their job to find reviewers
and to synthesize the reviews and the other person who is on the paper is just
serving as a normal reviewer and it works that way in part so that
when discussions of the paper happen there are two people on the committee
who have read the paper. Another thing that can happen at this
point is that sometimes papers are desk rejected by the associate
editor. Typically you need to skim a paper
anyway to be able to find reviewers and so if that paper really really
misses the mark for whatever reason and it feels like it would be a waste of
time to send it out to reviewers like it's
really just going to get rejected it's actually
a kindness to desk reject because then the person who submitted it gets the
feedback sooner and they can send it to a different
venue or make revisions. Sometimes this happens
when the paper is just completely inappropriate for the venue
but in any case that next step in both of these situations
is to find external reviewers. Now how does the
associate editor or associate chair who is assigned to that paper
pick people to review the paper? So there are a lot of different answers to this
question and there might be sort of guidelines or
suggestions for individual venues but typically it's either something like,
they ask people that they know who do work in that area,
they look at the citations in the paper for other papers that are similar and
then find the people who wrote them, they might
just do a google scholar search to find people.
Sometimes for journals there is a sort of slate
of people who've reviewed in the past that might be organized in some way,
maybe tagged by topic. Sometimes people use some big databases
that people can kind of sign up for when they do reviews.
For conferences because reviews happen all at the same time it's common for
people to volunteer to review a certain number of papers and they
appear in some kind of database, sometimes
tagged by expertise or something like that where
people can find them in that database to ask them to review.
So here's a really important thing: people who are asked to review papers
say no. A lot. Which is completely
understandable because paper reviewing is kind of unpaid volunteer work.
I guess you could argue that if you're a professor then it constitutes
academic service which in theory is part of your job
which you are getting a salary for however we all know that we spend way
more time on service than the 20 percent of our work that
it's supposed to be. And if you're - and if you're really
having trouble finding reviewers you might have to keep going sort of
farther and farther away from what you would consider the
appropriate or ideal expertise. For people to review that
paper. That's just how it goes. Also if you are asked to review a paper
and you can't because of timing or whatever the
kindest thing that you can do is to suggest
someone else okay so once you've managed to
find reviewers and they have said yes celebrate if you're that associate
editor or associate chair and then you wait for the reviews to
come back. Meanwhile the reviewer is doing their
reviewing thing and again that's the topic of another
video. Typically as part of this process the reviewer makes their individual
recommendation. Should the paper be accepted or rejected
or things in between, major revision, minor revision, whatever that venue has
for the options for a decision for that paper.
Once all of the reviews come back what happens next?
So the associate editor or the associate chair's job
is to synthesize the reviews and make a final decision or recommendation.
If there's a lot of variability in the reviews this task
sucks. If you have three reviewers and one loves it and one
hates it and one is meh what do you do? This does become a little
bit easier when there is an option for the paper to
be revised. Because sometimes the person who hated
it, if the paper is revised they will not
hate it as much. So that's also the associate editor's
job is to look at that bad review and determine
whether the things in it are things that can be fixed.
Now for some venues and I'm thinking in particular of some computer science
conferences, there is no revision. It's either accept
or reject. I personally am not a fan of this
model. However, that again is the topic of
another video. Very often for journals the associate
editor makes a recommendation to the editor-in-chief of the journal who then
makes a decision. But in any case this might look slightly
different depending on different venues. All right so then that editor-in-chief
will send a decision to the author. But for
conferences, there gets to be a super fun meeting where everyone sits around
and discusses papers. Often there is a numerical or some kind of cut off for
papers that are definitely accepted or definitely rejected. And then the
discussion is for the things in the squishy middle. These used to be in
person and they're actually kind of fun. nNw they're mostly virtual which in the
end is good, it's more accessible, more sustainable etc. But yeah sometimes
I'm spending like literally three days on video conference
all day doing this. Actually iIguess we're all spending all our time on video
conference these days. Yeah this is rough. I will say
One of these sort of here's how the sausage is made
things that I realized about program committee meetings for conferences,
it really shows how much power the associate chairs have over the fate
of a paper, in part because they're the only ones in
the room to explain the paper to everyone else.
But really in both cases, journals, conferences,
any peer review, there's someone who's making a final decision
and that person, they have a vote that carries a lot more weight than the
individual external reviewers. But that's not the
only reason they have power. Because remember they're also the
ones who are picking the reviewers. Choosing who the
external reviewers are has a huge amount of power over
the fate of that paper. This is what i mean by the randomness
in the review process. Who does your paper happen to get assigned to
to shepherd it through the process and who do they happen to pick
for external reviewers? Sometimes you get unlucky.
But then at some point a decision is sent to the author.
If it's a reject move on, look at the reviews, decide whether you
agree with them, make revisions for things you think will make the paper
better, don't make revisions for things you
don't make the paper better. Because those reviewers are likely not
going to see it again. nNw that said that is not always the
case, especially if you are in a small
field, the same person being asked to review your paper
after you submit it to a different venue could totally happen.
This actually happened to me very recently, I was like, I've seen this
before. But that said you shouldn't change your
paper in a way that you think will make it worse
just to make a reviewer happy especially if you don't even know for sure that
that reviewer is ever going to see the paper again.
If the decision is that you need to revise the paper and submit it again
you will be getting the same reviewers, hopefully.
Pay close attention to what the meta review or the associate editor's
comments are telling you to do. They sometimes will give you kind of
hints about which reviewers you should pay attention
to and again they're the ones with most of the power
so if they tell you to do something you should probably do it.
And on that note if you review a paper and then are
asked to review a revision, do it. Unless there's some really good reason
why you can't you're kind of a jerk if you do not
agree to review the revision because if a new reviewer
has to come in it's really kind of unfair to the author
because they're going to have a whole new set of issues with the paper
that they weren't told to revise the first time around.
And when that happens it's really unfortunate
though sometimes it just can't be avoided. And then after that
it all starts again and eventually you will get a final decision
whether it's good or bad. So i don't know if you learned anything new about this
process but let's talk a little bit about how much
we can trust it. I could possibly make a whole video
about ways that i think the peer review system are broken.
For right now I will link to a Twitter thread about that down below.
But in any case this is what we have and it does
work generally. But there are some things that are important to understand. And one
is that you may or may not be getting real
experts reviewing your paper and even when you do you often don't
have experts on every aspect of the paper. One of the
things this means is that published papers
will absolutely have mistakes. Sometimes perhaps because the reviewers didn't
know it was a mistake, but also because there's only so much
labor these reviewers can give. For example they're not going to be
pulling the things that you cite to make sure that you are representing
their findings correctly. Unless it's a law review and again that's a whole
other thing. But also it's not like the reviewers are
going to rerun your statistics or redo your
experiment or something that can actually check if
your data is analyzed correctly. A number of years ago there was this
huge scandal about a political science graduate student who fabricated data
for a really important paper. I will link to a story about that down below,
it is bananas. But it got a lot of criticism for like oh the peer review
process is broken, they should have caught this.
In some ways the amount of work that would have had to be done
to catch that problem is really more than we can expect for
unpaid labor. And another thing is that a lot
of the integrity of this process relies on the individual integrity of
the people who are involved in it. For example conflicts of interest
essentially rely on the honor system and you can't expect the associate
editor to do deep research on the authors of every
paper to find out who their co-authors are so
that they aren't asked to review one of their papers. Sometimes the
process is single-blinded which means that
the reviewers know who the authors are even though the authors will not
know who the reviewers are. Sometimes it's double-blinded which
means that the reviewers don't know who the authors are
either. But if you know who the authors are
and you're like oh that's my best friend, I am definitely going to give this a
good review, that is unethical. But unfortunately it sometimes depends
on the honor system. And really there's a lot of places where
we just have to rely on there not being bad
behavior. Bad things you could do: apparently there
are some people who collaborated to like create this group of people who were all
going to give good reviews to each other's papers.
Obviously deeply unethical. Or what if as a reviewer you
tank a paper because their work is so close to yours
that you think you're gonna get scooped? Unethical.
So I hope that demystified the process a little bit if you didn't really know
all of the ins and outs of peer review. Again this can be really different for
different journals and in different disciplines
and so you might want to find out from someone who knows more than you
exactly what it looks like for your discipline.
Maybe the sparked questions for you, what questions do you
have about peer review? Put them in the comments down below and i will answer if
I can. And if your question is, how do I do it
how do I review a paper, stay tuned, maybe hit the subscribe
button, and i'm hoping to do that soon. And since
we're talking about peer review a final reminder: your papers
will get rejected sometimes, this is just the way of the world.
so you should see the peer review process as an
opportunity to improve your work. I know it's really hard sometimes
and for all the reasons I just said reviewers sometimes get it wrong.
So that means that you should feel good about your work and submitting it
somewhere else where you'll get a new set of reviewers
and will have a new opportunity for that randomness to work in your favor.
Keep your chin up and revise and resubmit.
All right this is all I have for you right now though I plan to continue
doing some videos about the peer review process as well as paper writing
and all other kinds of things that graduate students and academics
might want some advice about. I'm Casey, thanks for watching.