Hi everyone, welcome to presentation for our alt CHI
paper titled, CHI 2023 decisions on paper 42 CHI 2023 A playful
twist on the peer review process and methodological gaslighting,
written collaboratively by Tessa Eagle, Leya Breanna Baltaxe-Admony. Lee
Tabor and Kate ringland, from University of California, Santa
Cruz and the misfit lab. So for a hypothetical scenario, we have
a point of view of an ethnographer and machine
learning researcher, statistician perhaps. And they
both receive an email to for a request to review. So in this
example paper, they both receive a request to review the paper a
digital ethnography of social media community. Both of them
think about it, as we all probably do look at the
abstract. And they both decide to accept. I'm sure there are
many reasons that go into accepting and rejecting a review
request. However, what could go wrong in this situation? Has
this happened to you, you get a request to review a paper with a
topic or methodology you know little about or covered in one
day of grad school methods years ago. Your paper is reviewed by
people who don't seem to understand your methods, but
think they do and think that you did it wrong. Or you felt like
the reviewer did not read the same paper that you wrote and
submitted. If that's the case, you may be entitled to financial
compensation. Just kidding. But you aren't alone in these
experiences of peer review challenges. So the goals of our
paper, not to widen the existing divide between methodological
camps. We all have our strengths and mixed methods, but to
promote discussion around feedback that is being given in
the reviews that we are receiving at ACM venues and
specifically CHI. Another goal is to discuss how we can be more
intentional with our reviewing practices. And we would like to
note that these issues are not limited to qualitative
researchers. But we write from our recent experiences with our
qualitative submissions. So this paper and presentation, these
are intended to be educational, and to give a brief introduction
to a discussion of ethics, positionality and community
based work, community based work as they have appeared in our own
reviews, as well as broader academic discussions, and reveal
these intersections that belong in all work, no matter the type.
We also hope that this paper is reflective and that we aim to
stimulate discussion around the peer review process, and how to
hopefully make it better going forward. And our paper is
intended to be humorous without mal intent and without pitting
methodological camps against each other. I'm sure we all have
experiences with the so called reviewer two. And here's an
example of a review commentary that we received on a
qualitative community based paper. So the reviewer said, Why
did you not use statistics or machine learning on a
qualitative community based paper and then suggested that
this paper was not suitable for PI due to the exclusion of
these. So in terms of positionality, and methodology
clashes a few points, so the people fighting to change
problematic systems are typically the ones being hurt by
those systems. This work is the result of many discussions and
frustrations with the peer review process within the field
of HCI that our lab has been having. Over the past few years.
As primarily a qualitative researchers, we have found our
work to be taken less seriously and often misunderstood by
reviewers who have little experience with our chosen
methods. This is not to say all of the reviewers have done this,
but we have seen this more often than not in our work. And
intentional work, especially qualitative is slow work that
takes longer than some people may like and it can be
demoralizing to receive reviews from researchers that have a
fundamental misunderstanding of our Are your methodological
approaches and have to extensively justify why our work
is Cuy enough. The peer review process can be especially
important for early career researchers. So younger
academics may feel pressure or guilt to accept or review
requests despite knowing a little about the topic. earlier
stage academics in particular can be most harmed by not having
access to appropriate reviews. So I'm sure we along with many
others, we have spoken to seem to dread getting paper reviews
back as they are often written in an unhelpful or unkind ways.
And again, this is not saying that everyone is doing this. But
we have experienced this. And labs we've worked in have
developed group rituals for dealing with the emotional work
of parsing overly critical reviews. And we feel this
process can dissuade people from pursuing or continuing work in
higher education. And in fact, there's little to no incentive
to review papers other than your papers being reviewed in turn,
which is obviously a big incentive. However, there are
systemic issues with number of reviewers and number of willing
reviewers that we'll get into on the next part. So reviewers and
paper chairs are overburdened and tired, people are inundated
with requests to review. And interestingly, researchers often
receive little to no training before being expected to carry
out a review. The process also varies across submission types
and venues so even within CHI. So the CHI peer review process
is very different than the technical paper review process.
We found that critical reviews without substantive feedback are
common. And this is echoed in the parody reviews that we've
written in our paper. As mentioned before early career,
researchers may be pushed out of the field by overly critical
reviews that focus on the wrong parts of the work. Also, we
found conflicts of interest can impact the reviewer pool and
this disproportionately affects budding or marginalized
communities where the pool of collaborators is often more
tenuous or limited, such as in the accessibility track. An
example of a potential way forward is taken from the
reviews we received for all chi papers. So five, researchers
took it upon themselves to co write a collaborative review for
our submission. So they connected in the discussion
forum on PCS. And then they co wrote in a Google Doc and
uploaded it to PCs for us to review and it was co signed by
all five of them. These five researchers also suggested in
their review a few next steps for the community. So they
suggested first mapping out common reviewing pitfalls. And
second taking inspiration from other domains venues that
utilize alternative peer reviewing process. So some of
the questions that we have for discussion and in general are
around what makes one reviewer an expert versus very
knowledgeable. And what are they an expert in? Is it the research
area, the method theory, or just their position at their
institution? Being an expert in a research area does not
automatically make one an expert in a given method used inside
topic. However, these expertise levels can be confusing,
especially for earlier career researchers and grad students.
We would like to thank all of the reviewers that took time to
engage with our work and we would love to hear your
comments, experiences with CHI reviews, and any questions that
may have come up