The HCI Community increasingly cares about research ethics, openness, and transparency.
In this work, we examine how the field of HCI is progressed in addressing these issues.
Based on these aspects we synthesize 45 criteria from existing literature. We then
use these to analyze samples of Empirical research papers from CHI 2017 and CHI 2022. Lastly, we also
developed a proof of concept system to assess automated approaches for evaluating such criteria.
First, let us have a look at each of these aspects along with some example of the criteria. Research
ethics aims to protect research participants and promote socially responsible collaboration
between science and Society. Example criteria are whether the study received an Ethics approval or
whether researchers obtained informed consent from study participants. Openness is the availability of
research publications and materials. Example criteria are whether the paper is not behind
a paywall and whether the supplementary materials are available in a repository where materials are
findable, accessible, interoperable, and reusable. Transparent research practices are actions
taken by the researchers to disclose details of methods, data, and other research artifacts. Example
criteria are whether the paper authors shared the study materials, data, or the data analysis code.
Some of these criteria are applicable to quantitative studies, while others are for
qualitative studies. In total, we developed a set of 45 criteria. Eight criteria are in research ethics
five are in openness, and 32 are in transparency. Table 1 in the paper is a handy list of these
criteria that could be printed out as a reference sheet. For a longer discussion of each criteria,
take a look at the supplementary materials. We sample papers published at CHI 2017 and CHI 2022.
To ensure that the sample is representative across domains of CHI, we first randomly selected
conference sessions and then randomly selected one paper from each session.
We screen for papers with empirical studies because research ethics, openness, and transparency
are less applicable to other contribution types such as literature survey or theoretical papers.
That the paper was not an empirical study, we repeated the random sampling process.
In total, we analyzed 246 papers. The papers were analyzed by seven coders. All of them
were post-docs or experienced PhD students. Each coder was assigned about 35 papers
and we ensured the quality with regular discussions and monitoring agreement scores.
A more detailed description of the coding procedure can also be found in the paper.
Each coder receives step-by-step instructions like the example shown here. The instructions
listed the keywords to look for where to find them and why each Criterion was important
If you wish to replicate our study, you can find all the instructions in the supplementary
material. Let's look at the results. Four out of seven criteria for research ethics show better
compliance. For example, practices relating to obtaining IRB approval and Reporting consent
have almost doubled in the sample from CHI 2022. 29 percent of the papers use data from participants
from vulnerable populations such as minorities or children. Among these papers, around twice as
many in CHI 2022 reported additional ethical measures to protect vulnerable participants.
On the other hand, anonymization practices were low in both samples. As for the openness criteria, more
papers in CHI 2022 were freely available compared to CHI 2017. We examined whether additional
research artifacts were shared beyond the paper and found a substantial increase. 62 percent of
CHI 2022 papers shared supplementary materials, an increase from 27 percent in CHI 2017. On the other hand,
we also verified if the locations of the artifacts conform to the fair principle. Sadly the results
showed that the Improvement is negligible. Surprisingly, we discovered nine cases where
the paper promised to share the data, but we could not find them at the location stated in the paper.
A few transparency criteria show Improvement. For example, more papers in CHI 2022 clearly specified
qualitative analysis procedures. Many more papers share the interview guides. However the majority
of criteria need Improvement. In 2018 and 2019, Wacharamanotham and colleagues sent a questionnaire to all CHI authors
asking them about their transparency practices. when we compare their results with our study, we
found that actual practices are in much lower percentages than self-report survey responses.
For example, this chart shows the results about sharing software and Hardware prototypes.
The self-reported percentage is about 30 to 40 percent. Our sample shows around 20 for software and three
percent for Hardware. For further results such as statistical reporting criteria, see the paper
We had 45 criteria and this number could be prohibited for authors and reviewers to keep
track of. For most of the criteria, it is possible to narrow down parts of the paper for review based
on keywords which suggests the potential for automation either fully or partially. Therefore
we developed a proof-of-concept screening tool based on pre-trained language models. We
evaluated the tool on eight of our criteria. Some of these were simple others are more challenging.
The results show the potential for automation to help in the paper writing or the reviewing process.
Please see the paper for detailed architecture of the tool and the results.
We did find several cases of improvement, which is great but the fact is that overall rates are still
low. For example, among the criteria related to research ethics, the highest rate was for consent
forms with 57 percent adherence. But this means that almost half of the empirical papers in CHI did not use a
consent form or did not report consent collection in their papers which is alarming. The rates for
transparency practices are even lower. For example, the sharing of interview protocols has improved
greatly but 75 percent of qualitative studies do not share their interview guides. So there is much
room for improvement in these practices. Another area that needs Improvement is artifact sharing
where authors should be more mindful of sharing software and Hardware. But needless to say, these
slides represent only a fraction of the work. In the paper, we provide specific information
for each criterion and some notable insights we learned during the criteria development process.
We discuss how we can move forward as a community, including researchers, reviewers, and students.
We also discussed the implications of using assessment systems for authors and reviewers.
Our paper and all the research artifacts are available at the link on the screen.
We thank you for listening we look forward to speaking to you at CHI.