Hi there, my name is Nick Ballou.
I am a PhD student at Queen Mary University of London
and on behalf of myself and my co-authors, I want to tell you a
little bit about our project on transparency and openness guidelines in HCI journals.
In the last decade in particular there's been a growing awareness that the ways that we
create and then ultimately disseminate scientific knowledge are flawed in some ways,
and can lead to a less trustworthy evidence base than we would like.
A lot of this has fallen under the "open science" label but both by people who
identify with that label and those that do not, there have been a lot of
initiatives to reform scientific practice and make the literature more trustworthy.
A lot of these have focused on the bottom three levels here,
though, and today I want to talk about the top two
and the role that publishers play in trustworthy scientific knowledge generation.
One particularly influential attempt to reform scientific publishing practices
has come from the Transparency and Openness Promotion Guidelines, the TOP guidelines,
and these are a set of 10 modular standards that primarily address the
hypothetico-deductive model of largely quantitative knowledge generation.
And these have two goals, namely to increase the trustworthiness of the literature
by reducing things like questionable research practices, and to facilitate
cumulative knowledge growth by allowing people to build on others' work as easily as possible.
This aligns pretty closely with my own philosophy of science so it contains things that I think are
really important, like whether a journal encourages replication studies, whether
data is required to be shared, or whether null findings are systematically disadvantaged or not.
I won't talk about all of these at length so feel free to either pause the video or check the link
up here for more information, but essentially how a journal handles these ten aspects of
transparency and openness can be scored using the TOP guidelines from zero to three, where in most
cases 0 means that the journal has no policy for that particular aspect; a 1 means that there's
some kind of description or encouragement; 2 is where it is required; and 3 is where
it is required and verified or computationally reproduced, for example, before publication.
So we wanted to know more about the journals that the HCI community uses most frequently,
and whether they are currently aligning with or implementing these transparency policies.
And we focused on journal publications
instead of conference proceedings because those were the original targets of the TOP guidelines.
So what we did is we used the Scopus API to take the publication history of
all the recent CHI first authors, from the last five years,
approximately since the TOP guidelines were implemented.
We get a list of 3.3k authors and we take their entire publication history
and looked for the journals that were most common
in that set. We took the top 51 of those, because there was a tie at 50.
This was accounting for about 40% of all the publications of this group
total. And then we coded them with two coders and three rounds.
For some standards we had excellent interrater reliability, for some of them it was quite poor
but that in itself was an interesting finding and we're gonna return to that in a second.
At the end of it what we found is this. Across the board, very few of the journals in our sample
set or specify clear policies with regard to these ten aspects of transparency and openness.
The modal score was a 0, the mean was a 2.5—out of 29 total.
We see the highest rates for data citations, so about half of journals specify a particular
reference format for how you can cite data so that people can get credit for sharing data.
But for some of the other ones like analysis preregistration,
or specifying an analysis plan in advance, only 2 of the 51 journals had policies for that.
The highest performers generally were the domain-general journals, so we get
journals like PLoS One, Frontiers in Psychology, Scientific Reports also scored highly,
and then some of the uh journals from particular publishers like MDPI scoring most highly, whereas
journals from other publishers like IEEE and the ACM scored universally poorly, with ones or zeros.
This of course is not a particularly impressive state of affairs, so we have
here four things that we think people can do to help improve the situation going forward.
The first is to leverage HCI expertise to make the transparency *easy*.
What we found is that these policies are an absolute mess, sometimes spread
across multiple pages, hidden behind broken links, or even being self-contradicting.
If TOP guidelines align with your own personal scientific values,
then we also encourage you to reward transparent publishers.
You can use TOP scores, available in our supplemental materials for
the journals that we reviewed or elsewhere,
as factors when considering where to submit your papers and where to review for.
If, of course, you're on an editorial board, then we strongly encourage you to
use TOP as a starting point for implementing clear policies at your own journal.
The TOP guidelines have all kinds of templates that you can take and modify
to suit your particular discipline.
And then finally we really want to emphasize that we need to continue to reflect on whether
these particular set of transparency standards are suitable for HCI or for conferences.
These shouldn't be implemented unthinkingly and we don't expect that every publication outlet
should get a maximum score on this; instead,
it's a starting point for thinking about what aspects of transparency and openness
uh are applicable for HCI and how can we implement those most clearly and easily for our authors.
Thank you so much for listening - that's it from me, and please,
please, please get in touch if you have any questions or ideas for the future.