Welcome to the presentation of our TOCHI Paper „Understanding HCI Practices and Challenges
of Experiment Reporting with Brain Signals: Towards Reproducibility and Reuse” This
is joint work by Felix Putze, Susanne Putze and Merle Sagehorn from Uni Bremen together
with Chris Micek and Erin Solvey from WPI.
Measuring brain data has been a method in HCI for multiple decades: Researchers use
it to create self-adapting interfaces, to create new modes of communication or control,
or for evaluation purposes.
HCI research using brain data has appeared at CHI in 1996 and has been present every
year since 2001.
Research in this area comes from many different fields: Cognitive science, neuroscience, signal
processing, machine learning, and others.
While this interdisciplinarity is responsible for a vibrant and creative community, it also
creates a challenge: Many researchers in the field act as lone rangers, create their own
ecosystems from scratch, do little in terms of reproduction for mutual validation, and
rarely build on each others‘ work.
The goals of this paper are thus the following: First we wanted to create a compresehensive
corpus of papers on brain data which self-identify as HCI-centric through its publication outlet.
This resulted in a database of 110 papers Second, we analyzed paper demographics, that
means we studied the frequency with which domains, modalities for data acquisition,
types of applications, etc.
occured Third, we derived an experiment model to structure
experiment reporting in a systematic, reproducible way
Fourth, we analyzed the reporting practices we enountered and identified patterns of different
reporting styles Finally, we destilled a number of actionable
recommentations for future researchers, reviewers and editors in the field
When going through the 110 papers, we encounted a very diverse research landscape: Many different
and unique cognitive and neural constructs were reported, from creativity and humor to
fatigue and confusion.
Among the recurring ones, the most frequent constructs were workload and attention.
We found that the majority of the works was purely methodological, that means not tailored
towards a specific application, making them universal, but also not rooted in a real use
case.
Furthermore, we found that the papers which use brain data for evaluation of interfaces
or interface elements report mostly technical contributions, for example measured in classification
accuracy, while papers which use brain data for explicit control of a system or for neurofeedback
focus strongly on user centered contributions, for example measured with user satisfaction.
Given this diversity and the need to support reproduction and follow-up work better, we
collected and structured the different attributes that were used to report the performed experiments.
This process resulted in a formal experiment model.
The model covers 50 attributes in 6 categories, ranging from technical aspects of the recording
to aspects of the experiment flow and methods of brain signal integration.
It can be used both as a tool for systematic analysis of experiment reports as well as
a guideline for reporting when writing up a new experiment.
Using the experiment model, several experts annotated all papers from the database for
the presence or absence of each of the attributes.
We found that there are stark differences in the frequency of reporting: While reporting
of the recording device is a nearly universal information given by nearly everybody, only
half of the papers report details such as the reference electrodes for EEG and some
aspects, such as details of hyperparameter tuning or anonymization procedures, are rarely
reported at all.
The frequencies also change over time – while more recent publications report more information
about questionnaires for evaluation and technical aspects such as synchronization and artifact
handling, low-level signal processing has dimished in frequency over time.
To further uncover latent patterns within the reporting strategies, we performed an
explorative factor analysis on the paper annotations.
We found a clear structure among attributes defined by the main factors, which we could
use to verify and extend the original categories.
Furthermore, the analysis revealed that different types of papers followed very different reporting
strategies.
The paradigm of brain data usage was among the central determinants: for example, we
observed a distinct absence of data processing details for neurofeedback papers and a strong
focus on machine learning details for paper on mental state assessment.
This observation indicates the existence of several sub-cultures with their own practices
for reporting experiments.
From these observations, we derive a number of recommendations for future authors, reviewers,
and editors.
We suggest to exchange best practices in experiment reporting, where different parts of the field
have different strenghts and weaknesses and should be able to learn from each other.
On the other hand, everybody should be aware of the heterogeneity of the field, for example
when reviewing . Researchers should create oppportunities to build on each other‘s
work.
One important aspect is sharing of data and code through open platforms.
Another aspect is publishing expanded experiment descriptions, for example on Github, OSF or
through other means when the space in the paper is limited.
Of course, such descriptions can apply the experiment model to structure the report and
check it for completeness.
Finally, this model of course needs to be extended to cover future use cases; for this,
we invite you to contribute to the repository which holds that model.
These were some of the highlights of the paper, which still contains much more information,
such as details on the literature database curation, detailed examples contrasting HCI
research using brain data with traditional BCI research, many more statistics, including
an analysis on exploratory vs. confirmatory reporting and finally an appendix with all
attributes of the experiment model with definitions and examples from the database.
Thank you for your interest and get in touch if you have any questions!