In this study, we examined sources and patterns of gender bias in research participant populations.
This research was conducted at the University of British Columbia.
Gender bias in data can have serious consequences for people with underrepresented gender identities.
For example, previous research shows women are approximately 50% more likely to sustain
injury in the event of a car accident, which is attributed to a neglect of women’s data
in safety system design.
Leaving non-binary identities out of design and training data for computer vision systems
leads to creating gender recognition systems which misgender people.
In HCI, study participants are the primary source of research data.
Gender bias in these participants can lead to theories, systems, and design guidelines
that marginalize or harm underrepresented individuals.
Previous surveys of HCI research participants have shown a gender disparity in participant
samples.
As these studies focused on evaluation methods and sample sizes rather than gender, further
data is needed for a gender specific data driven analysis to verify the extent of these
patterns of bias.
We tackle these issues by exploring the following questions:
How are women and non-binary individuals represented or underrepresented as participants in HCI
research?
What are the variables that are associated with the gender demographics of HCI research
participants?
How can we systematically collect gender data from published research for a data driven
analysis?
We conducted this study in two phases, interviewing 13 researchers to identify variables connected
to gender bias in research participants, then collecting participant data from published
HCI papers for a quantitative analysis to identify trends of gender bias across all
the years of CHI.
From our interviews, we identified three key variables, participant recruitment methods,
the research area of the study, and the year the study was published.
We then created a data schema to codify participant gender and related variables, and used it
structure data collected from 1,142 randomly selected papers from CHI’1981 to CHI’2020.
From our data, we identified several important patterns.
While we are starting to see some reporting of gender beyond men and women, non-binary
participants are practically invisible.
Looking closer, it seems likely that non-binary identities are not being included in demographic
surveys.
Our next few findings focus on the comparative representation of men and women.
To analyse this comparison, we developed a ratio metric called Distance from even representation
of Men and Women, DER for short.
A DER of 0 means equal numbers of men and women participated in the research, negative
one means there were men participating but no women, and 1 means there were women but
no men.
When examining the comparative representation of men and women, we find that the under representation
of women is persistent.
The average DER of -0.15 that the median hovers around is equivalent to 10 men for every 7
women.
This is the same ratio of men to women that the previous literature found in 2014.
When looking at effects of participant recruitment, we find the impact that recruiting sources
have on participant demographics can change over time.
For example, Amazon’s Mechanical Turk as a recruiting source appears to be becoming
more biased in favour of men.
Finally, we found that research topics that had a more even representation of men and
women tended to be about social interaction, whereas topics that are more prone to bias
in favour of men tended to be about physical interaction.
This analysis has identified several weak spots in participant gender practices in HCI
research.
Our data shows that recruiting practices can introduce gender bias into participant populations,
and certain research areas are more prone to bias than others.
We therefore recommend these research areas to re-evaluate their recruiting methods.
Comprehensive data is necessary for being able to conduct a gender inclusive analysis
of representation, and the invisibility of non-binary participants in our data suggests
that non-binary identities are not being included in data collection.
To combat this issue, we recommend researchers to follow current guidelines to collect gender
demographics in a way compatible with complex gender.
As a final note, we would like to stress the iterative nature of this work.
The coauthors are interested in carrying it forward, and we welcome suggestions for future
directions.
Our dataset has been made publicly available, and we encourage further analysis and discussion.