Hey everyone thank you so much for coming to my talk my name is Yiqun Chen and I'm
currently a postdoc at Stanford data science. I'm super excited to talk to you today about
our CHI 2023 work on why when and from whom do we collect and Report race and
ethnicity data in HCI research. Before I start my talk I want to give a huge shout
out to my amazing mentors and collaborators on this project Angela Katharina and Alexandra
now to this group of audience it's probably not a surprise that it's important for us to engage with
a diverse pool of participants in HCI research in particular a racially and ethnically diverse
participants is really critical for creating safe inclusive and Expo technology for instance if you
just do a quick news search or even search within the ACM Digital Library you will see that there's
been increasing attention and work dedicated to this topic however what remains a little less
clear is where we are at a status quo in terms of engaging with such diverse participants and how
do we make progress from what we are right now at this day and age in particular in our work we've
broken it down into three different pieces in the first research question we're asking when are the
participants raised and authenticity reported in HCI research in the second question we asked what
are some of the considerations that speak both for and against collecting such information finally we
outline some relevant considerations on how do we collect report and use race and ethnicity of
our participants should we decide that it's important for us to collect this information
now to answer the first question we conducted a systematic literature analysis of recent five
years of chi proceedings and try to understand who are the research participants that are currently
present we start with about 4 000 articles in ACM digital library and then we filtered it down to
about 10 percent by using the following keywords which include either race racial ethnicity or one
of the identifying keywords of our major race or ethnicity group in North America
this restricted our articles down to 340 and then we went through the Articles manually to
understand which ones included detailed race and ethnicity information of the participants
it turns out that out of other articles only 93 of them included such information and this
amounts to less than three percent of all the chi papers from 2016 to 2021
in addition we note that more than 90 percent of these articles start us-based and we hypothesis
that this could be either due to the fact that a lot of these keywords are more systematically
collected and annotated in the U.S context alongside with the fact that U.S census have been
collecting such data for historical reasons for a very long time in addition we note that us-based
studies have a very similar participant's profile in terms of race and ethnicity distributions when
compared to the recent census data in addition we've also broken down this data by looking at
the distribution of race and ethnicity of the study participants both in terms of a Time
Dimension from 2016 to 2021 and from a study size component looking though at those studies that are
larger meaning more than 30 ish participants and the studies that are a little bit smaller
as you might note on this graph the smaller to medium studies especially in recent years have a
lot more higher proportion of non-hispanic white participants and this is partly due to the fact
that a lot of the recent work are dedicated to understand such technology use and uptake being
dedicated race and ethnicity minorities which explains the uptick of this group in recent years
now moving on what are some of the considerations that we've outlined that speak both for and
against collect race and ethnicity now we first took a crowdsourced wisdom approach
where we collected quotes from the authors who have published the paper that included this
information in the recent Chi proceedings as well as a follow-up survey with authors to understand
why they chose to collect such information of their participants in the first place
of course this is only going to be a subset of people who mostly we're going to advocate for
collecting this information therefore we also conducted a comprehensive literature research
where we synthesized existing discussions on such information collection in adjacent
fields and topics including U.S census biomedical data science social sciences and computational
fairness to understand the potential benefits and especially the intended consequences of
reporting racial and ethnic data HDI so what are some of the big takeaways that we get from this
in terms of the reasons why the authors collected such information of their participants for their
kite paper we see that top two reasons that jump out to us are first the authors want
to use this information to increase or at least acknowledge the lack thereof of their
external validity of the study in addition authors are motivated by the inequality and
differences between different racial groups in terms of their topic of study in Prior work
in addition some other reasons include for instance they might be designing a study to
understand the differences to motivate future work and it could just be that they're required
by external funding requirements which speaks to a small subset of the participants we surveyed
now moving on to the more interesting part about the challenges we see that deciding around race
and ethnicity data collection is not easy there are a list of important concerns and topics here
for instance authors might be concerned about privacy and post-hoc explanations that could be
using some of these findings to actually reinforce harmful racial stereotypes in addition while a lot
of our emphasis in the first question are are concentrated in North America categories we
acknowledge that there is not always a systematic categories or best practices people can leverage
in other sides of the world where race and ethnicity a lot more contextual and very hard
to come up with categories especially there hasn't been a prior study on it in addition we note that
some authors pointed out that non-representative categories might actually further perpetuate harm
because the group of participants are actually not accurately represented especially their challenges
finally we also want to highlight that even though the context of a paper is largely focused on the
collection of data data collection and Reporting in itself is not really a solution to existing
inequalities what we want to call a special attention to the fact that data collection is
merely a start rather than an end and inequalities should always be actively acknowledged and
addressed because just acknowledging the differences via data is not enough
now finally our last aspect look into how we should collect and Report race and ethnicity data
here are some of the very Abridged version of our recommendation so we would love to have you read
through them in our actual paper and provide any feedback you might have in this very brief
version we recommend our authors always start with existing resources for instance it could
be the census data and categories in the US but we also want to make sure that the authors and
researchers are aware of the fact that these categories no matter how comprehensive they are
will still have limitations therefore we encourage researchers to seek feedback from the community
and groups they're trying to get this data from multiple choices in this case could be a good
start but always leaving a self-described option will enable us to understand better for instance
what are the survey population most identified with second of all when racial and ethnic
categories are less established in other parts of word for instance it's really important to
start with a small group pilot study to understand participant self-description in addition we also
want to highlight the fact that we talked about a site ago which is the fact that it's often easy
for people to get post-hoc explanations based on race and ethnic categories so we should be very
careful about potential misinterpretations that our results and our interpretations could lead to
finally while race and ethnicity is undoubtedly a very important dimension of participants in many
studies we also want researchers to consider other data for instance if we're trying to understand
the difference in terms of the language spoken at home we should just really be collecting that data
directly rather than looking at a proxy such as race and ethnicity now finally what have I told
you today first of all we note that very little recent type papers reported participants raised
an ethnicity and key motivations for those who did include strengthening the external validity of
their study and to address the exit the existing gaps in terms of the subject of understudy across
different racial and ethnic groups second of all we want to call particular attention the racial
and ethnic data collection is just a start rather than an end we should really be leveraging this
data as an opportunity and motivation for further combating systematic inequality discrimination
and oppression in HCI and Technology design more broadly finally words of wisdom our work
is definitely not a one-size-fits-all solution if anything we really meant it to be a starting
conversation about race and ethnicity data collection and interpretation HDI research
so feel free to give our paper a read and we're really excited to get your feedback on this topic