Hi, my name is Evgenia Christoforou and I would like to present to you the work we have
done with my collaborators Pinar Barlas and Jahna Otterbacker.
It has become apparent that crowdsourcing and in particular image annotation tasks can
amplify and perpetuate society’s biases in computer vision algorithms.
Crowdworkers usually describe what is “worth saying” in addition to the content of the
image, especially on open-ended image annotation tasks where we can see reporting biases included
in the data.
In this study we view the temporal effect that significant global and local events might
have on the collected data.
By noticing the natural change in the crowdworkers environment due to the global pandemic affecting
humanity and the social unrest due to racial discrimination happening in the U.S.
Our hypothesis is that the impact of these events and the extensive media coverage can
lead to relevant concepts such as health and identity to be “at the forefront of thought”
for crowdworkers.
In order to study this, we have replicated an image annotation task that was originally
executed on December 2018 on the Appen platform and we replicate it from May to June 2020,
asking U.S. participants to annotate 597 images of people from the Chicago Face Database including
White, Latino/a, Black and Asian subjects.
For each image, we ask 3 distinct workers to provide us with 10 tags describing the
content of the image.
We collected the tags from the 2020 and 2018 experiments, and we have isolated the ones
referring to the topic of health and identity.
Our results show a significant increase in overweight tags in 2020 which in combination
with the spike happening in a similar period to search terms like “recipes” and “quarantine
workout” makes us believe that is possible that workers mention when describing an image
what is directly affecting them at the present moment.
Furthermore, we have seen evidence that workers might be subject to attentional bias if we
consider that looking at the external features cluster tags we have specific tags observed in 2020
compared to 2018.
With White skin tags being significantly more in 2020, while Pale/light skin tags being
significantly more in 2018 and a similar thing happening to Black and Brown skin tags compared
to Dark skin tags.
Additionally, we have seen more race-marking of images depicting non-White subjects in
2020 with significantly more race-related tags describing Black subjects and significantly
more tags in the African sub-cluster compared to the opposite behaviour happening in 2018.
Finally, we have seen a significant increase in the use of tags referring to subjects as
American or USA citizen in 2020, which might be due to the workers’ “state of mind”
that is influenced by the BLM movement, the upcoming U.S. presidential elections at that
the time and anti-Chinese sentiments due to COVID-19.
So, in this study we have seen the existence of a parameter conforming with societal events
that together with other factors can influence and create temporal variations in crowdsourced
data.
So, it is essential to take steps to identify crowdsource data that can be influenced in
a positive or negative way by temporal variations, in order to promote or restrain said effects.
And in order to do so we can take a few simple steps such as being aware of whether the crowdsourcing
task is creating new datasets or using existing datasets in order to be aware of pre-existing
biases and in order to be alert for temporal variations happening in the present dataset.
Furthermore, we need to recognize if the collected data reduce or augment the potential bias
due to major events and take a few steps to reduce the effect of undesirable temporal
variations together with a thorough documentation of the collection process during crowdsourcing
in order for the data to be able to be used and re-used during the training process.
Thank you!