Hello, and welcome to the video presentation of our paper
"Between Subjectivity and Imposition Power Dynamics
and Data Annotation for Computer Vision," co-authored
by Martin Schuessler, who's co-presenting today,
Tianling Yang, and me, Milagros Miceli.
This paper features a qualitative study informed
by constructivist [?] theory and also by the three research
questions you can now see on the slide.
We conducted an investigation into work practices
of data annotation in companies producing data sets
for computer vision.
More concretely, we conducted several weeks of fieldwork
at two annotation companies in Argentina and Bulgaria
and interviewed annotators, managers, QA analysts, and copy
revision practitioners.
So in this paper, we defined data annotation
as sensemaking practices, where workers classify data,
in this case, image data, and allocate meaning
through their use of labels.
We also argued that that interpretative work
is informed on the one hand by the individual subjectivity
is of the annotators.
But also-- and this is the main point of our investigation--
we argue that the work of data annotators
is decisively shaped by corporate structures,
organizational routines, and power dynamics.
Now, previous research in the field of data annotation
has largely focused on the individual subjectivity
of annotators to account for biased labels, for example.
However, our findings showed that industrial settings
actually don't allow that much room for workers'
preconceptions or biases to broadly affect data
because their subjectivities, the subjectivities
of annotators are persistently constrained
throughout hierarchical structures.
And those structures are a reflection
of power imbalances within the annotation companies,
but they are also a reflection of power imbalances
between the annotation companies,
service providers, and tech companies commissioning
and also paying for the annotations.
Consequently, workers, annotators generally
annotate data according to labels imposed upon them
by their clients.
In this paper, we illustrate through the description
of three annotation projects how that imposition of meaning
is stabilized in everyday work practices
through the control exercised by managers and QA analysts,
and also, we show how it is naturalized
as business as usual.
So in the end, power to impose labels and classifications
upon data correlates with the position
occupied by each actor.
And more specifically, it correlates
with the position of capital or the position of financial means
or of influence of that actor.
Labels imposed upon annotators and through them on data
are hardly ever questioned.
But even if annotators ever perceive
those labels as opposing personal or corporate values
or even if they ever question the labels,
the view persists that the one who is paying
has the right to impose meaning.
So power imbalances that are inherent to
capitalism not only translate into asymmetrical labor
conditions in the space of need annotation.
Our findings showed that those very same power imbalances also
decisively shape labels and data.
And they will have later a crucial effect
on computer vision systems that are trained on that data
and on the individuals and communities impacted
by those computer vision systems.
So from these findings, we distilled several implications
for petitioners, namely that they need to document, reflect
and communicate the subjective choices that
went into data annotations.
And we see this as a responsibility
of both the clients that are commissioning
the annotation of data sets, as well as the management
of the annotation companies.
And the main solution here is documentation,
so the purpose, the composition, and the intention of data sets
should be documented.
And this is actually a best practice
that's been currently adopted in machine learning research.
If you checked NIPS and ICML recently,
you would have noticed that authors are now
required to include a complete description of the data
collection process, as well as the instructions given
to annotators, the method of quality control
if the major contribution of a paper is a data set.
And so translated into more practical terms,
we suggest that companies really keep track of the instructions
that they were giving to annotators.
But we also suggest that documentation
should be extended with the power-aware perspective
to make explicit the structures, decisions, and actors, as well
as the frameworks that have shaped the annotation of data,
and how you would incorporate such a power-aware perspective
we'll now illustrate with a couple of potential questions
you might ask.
So you could, for example, ask whose product do
the annotations serve and how?
Whose rationale is behind the taxonomies
that were applied to data?
Who resolved discrepancies in the annotation process?
And who decided if labels were correctly allocated?
With this presentation, we hope we
can inspire other researchers to make the work that
goes into creating data for machine learning visible
and to direct their attention to the power dynamics that shape
work practices in this space.
Thank you very much for watching.