hello everyone my name is ding wong and
i'm an hci researcher from google
research india i'm delighted to present
our work whose ai dream in search of
aspiration and data annotation to you
today and this is work done by me my
colleague shantanu and nitia
artificial intelligence machine learning
and internet of things have already been
on a fast growing trajectory with wide
applications in healthcare education
entertainment finance and more
the covid pandemic has further
accelerated the pace of ai adoption such
ai boom further increased the need for
data that's fundamental to the
development of ai and machine learning
models
data annotation and labeling work has
historically been dependent and carried
out by independent workers on crowd work
platforms
there has been a rise in private
annotation firms dedicated to providing
data labeling and annotation services
these third-party companies take up
contracts for such working bulk and
employ thousands of full-time staff for
annotation work
and this rapid growth has been well
reflected in the data annotation market
projection we can see it from the huge
numbers that's listed here
and various government has also picked
up the signal that both a both india and
chinese government have picked up the
importance of data annotation to both
job generation and poverty alleviation
and we know that the human in the loop
is here to stay that rather than being
simply displaced we are seeing the
professionalization of data annotation
at both both organizational and
individual level
prior hca and cscw work has examined the
experiences of platform-based workers
however we know little about organized
data annotation work how do material
infrastructure and organizational factor
of data annotation firms affect the
workers and affect the data in this
paper we investigate the work practices
concerning data annotation as performed
in the industry in india
our work further motivates the critical
mode of inquiry into organization of
work and labor in data production and
how they spread or evolve into other
forms across the gig economy
for this paper we conducted interviews
with the annotators the annotation
managers and ai practitioners the
interviews with the annotators form the
basis for our paper
for many being an annotator seem to
provide the gateway to ai industry
it is common for data annotation
companies to leverage the booming ai
industry in their job advertisement
from our interview we were surprised to
learn that most of our participants have
a bachelor's degree in stem subjects and
even one have a master's degree in
computer science
additionally there is this technical
round during their interview to test
their coding ability which is not
necessary at all for annotation tasks
despite an emphasis on technical
eligibility relevant past experience is
not valued
once recruited the annotators will
usually receive two kinds of training
the orientation one and pre-project
training during orientation training the
emphasis is on getting them familiarized
with with the work interfaces and
annotation work in general while the
project one is project focused specific
annotation tasks despite the interest
that annotators have in ai and its close
relevance to their work there is very
little coverage of ai during the
training
and the training quality varies from
company to company that not all the
firms provide adequate training before
dropping the annotators to the tasks
a day in an annotator's life starts from
8 to 9 a.m in the morning they work 8-10
hours to complete their daily tasks
before the pandemic they work on-site
and during the pandemic many of them
work from home
and when the question occurs during
their work they can only report it
through the packing order using the
company communication tools towards the
end of their day they send their work in
for quality check and they work in a
group of four to five under one quality
check
we contend in our paper with more detail
that being an annotator or working
annotation task is more than technical
that it is a constant exercise of power
through the organizational hierarchy
we know that annotators are chasing
every increasing daily target there is a
repercussion for missing such target but
no rewards for making or exceeding the
target the career perspective as an
annotator is very limited as they work
on this hierarchy with a huge bottom and
a tiny top despite the market growth
long-term job stability for them is
still very low
the annotators of course share mixed
view about the work they have and also
their career and annotation industry in
general
the majority of our participants we
spoke to were attracted to the field of
annotation under the impression that
they were working on machine learning ai
and autonomous vehicle projects though
the line of work they do is critical to
the success of such models which were
built on and tested upon the data that
they annotate such success has little
impact on their personal pursuit of the
ai and machine learning dream that they
were from they were far from becoming
engineers that they hoped to be one
day
the irony on the image
depicted on an image on the right well
captured irony in data annotation
industry as well this leaves us to ask
the questions that who benefits from
such work and who gets the labor for it
also where does this leave the
annotators when their work is
contributing to the elimination of their
career
the work we do contends to provide a
sensibility of data annotation industry
in general before we provide more
specific designs and tools
that currently the emphasis of data
quality puts dramatic pressure on the
annotators that we
how we measure the quality
at the moment speaks nothing towards the
annotator's experiences or the skills
they bring to the tasks it is merely a
performance metric
the transition from crowdwork to
full-time employment does not make the
jobs they do more visible the benefit of
a fast-growing industry does not really
translate to the improvement of the
annotator's job
we learned that two though the
annotators see their job with with a
very mixed view many still see the value
in the work they do and this value ought
to be recognized by more than just the
annotators themselves
what we ought to do is to provide a
viable transitional path for the
annotators to follow their dreams into
becoming a recognized part of the ai
production
and with that said we end our
presentation and thank you for your
attention