hi i'm mitchell in this talk i'll
describe how we're aligning machine
learning performance metrics with the
reality of users experience
moderators researchers and engineers
want classifiers for critical social
computing tasks that they can easily
deploy at scale
tasks like toxicity detection or
misinformation detection
and enforcing many other
community-specific norms
taking toxicity as an example it would
appear we're in luck
for instance in this popular toxicity
detection task on kaggle
models achieved up to 0.95 rock auc
and these high scores can make people
feel pretty safe about using these
classifiers in their systems
in this talk we're going to demonstrate
that that 0.95 raw ksc score
is inflated as are many scores in social
computing tasks
we're going to show you that this
specific score should actually be
closer to 0.73 a much less convincing
number
so what's going on here well let's start
with an observation
unlike traditional ml tasks even experts
often disagree on the correct labels
for tasks in social computing and hci
for instance
we expect everyone to agree that this is
a dog but
while this comment contains some clearly
objectionable material
in the expert label data sets that we
looked at not every expert felt this
comment crosses the line
it was basically a split jury and this
means that the traditional approach of
evaluating ml models by aggregating
annotations to reduce noise
breaks because it flattens out
disagreement from test sets
so in this example a classifier could
predict toxic and receive a perfect
score because it matches the test label
but two phases of annotators might
disagree so scores become artificially
inflated
above the user experience in this work
we ask
can we transform existing ml metrics to
better handle this agreement
to try to do this we introduce the
disagreement deconvolution
so as i mentioned earlier there's lots
of disagreement
for tasks ranging from toxicity
detection to reddit moderation
somewhere in the range of a third of
annotators disagree when content is
labeled as problematic
and this agreement isn't necessarily
wrong or bad
tasks in social computing and hci often
attempt to categorize the messy
realities of our lives and
we don't want to drown out the
viewpoints that people hold
but it does place fundamental limits on
how well any classifier can perform when
deployed
because in practice these classifiers
often require a single decision
do i remove this comment or not and no
matter what decision we make
people are bound to disagree we think
that at the very least we should compare
predictions to each annotator's belief
essentially inserting one table for
annotator into the test set
so in this case an alpha classifier
predicts toxic
would receive a score of 0.6
but here's the problem some of those
annotations might be noise
in these crowdsourced data sets people
worry that labels might be clicked
accidentally or
someone might have momentarily
misunderstood or not paid enough
attention so we can't just take the
labels as is
how can we cut through that noise well
imagine a world where we could ask the
same person the same question
multiple times if we could do that
then we could find what we propose to
call an annotator's primary label or the
label they'd say most often
in our work to help reduce annotator
noise we propose creating tests that's
consisting of each annotator's primary
label
and it turns out this world is not
entirely imaginary
many crowdsourcing datasets already do
this for a small number of examples as
attention checks
so we introduce a procedure that takes
this existing
test re-test data and uses it to
estimate each annotator's primary label
for an entire data set
very briefly here's how that procedure
works first we estimate p
flip the probability that a given
annotation will differ from an
annotator's primary label using test
re-test data
or if there isn't enough test re-test
data we use svd to estimate p-flip
then we re-sample a new test set from a
label distribution
which we find by correcting the observed
distribution using p-flip and
with our procedure we find that when you
consider annotations agreement
that .95 raw ksc classifier which seems
like it should be good enough for
production
should actually be closer to a
dramatically lower 0.73
which is really not in fact we find that
no classifier could do better than 0.73
even if we simulate a perfect oracle
classifier that cheats and always
predicts aggregated labels
straight from the test set we can't do
better
and we also find big corrections for
other social computing tasks like
misinformation and adult content
detection but for
traditional ml tasks like image
classification the corrections are far
smaller
so to leave you with some parting
thoughts first how might we design a
product differently if we knew that the
classifier's best possible performance
was 0.73
and second this work is just to start we
hope the hdi community will do more to
help guide ml researchers work with
better metrics
thank you