Today, I'm going to be discussing
the origins of algorithmic instabilities
and crowdsourced ranking.
First, I'll give some background.
Our work is inspired by the music lab study
by Salganik et al in 2006.
In this study, they looked at how
social influence affects the ranking of songs,
how popular songs become.
They find, through an independent metric of quality,
the best songs and the worst songs were correctly ranked,
but mediocre songs were hard to rank.
Sometimes they're near the top, sometimes
they're near the bottom.
And we want to understand why that is.
More specifically, our research questions
are, how does quality and option appearance jointly
impact decisions?
When is algorithmic ranking unstable
and doesn't reliably identify the best option?
And finally, how can we stabilize algorithmic ranking
such that the best options typically rank first?
To answer these questions, we create an experiment
with three conditions.
In the first condition, subjects are
given a question such as the area
ratio of larger and smaller shapes
and asked to guess the right answer.
In the second condition, subjects
are simply asked to choose the best of two answers visible.
And then the third condition, subjects
are asked to choose the best of two answers visible with one
answer randomly chosen to be called more popular.
In total there is 1800 subjects or about 600 subjects
per condition.
There are 10 questions per session.
And finally, questions were meant to be simple,
but with hard answers.
To better answer Research Question 1, the impact
of quality and appearance on decision making,
we plot the likelihood the best answers
chosen versus the answer quality difference delta.
And this metric is defined such that when delta is 0,
answers are equally good.
When delta is different from 0, either greater than or smaller
than 0.
The best answer is more obvious.
So long as delta is not equal to 0,
there is an objectively better answer.
In the control condition, we find that when delta is near 0,
there is equally likelihood of picking each answer,
even though, again, there's an objectively better answer.
Moreover, there is little impact of answer order.
Whether the answers rank first or second
doesn't impact the likelihood that the best answer is chosen.
And finally, we see good agreement
with the model, which is the dashed purple line
and the green solid line.
In the social influence condition,
we see a much stronger preference
for the first answer.
In other words, a much stronger position bias
is shown with the purple line being much higher
than the green line.
But moreover, there is still the similar qualitative behavior
as in the control condition, including
good agreement with the model.
Next, answer to Research Question
2, what creates ranking instability?
We plot the phase diagram of ranking stability
versus the answer quality of the worse answer on the y-axis
and the position bias P on the x-axis.
The model predicts instability and ranking by popularity
when the position bias is too high, i.e. in area 1,
and when the answer qualities too ambiguous.
For example, the worse answer is near the quality
of the best answer, which is when the A worse is near 0.
Instead, answers are ranked correctly
when the worse answer is very bad
or when position bias is low.
This is in qualitative agreement with Salganik et al,
and therefore, explains the paradox of popularity
in which mediocre answers vary wildly in popularity,
but a good answer still float to the top.
Finally, to answer Research Question 3,
stabilizing algorithmic ranking, we show in the right plots
that simulations demonstrate instability
of ranking popularity and recency ranking, i.e.
Ranking by the last vote.
It also shows that crowdsourcing breaks down
if a good answer has fewer votes, for example,
arrives later.
To address these issues, in the black lines on the right,
we create this algorithm called RAICR,
rectifying algorithmic instabilities
in crowdsource ranking.
This method is guaranteed to be the best method if data matches
the model, and it clearly shows advanced over the other methods
of the baselines.
So in conclusion, for the first time,
we modeled the relationship between option quality, option
presentation, and decision accuracy.
We find that answer quality helps
explain algorithmic ranking and stability.
And we create a simple algorithm with theoretical guarantees
to best rank answers.
The implications of our work is threefold.
First off, crowdsourcing is messy.
Our model shows that roughly 20% of votes are random,
and moreover, random votes, along with position bias,
help create ranking instability.
To reduce ranking instability and to improve crowdsourcing,
designers must reduce or control position bias, especially
with the first few votes given to each answer.
With that, I would like to thank my colleagues, Tad Hogg, Raissa
D'Souza, Kristina Lerman, and Marton Posfai,
as well as our funding sources, the US Army Research
office and DARPA.
And I will thank you for your time.