Hello, my name is David Holtz and I’m an assistant professor at the UC Berkeley Haas
School of Business.
Today, I’ll be talking about a piece of research conducted in collaboration with Sid
Suri and Liane Scult, both at Microsoft, titled “How much do Platform Workers Value Reviews?
An Experimental Method.”
Online markets such as Uber, eBay, Airbnb, and Upwork are increasingly central to our
economic and social lives, and a growing body of academic research aims to understand how
the firms that build these platforms control the behavior of workers through platform design.
In a 2019 paper, Greisbach and coauthors coin a term for this: “algorithmic management,”
which they define as “the practice of controlling the behavior of workers through strategic
use of information asymmetry, reputation systems, behavioral nudges, and opaque ranking and
matching algorithms.”
The work I’ll be sharing today builds on the existing research on algorithmic management,
and in particular, reputation-system based algorithmic management in two ways.
First, by developing a method that can produce numerical measurements of the intensity with
which workers experience algorithmic control, and second, by illustrating the utility of
that method through a case study conducted on Upwork during the summer of 2020.
A number of papers, including the aforementioned paper by Greisbach and coauthors, have focused
on defining and qualitatively measuring algorithmic management in a number of contexts.
There’s also an emerging literature, including the work I’m presenting today, that focuses
on supplementing these studies by taking the concept of algorithmic management and numerically
quantifying it.
Our approach to numerically quantifying algorithmic management uses online choice experiments,
which have recently been used in other contexts to characterize demand for free, digital goods.
While we measure workers’ perceptions of the value of online reputation, a number of
papers in economics have measured the actual value of online reputation on long-term sales
and other business outcomes.
Finally, recent research in economics and marketing has studied different types of incentivized
review programs, including some in which, similar to our study, sellers specify a price
that they are willing to pay in exchange for a review.
It’s easiest to explain our proposed method through a case study, and the research setting
for this case study will be Upwork, a large online marketplace for hiring freelancers
across a wide range of work categories.
One important design feature of Upwork is that both freelancers and clients are able
to rate each other, and the previous feedback received by workers is visible during the
hiring process.
The image on the left gives you a sense of what search results on Upwork look like; note
the “job success rate” displayed for the first freelancer.
Our proposed method works by first recruiting platform workers, in this case, freelancers,
into our experimental sample by hiring them for a very short job for which they are compensated.
In this case, we asked them to write a paragraph about the tools they use when freelancing,
and paid them a wage equal to an hourly rate of about 60 US dollars per hour.
After they had completed that recruitment task, they were faced with a choice experiment
– would they prefer a monetary bonus, or a five star review with positive text?
The image on this slide shows the choice with which they were presented.
The magnitude of the monetary bonus was randomly varied, and the freelancer received whatever
they chose - either the bonus, or the review.
We ran this experiment on a sample of approximately 500 Upwork freelancers across two categories
- graphic design and customer support.
These categories were chosen because of differences in the barriers to entry, the amount of horizontal
differentiation in the different labor markets, and differences in the demographic attributes
of the worker populations.
This slide shows our main result.
By analyzing the data collected in our choice experiment, we can estimate what percentage
of freelancers would choose a monetary bonus over a positive review as a function of the
size of the bonus, and estimate what amount of money the median freelancer would need
to be offered in order to forego a positive review.
We estimate that the median willingness to accept in our sample is approximately 48 US
dollars.
Notably, even when offered 175 US dollars, about a third of freelancers chose the positive
review instead of the bonus.
We were also able to detect heterogeneity in the value assigned to a positive review.
For more experienced freelancers, the median willingness to accept was about 35 US dollars,
whereas for less experienced freelancers, the median willingness to accept was approximately
66 US dollars.
We also tested pre-registered hypotheses regarding heterogeneity with respect to cost of living,
type of worker, and quality of pre-existing reputation, and did not find statistically
significant differences.
Although this was not a part of our original experiment design, many freelancers provided
open ended text responses explaining their decisions.
Coding these responses and identifying which explanations were most common helped us understand
the extent to which the willingness to accept that we measure using our experimental method
is a measure of algorithmic control.
Among freelancers who chose the positive review, the most common reasons provided were simply
valuing the positive review, considering the impact of the review on future earnings, and
maintaining or improving the freelancers’ reputation.
The three quotes shown on the right hand side of the slide provide examples of the types
of responses we received that correspond to each of these labels.
In contrast, among freelancers who chose the monetary bonus, the most common reasons provided
were economic hardship, which was often related to the COVID-19 pandemic, already having many
reviews, and simply valuing the money.
Once again, the three quotes shown on the right hand side of the slide provide examples
of the types of responses we received that correspond to each of these labels.
Importantly, across both the freelancers who chose the monetary bonus and those who chose
the money, there was a consistent discussion of the trade-off between potential future
earnings and immediate money in the form of a bonus.
To summarize, this work develops a behavioral experiment-based method to measure the intensity
with which workers experience reputation system-based algorithmic management.
To test out this method, we ran a case study on Upwork in June 2020, and found that the
median valuation of one review among workers in our sample was about 49 US dollars.
Surprisingly, even when offered 175 US dollars, one third of workers still preferred the positive
review.
The median valuation of 49 US dollars is extremely high when taking into account the average
salary among workers in our sample, which was about 13 US dollars for customer support
workers and 23 US dollars for graphic designers, and also the cost of living in the freelancers’
home countries.
If the proposed method were used across a number of different platforms and at different
points in time may help researchers and platform designers understand what product features
are most strongly associated with algorithmic control, and inform the direction of future
efforts to advocate for platform workers.
Although we believe these results are exciting, this work does have limitations - for one,
our case study analyzes a non-random sample of Upwork freelancers, and the fact that our
analysis only includes approximately 500 freelancers limits the statistical power that we have
to measure heterogeneity in freelancers’ valuation of positive reputation.
Future work can build on this research and also attempt to address some of this work’s
limitations, both by quantifying different aspects of algorithmic management, potentially
on different platforms and across different points in time, and by using alternative choice
experiment approaches, such as Becker-Degroot-Marshak lotteries or best worst scaling, which might
allow for larger sample sizes at lower cost.
That’s all I have for you today.
My co-authors and I are very open to questions, comments, and other types of feedback on this
work, so please feel free to reach out by any of the communication methods listed on
this slide.
Thank you for listening to this talk!