Hello.
My name is Juho Pääkkönen, I’m a doctoral candidate in sociology at the University of Helsinki.
In this paper, called “Bureaucracy as a lens for analyzing and designing algorithmic
systems”, we explore how theories of bureaucracy could be used to think about decision-making
in sociotechnical algorithmic systems.
delegated to algorithms in important societal contexts, such as for instance child welfare protection
The background for this is that there are increasing worries about decision-making being
police work, or judicial decision making.
In such contexts, the worry is that algorithms are more prone to mistakes than humans when
making decisions about complex cases.
While there is a large critical literature on the power of algorithms and a technical
literature that seeks to improve their fairness and accountability, we believe that there
is still a need for accounts that conceptualize decision-making in algorithmic systems that
are sociotechnical, that is, involve both humans and algorithmic agents.
Theories of bureaucracy have recently been proposed as useful for this purpose.
Our paper explores Michel Crozier’s theory of bureaucracy to argue that
uncertainty is a key concept for analyzing how decision making works in algorithmic systems.
To do so, we first give a schematic introduction to three different families of theories
This conceptual argument is the main contribution of our essay.
about bureaucracy.
First, there is the classical ideal type of bureaucracy that originates
with the work of Max Weber.
Here, bureaucracies are hierarchical machine-like systems, where decision-making power is equated
with mechanical execution of organizational rules.
Second, this contrasts with the theory of “street-level bureaucracy” of Michal Lipsky,
which holds that rule application in bureaucratic organizations always depends on the judgments
of individual officials, who actually make decisions about people, such as police officers,
social workers, or judges.
In this view, in contrast to the previous one, decision-making only becomes possible
through human application of abstract policies to complex real-world situations.
Finally, there is Crozier’s theory, which also holds that bureaucracies fail to operate
on abstract rules alone.
However, here the focus is not on analyzing how rule-governed decision-making becomes
possible through human action.
Instead, Crozier emphasizes that what defines bureaucratic organizations is that they fail
A case he used to argue this was a French factory where machine breakage was an important
to overcome uncertainty in decision-making.
source of uncertainty in production, and consequently the maintenance staff had achieved an informal
position of power within the organization.
In Crozier’s theory, bureaucracies are organizations where uncertainty becomes a strategic resource
that both formal hierarchy and humans struggle to control.
The strict rules that seek to govern action in Crozierian bureaucracies fail to eliminate
uncertainty in the system’s operation, but instead redistribute it so that human action
can strategically draw on it in some other processes within the system.
As said before, our focus in the paper is on using this latter view as an analogy to
However, before going there, it is useful to briefly discuss the recent paper by
decision-making in algorithmic systems.
Ali Alkhatib and Michael Bernstein, who use the theory of street-level bureaucracy to analyze
errors in algorithmic systems.
role that is analogous to human street-level officials, in that algorithms are currently responsible for
Alkhatib and Bernstein argue that in many contexts, algorithms have come to occupy a
making actionable decisions about people.
This they view as problematic, because unlike human bureaucrats, the decisions of these
street-level algorithms are necessarily based on limited training data or prespecified rules,
which cannot be modified without retraining or reconfiguring the algorithms.
This, they argue, leads to increased potential of mistakes when making decisions about novel cases.
Thus, Alkhatib and Bernstein argue
that algorithm design should draw on literature on bureaucracy to analyze
how errors can be managed in inflexible decision-making systems.
In particular, they argue for increased attention to recourse processes and institutional checks
on algorithms.
We are sympathetic to this account, but contrast our approach with it.
The reason for this is that, first, the account does not acknowledge that in many cases, algorithms
act as support systems, and in fact do not occupy the role of human bureaucrats in making
Further, even in highly automated systems human action might not be eliminated, as we
autonomous decisions.
will soon see.
For this reason we turn to Crozier’s theory and view algorithms not as analogous to human
bureaucrats, but instead more as analogous to administrative mechanisms or tools for
controlling uncertainty in decision-making.
This view highlights that it is not the inflexibility of algorithms, but rather their speed, comprehensiveness,
decision-making.
and the autonomy ascribed to them that should be considered as the relevant novelties in
Most importantly, however, Crozier’s theory suggests that uncertainty is the key concept
Thus, in analogy with this view, we propose that also in algorithmic systems, uncertainty
with which to analyze how decision-making works in these inflexible systems.
– understood as unexpected events or situations which cannot be evaluated in advance –
becomes a resource which humans draw on to influence decisions.
bureaucracies, also algorithmic information processing leads to human discretionary action
As with the rigid hierarchical rules which cannot eliminate uncertainty in Crozierian
being redistributed to work at points of the decision-making process, where algorithmic
operations cannot yield actionable decisions.
In the paper, we draw on three well-known cases from previous literature, to illustrate
that this can happen both when algorithms are used for supporting decision-making, as
well as in the case of highly automated systems where humans are replaced by algorithms.
First, we look at the case of recidivism prediction support systems in U.S. court.
Algorithmic risk prediction has become increasingly popular in judicial decision-making during
the past decade, with the aim of reducing incarceration rates and making decisions more
objective.
However, the systems have also been subject to criticism and, as the sociologist Angèle
Christin has documented, the evidential status of algorithmic predictions is frequently contested
by judges.
Drawing on Christin’s study of the judges’ contestation strategies, we argue that the
interpretation of algorithmic outputs as evidence constitutes an important source of uncertainty
in algorithmic support systems, which can be controlled by actors in authoritative positions
in the institution.
Second, we look at the case of a housing allocation system in Los Angeles
as documented by Virginia Eubanks.
This system’s aim was to improve the efficiency and objectivity of housing allocation through
extensive automation of the decision process.
However, as Eubanks shows, the system also depended on survey data gathering which could
not be extensively enough standardized.
As a result, the data gatherers could impose their own views on how to score applicant
This shows that the “repair” work of keeping algorithms operational – such as data gathering
responses.
or technical maintenance – is a source of uncertainty even in highly automated systems.
Finally, we look at the case of quality control on crowd work platforms – another case where
decision-making is highly automated.
In the context of crowd work, automated quality checks make the evaluation of appropriate
task completion possible at massive scale.
However, as has been argued by Mary Gray and Siddhart Suri in their recent book Ghost Work,
we see simultaneously that automated evaluation processes lead to the responsibility of work
assessment being redistributed to the users of the system.
In this case, the workers whose output was being evaluated have to coordinate their actions
in order to avoid rejections by the automated system that were regarded as unfair.
On the basis of these cases, we see that algorithmic inflexibility is only one source of error
in sociotechnical systems.
In algorithmic systems – analogous to Crozierian bureaucracies – decisions are a joint product
of human action and automated processes.
Thus we argue that the bureaucracy analogy should be posed so as to recognize the continuum
of design spaces between supporting and replacing human work in algorithmic systems – a point
which has been long recognized in HCI and CSCW literatures.
However, Crozier’s theory provides a way to do this systematically, while highlighting
uncertainty as key to understanding how decisions and mistakes are made in algorithmic systems.
This point also relates to the goal of designing institutional checks on algorithms.
Unless we understand how humans draw on uncertainties to influence the behavior of these systems,
designing institutional structures to correct for errors will be difficult.
On this basis, in the paper we also discuss various initial design directions for recognizing
the role of human action in different kinds of algorithmic systems.
For instance, in relation to algorithmic support systems, we discuss the idea of a negotiation-style
system that could evaluate algorithmic and human expert predictions against each other.
In relation to cases where algorithms seek to replace human work, we argue that design
should pay more attention to how extensively automated algorithmic systems are presented.
The analogy between algorithmic systems and Crozierian bureaucracies helps highlight the
hybrid nature of decision-making in most automated systems, and the role that human action has
in steering the decisions made by seemingly autonomously operating algorithms.
From the Crozierian perspective, the central challenge for design is to recognize which
uncertainties cannot be automated away in algorithmic systems, and how automation redistributes
human action to influence decisions at these locations of uncertainty.
We argue that it is precisely this point at which the bureaucracy literature could be helpful:
In understanding how human discretion works at different levels of automation, and
how this residue of human action could be leveraged to improve the fairness of decision-making.
Thank you.