Today I'm presenting on our project Camera Adversaria, which is a mobile application
which uses procedural noise to filter a users photographs, such that they are more resistant
to deep learning based systems of surveillance.
This is work done in collaboration with Ben Swift of the Code, Creativity and Culture
Studio at the Australian National University and Terhi Nurmikko-Fuller of the Center for
Digital Humanities Research also at the Australian National University.
When we think of surveillance, usually we think of metaphors like Orwell's Big Brother
and Foucault's Panopticon.
And these two metaphors also dominate the academic theorising of surveillance.
Both focus on totalising systems and the state and so they tend not to be ideal metaphors
for describing non-state surveillance.
This paper is particularly interested in surveillance oriented by economic power rather than 'Discipline
and Punishment'.
And so there have been a number of attempts to describe this phenomenon, but this paper
takes its theoretical point of departure from Shoshana Zuboff's surveillance capitalism.
Zuboff describes surveillance capitalism in her book published last year not as a technology
itself but rather the logic that imbues technology and commands it into action.
Particularly the economic logic.
So the point is that technological development is not a natural deterministic process but
rather it is largely oriented by economics and profit-making.
And this is a similar argument to that made by Max Weber.
So to understand this kind of surveillance we need to understand it in a broader context
of economic relations.
The challenge this presents is that we're dealing with extreme asymmetries of knowledge
when it comes to these systems.
If you haven't seen it before I want to point to 'Anatomy of an AI System' by Kate Crawford
and Vladan Joler.
This is a large-scale design project that attempts to reveal the extremely complex economic,
social, technological networks that are required for Amazon's Echo to function.
And this includes everything from mining operations to AI through to networks through to end-of-life
processes for these devices.
The project has slightly different aims, but in terms of what exists, this is the best
picture that we can start with as a model for systems of corporate surveillance, and
particularly today's models which really are powered by deep learning and what is sometimes
called artificial intelligence.
So there are two things I want to draw your attention to with regards to this image.
First is just how brittle this system is and that we as users only touch a tiny fraction
of the broader system.
I also want to note that most of this network is still a collection of human relations,
and there may be scope in HCI for us to consider these interactions that are beyond the confines
of use that we normally consider.
And given that we can only touch a tiny fraction of the system that enables surveillance capitalism,
what can we do in this space to give some power back to the subjects of surveillance
capitalism.
So we want to treat this as a design problem.
And particularly we're adopting a methodology of critical design.
Critical design is 'critical' in the sense of critical theory of the Frankfurt school.
It comes from theorists Dunne and Raby who worked primarily in industrial design but
it has been adopted also in HCI as a research-through-design methodology that foregrounds the ethics of
design practice.
So particularly our project is concerned with photographic surveillance, and how that is
consumed by technology companies.
This anatomy of an AI system does a good job of revealing the scale of these networks but
in general I think we are dealing with an extremely information limited scenario, where
the way that this system operates tends to be entirely unknown to end-users.
Mobile computing devices have totally revolutionised personal photography.
So we've been taking photos at a much higher rate for decades now and these photographs
have been accreting in social media databases.
But for a long time the tech really didn't exist to interpret these images beyond what
you could convince a user to describe in them.
And all of this has changed with deep neural networks.
Deep learning has totally revolutionised computer vision and really made a semantic interpretation
of images a possibility at scale.
What we're really talking about in terms of this semantic image classification is often
a system, which for our purposes we can treat as a black box, that takes an image in pixel
data and returns prediction values for a series of labels.
So deep learning activated the economic value of large databases of users' photographs.
One example of how this has occurred is work by Geoff Hinton's lab at the University of
Toronto which was purchased by Google and moved into production in just a little over
six months, according to the Google research blog.
And so, its clear from that kind of statement just how economically valuable this technology
was and is to tech companies.
The other thing that I want to talk about is the camera itself is becoming a surveillance
device in a more literal sense.
So in another article posted by Google's research blog in December, they confirmed that a future
direction was the inclusion of deep learning within the camera app itself.
So if you care about privacy, this should be setting off some alarm bells.
Privacy work in HCI has focussed so far on redactive methods.
And by that I mean, taking visual information out of a photograph; whether its by blurring
or covering it up or camouflaging or even generating things to sit over the top.
Bizarrely, even cryptographic methods, and certainly blurring and pixelation, appear
to not affect deep learning based machine vision in the same way that they affect human
vision.
And so for all the talk of machine vision; human vision and machine vision have fundamentally
different capacities.
And this is a recurring theme of this paper.
So I think one of the things that we're missing in the HCI discourse is a question of "privacy
with regards to whom?"
And so redactive methods disrupt human vision, but what if we want to maintain the social
function of images, what if we still want to share images online, but we just don't
want to be subject to this kind of additional system of surveillance.
And so, we need divergent privacy practices when we require privacy with respect to humans
and machines.
So as surveillance systems have grown so has resistance to these systems, and particularly
I want to talk about briefly about the work that has happened in the art and design community.
This is a project 'CV Dazzle' by Adam Harvey, where makeup and hair design are used to disrupt
the standard facial recognition algorithm.
Another work in a similar vein is 'URME' by Leo Selvaggio, and in this work, Selvaggio
would sell very high resolution masks of his own face and allow people to walk around potentially
surveilled areas and adopt his facial features.
And this had the effect both of allowing others to pass undetected in his identity but also
it introduces noise into his own surveillance trace.
I also want to point out that, I think we can consider the redactive methods that we
see in HCI research as being equivalent to camouflage post hoc.
So we're still obscuring the face in some way.
Another mode of resistance is what I will call 'Avoidance'.
This is a project 'iSee' by the Institute for Applied Autonomy where they created a
mapping application that would lead you from point A to point B by taking the path of least
surveillance.
We can also consider obfuscation methods, such as loyalty card swapping in the same
manner.
Where intentional behavioural change is done in order to garble data.
Another mode of resistance is sabotage.
This is where subjects destroy or disable the means of surveillance.
We do see it playing out usually in cases of state surveillance.
I think we can consider sabotage as a final mode of resistance when all other options
have been exhausted.
So we can put these into a matrix with one dimension as automutative (or self-changing)
and exomutative (or external-changing) and another dimension as corporate and state surveillance.
Now I split corporate and state surveillance not because these necessarily capture the
exact differences but I think that there tends to be a different set of resistance possibilities
when we are talking about corporate and state surveillance.
And the point of putting this into a matrix is to identify what we think is an under-explored
region of the design space.
So, in the case of state surveillance, the state owns and operates the means of surveillance.
And so, a subject's only mode of resistance if they wish to change the system itself tends
to be sabotage.
Looking back at the 'Anatomy of an AI System' its actually the subjects of suveillance who
own and operate the means of surveillance.
So if we can counteract this knowledge asymmetry in the system of surveillance, we can design
to disrupt not only our own behaviours but other aspects of the surveillance system.
So I am going to go into a demo now.
So this is the app.
It should look fairly familiar to anyone who has used a standard mobile camera application.
Capture button is in the centre.
We can see that we're processing images and they gallery appears on the right hand side
and on the left hand side we can flip between front and rear cameras.
Hey!
And so, what I'm going to do now is just take some photographs of things on my desk that
I happen to know are part of the possible ImageNet labels.
My microphone and this birthday candle.
And so when we open the gallery view what we'll see is both the image itself and also
indication of what a standard machine learning model can actually see in this image.
And by dragging this slider at the bottom, we can actually increase the perturbation
of the image to the point that the network can no longer make sense of these images.
For something like this computer keyboard, it actually turns out to be a bit harder to
get to that level.
And so we actually have to increase the level of perturbation perhaps beyond what a user
might be happy with.
And so we can drag this all the way up to essentially full and get and extremely noisy
image, but really the point to this slider is to allow a user to make a decision as to
how much perturbation, how much noise they are willing to accept in their images in order
to increase the likelihood that they can avoid surveillance.
Now all of these perturbed images, in their noisy form are available to the rest of the
phone such that they can be backed up on a cloud service or posted to a social media
site and hopefully when they get passed through a computer vision system they're a bit harder
to interpret.
So how does this work?
Camera Adversaria adopts a set of practices developed by the deep learning research community
called 'adversarial perturbations'.
So to return to that model of image classification, an adversarial perturbations is a specifically
designed set of pixel values that can be added to an image such that the computer vision
system confidently misclassifies the image.
This property of convolutional neural networks was discovered in 2014 and since then it has
become a significant subdiscipline in deep learning research.
In around 2017, it was discovered that universal adversarial perturbations were possible.
So that is a single filter that can be applied to any image that causes the machine vision
system to misclassify in a significant number of cases.
Most adversarial perturbations are computed as an optimisation problem with the network
but this paper by Co et al. showed that it is possible to to generate universal adversarial
perturbations with just procedural noise.
This is an ideal approach for resisting surveillance because the adversarial perturbation can be
computed cheaply and uniquely for each image.
The initial idea was for a online service that would perturb people's images.
We soon realised that transmitting user's photographs to a foreign server would introduce
so many trust issues as to defeat the purpose.
In addition as we've discussed these deep learning systems are being applied in the
camera app itself as so it became clear that we would have to introduce our adversarial
perturbations at the source and so that meant creating a camera app.
So aside from designing the adversarial filter this application isn't particularly complicated.
And so this raises the question, why hasn't any one done this before?
Lots of other deep learning research has been quickly turned into free online services;
DeepDream, Pix2Pix, GANs, that kind of thing.
But nowhere we have discovered has anyone in adversarial perturbation research community
attempted to make adversarial methods available to subjects of surveillance.
And this includes some researchers who developed a mobile app purely to demonstrate their methods.
To understand why, we need to look at the language of the research community.
In deep learning literature, adversarial perturbations are described in the language of threats,
exploits and bad actors.
Deep learning systems are presumed to be good, or at least neutral and those wishing to disrupt
it are "malicious".
And the literature is full of value-laden language; these are just some examples from
the work that we have read.
"Machine learning classifiers are known to be vulnerable to inputs maliciously constructed
by adversaries to force misclassification."
"This linear behaviour suggests that cheap, analytical perturbations of a linear model
should damage neural networks."
"Know your adversary: 'If you know your enemy and know yourself, you need not fear the result
of a hundered battles.'"
And even the paper whose approach we modelled our filter design on says "it is important
to ensure that such algorithms are robust to malicious adversaries."
So I want to state, just for the record, that adversarial perturbations are entirely non-destructive.
This is not a form of sabotage.
They confound a machine vision system only within the scope of that particular image.
So I think this raises a really important question about the role that the research
community plays in countenancing these systems of surveillance.
And so part of the critical work of Camera Adversaria is to extract these methods out
of a literature that describes them as 'malicious' and 'offensive', and really reclaim it as
a mode of defence against surveillance.
To return briefly to Dunne and Raby, we find in their initial work on critical design this
significant point on ideology.
Most design affirms the status quo, reinforcing cultural, social, technical and economic expectations.
And clearly, I think, the same thing is happening in the research community.
We are building tools of surveillance entirely uncritically and maligning any attempts to
get around the system.
Now one significant limitation of Camera Adversaria is that it is just a stopgap.
It is fair to expect that computer vision algorithms are going to improve and probably
become less susceptible to adversarial perturbations.
And while it is fair to assume that adversarial methods will also improve the point of this
project is not to start an arms race.
And so the point of Camera Adversaria instead, is to reveal to some degree the surveillance
system and present an alternative.
Thanks for listening.