good morning hello my name is Kyle Oba
thank you to all the organizers and the
volunteers for you know this has been an
amazing experience so far I'm very
excited to be here so you can probably
tell I'm a little bit nervous so I know
I'm just gonna dive right in alright so
I work at a company called patio Cola we
work on local local problems with local
clients when you try to find local
solutions as much as possible or a
research and design and development
company My partner and I started this
company before moving to Hawaii but
she's actually from Hawaii so that's
where this whole like local little
centered design and research comes from
we partner with other researchers and
designers and programmers but enough
about me
I'm sorry you can't see my slides Oh No
should I just try to restart that okay
okay cool thank you thank you that's my
presentation keynote is the thing it
used to work
okay so cool alright so diving right in
so the unprecedented so Shoshana Zubov
just recently published a book called
the age of surveillance capitalism I'm
not gonna pretend to have read the whole
thing because it's like over 700 pages
and I'm a busy person but yeah so
there's this idea they unprecedented and
in her own words the unprecedent is
necessarily unrecognizable right so when
we encounter something that's
unprecedented we automatically interpret
it through the lenses that we are
familiar with or the familiar categories
thereby rendering invisible precisely
which is unprecedented when I read this
I was thinking wow this is amazing
I admittedly was going to Center this
talk all around context but I really
like her idea here and so if you were
living or visiting Hawaii about a year
ago exactly almost you would have
received this on your phone it's all
caps so you know it's important
ballistic missile threat in bounds of
Hawaii seek immediate shelter and then
if you were you're like is that real it
says this is not a drill which you know
usually I'm people are drilling me at
8:00 in the morning on Saturday but
whatever yeah so from my own experience
you know I I saw this you know it was
Saturday morning I just kind of calmly
like doing stuff in the kitchen look at
my phone and I guess I'm gonna die right
and you know usually when I see you know
we're all gonna die but whatever you
know but when I see this type of message
on my phone and maybe some of you got
this last night but it's like there's a
flood coming maybe you know there's a
hurricane on its way maybe maybe you
should care maybe right and so that's
kind of the lens that I used to
interpret this message which is kind of
like what Zubov is saying it was totally
the wrong lens to use it was just a lens
I was familiar to me so I immediately
started filling my water filter in the
kitchen with water which seems
reasonable but like when you but then
like you know 38
later when they say just kidding just
just you didn't that's not real I really
like had time to evaluate like my own
behavior and and it really didn't make
any sense I mean like if we got hit by a
missile as pouring water I mean I don't
you know didn't really matter okay cool
so that's the idea of the unprecedent
how we require these lenses in order to
interpret new things and so obviously
this talk it's a little bit about
artificial intelligence which is kind of
this weird broad term so like houses
houses relevant to us today
so as builders and designers as a lot of
us are we need to create that new lens
for ourselves in order to interpret that
which is unprecedented to us and to the
community or into society yeah one of
the common themes to a lot of the work
that we do is that we'd like to try to
find the unrelatable things that are
invisible things that are undefined and
try to make them more relatable visible
and articulate them more fully and it's
only through the creation of these new
lenses that we can bend further these
discussions with our communities so as
an example of that I wanted to take you
all through a project that we did with
the Honolulu Museum of Art which is just
down the street and I highly recommend
it so about the time that that alert
came out we had the opportunity to do
this project with the museum and we
called it a design intervention and we
wanted to sort of forward the discussion
of surveillance technologies so at the
time they had this program called
classified that was produced by the
Doris Duke theatre which is in the
museum which was to highlight screening
well highlights of the of the program
were screening of films including lower
Patras citizen for documentary about
Edward Snowden and there was also a
panel discussion so the panel discussion
took place on you know just one
particular day involved Kate Crawford
one of the cofounders of the AI now
Institute revver Paglen and artist who
works with surveillance technologies
Laura Patris the director Edward Snowden
actually Skyped in and Ben Weisner who's
his ACLU attorney and it was moderated
by asan ilahe another artist who works
with surveillance technologies or
surveillance in general and so we were
invited to sort of
come and create an experience and so we
dubbed it a design intervention to
complement what was happening in the
panel so so they the museum called it to
my profile tour I'm not sure how I feel
about that name but you know that it's
stuck so that's what it is it's a my
poster my profile took my profile choice
it's a little it's a little 2004 a name
I think a little bit but it's cool and
so the purpose of our of our design
intervention was to sort of raise raise
awareness about things in Google that
can go wrong so the fal ability of face
detection face recognition technology
and problematic data privacy issues so
this is like about a year ago so we
weren't well I'll just leave it at that
so we were totally sure what was gonna
happen with those issues but we knew
they were problematic anyway so this was
this was mentioned reduce public to
specific technologies and the use cases
of machine learning learning in the same
giant amorphous AI thing and to replace
sort of those marketing buzzwords with
some a little bit more articulate
discussion and specific discussion about
these issues all right so what did we
build so we built individual offices I
don't know
elevator pitch that we built into the
individual eyes tours based on guests
faces match to our currently on display
in a museum okay and I know a lot of you
are thinking
isn't that what Google did in their app
for Google arts and culture and now I'm
not lying when I tell you this and you
know that that's what Liars say but
anyway we they really least there at
like two weeks before we went live and
it was just sort of uncanny I actually
didn't find out about their app until
after we after the panel and I was like
oh wow that that's disappointing but at
the same time it was really interesting
because they were doing something very
very similar I can I'm so I'm gonna kind
of walk you through how we did it and
some of the differences but I think it
was an uncanny coincidence and I think a
lot of it had to do with how good
certain types of Technology were like
face recognition and detection we're
getting at about that time like just and
how easy it was to apply so all right so
cool to my probe out here how did it
work so step one you had to opt in
not everybody had to do this as you can
imagine at a panel discussion where
Edward Snowden is skyping in in the
state of Hawaii where he used to work
it's a little bit weird to have somebody
ask you to take a picture of you before
you go into the panel so you had to opt
in and then if you did opt in we took a
picture of you and it said hey this is
the picture we're taking would you like
to opt in if you did we kind of labeled
you in the wheel world we gave you a
sticker and a number on there it's sort
of like sort of a symbolic like your
this is happening to you yeah I mean you
know let's be fair not every website
does this okay well they don't give you
a sticker well actually the stickers are
a thing but anyway okay oh yeah
incidentally ask me later I asked you
should give up all right cool so the
four-step customized tour so the panel
was an hour long so before you walked in
we took your picture when you walked out
we handed you a customized tour so there
was a lot that went into sort of
printing and formatting and laying these
things out ahead of time so that only
the live customized content would be
applied cool so the process so I know
this is a little small for you to see
I'm sorry about that this is sort of
like a version from the version of what
we put out in a handout and so basically
the top part of it is stuff that
happened ahead of time and the bottom
part is stuff that happened on site that
day and I'll walk you through that so
step one of I think almost every one of
these machine learning slash you know
whatever you want to call it artificial
intelligence projects is data collection
right or getting data from somewhere and
so as you can imagine when you're
working with a museum who hasn't really
done this type of thing before data
collection is hard data collection is
hard I in my experience no matter what
project I'm working on so we had to walk
through the entire museum find out where
all the art was write down all the
numbers associated with that art and
figure out what they were called what
room they were in and then get a photo
of everything that was that was a lot of
work and then we sort of processed the
images of the art as well as the
sculpture so anything that they had that
was going to be on this display at that
time that wasn't in sort of like a
special exhibition where
allowed to photograph the art we pass it
through a face detection neural network
which is essentially just says hey I
think there's a face here and I think
it's inside of this rectangle and then
you can highlight that or crop it out so
in this case we just cropped out the
faces and then we used a second neural
network which was a landmark prediction
neural network which essentially says
hey we're I think the eyebrows are
probably you know based on you telling
me this is a face I think this is where
the eyebrows are the eyes the nose the
mouth the jaw etc and the image on the
bottom there which is me I just wanted
to throw in here that although we did a
lot of this in Python because the
beginning of every web project using
machine learning is to install Python on
your machine this is something that you
can actually do in the web so I wanted
to put this link here this is an example
that runs in ml 5j s and is rendered
with p5.js or you know sort of like put
into the browser with p5.js it's super
cool so if you're like if you if you
love javascript you can do all this
stuff from the comfort of your
JavaScript but just remember you have to
install Python because that's just
required but don't be afraid of Python
it's just executable pseudo code all
right
next let's see that's a joke for the
Python people all right so this prepares
us for numerical comfort comparison so
and we can actually pass that set of
landmarks through a face recognition
neural network weather then essentially
render that image into a bunch of
numbers right like big vector all right
and so now that we have sort of a face
and we know where the landmarks are
we've turned it into a bunch of numbers
we can start matching it to things okay
so like on the so beast a ball that
stuff away and then on the day of the
event
people come in we take their picture
right and then we find the landmarks on
their face and then we can then render
them into numbers and then match them to
the art that they're similar to cool and
then since we have facial scary facial
recognition software we can then say hey
do you want your tor
oh we know who you are so here's
- right so that was another thing we
added so it wasn't just about matching
faces to art we also did a number of
other things so this is the tour folded
up that we handed to them so it has
their face it has their landmarks and it
has a number of super problematic
classifications that we said we're
attributes of you based on your face and
this was not meant to be like true it
was meant to be like super problematic
and something that people should
question so special thanks to Kyle
McDonald who is an artist who trained
the neural network to work with these
classifications on the labeled faces of
the wild data set it's another one of
things where you you can pop that thing
into anyway I won't go into it so these
labels were we're super problematic and
I think in some cases kind of insulting
it might tell you you're not attractive
right and some it might tell you you
look tired you know stuff like that and
you know that's not something you want
to hear from computer or anybody but
nonetheless computers will do that and
um so the next page we actually this is
the stuff that was besides the labels
was custom generated so here we were
able to say okay we think you look like
this painting we think you look like
this sculpture and then based on
everything that was in the frame when
you took your photo we match it we're
matching you using a different neural
network that does object matching and
I'll go into that a little bit later and
then because of your classification that
we gave you
we're gonna say this piece of art is
something you might be interested in
based on what was strongest in your
classifications and then whoop and then
we created a path through other pieces
of art that's sort of like provided this
sort of like trail that you could follow
from one our piece of art to the next
this was an educational project so what
we really wanted to do was to provide
people a way to question the technology
and have it be personal so that people
could have a further more articulate
discussion in the community so we also
wanted to highlight where our
technologies failed I'll go into that a
little bit and then also we provided
details about how the artwork was chosen
a very technical level if people wanted
to get them out as well
so cool so one of the things that we
really wanted to do was to say you know
what are the ways that we can use to
find art that we can match to people and
so one of the techniques that we came
across was something called reverse
image search and so reverse image search
essentially you take an image you use an
object detection neural network and you
kind of short-circuit it to provide you
with numbers instead of names of things
and then with those numbers you can sort
of compare them to other objects and
find in the data set other objects that
look the same so I'll just kind of let
that let that picture clean itself one
of the object neural the neural networks
are used was called vgg 16 it's 16
layers you slice off the last layer that
basically tells you what the thing is
and you're left with it with a giant
vector for every object that's numbers
here's another example of finding stuff
that's similar to the vase on the left
so once you've got these numbers you
basically have coordinates in this
multi-dimensional space and you can use
traditional distance algorithms to see
what's close to what and then once you
do something called PCA analysis or
principal component analysis you can
slam you know four thousand dimensions
down into two dimensions using this
algorithm which essentially maintains
closeness of points in multi-dimensional
space in the two dimensional translation
and then once you're in two dimensions
you can use traditional graphing
algorithm or graph algorithms to find
shortest paths like you would to find a
route through a city so in this case we
have a picture of a person you may have
heard of him and the and a vase on the
right hand side and we tried to find
items in the museum collection that
would sort of like transition from one
to the next I'm not sure how convincing
this is I kind of think it's convincing
maybe it's like bias on me for selecting
this but anyway you kind of get the
picture this is how we selected art so
lessons learned we had to try a lot of
different ways to try to find things
that would work - Matt - convincingly
match you to something that's in the art
right - a piece of art and once we got
to this point we're like yeah that's
pretty good Zuckerberg looks like that
guy so okay we're like
we're done alright but there's also
mistakes that are made and I should have
put like a warning or something on the
previous slide so that we started out
with this thing called the opencv
classifier to find faces and it turns
out it's not that good or at least it
was good for a while but then when you
know these convolutional neural networks
came out we ended up using one byte
called D live which hasn't I put that as
a Python API it got much better but we
were getting results like this and I I
just I mean it's like automatic
screenshots save right also privacy
policy this is one of the least the
lesser successful parts of the project
was we put a giant wall sized terms and
conditions on the wall to remind people
that we were taking their images and
making conclusions about them and
recording it on a computer and and
basically saying all the horrible things
we could do with that information and
then at the very end which they liked
but we're not gonna do and we're gonna
believe your data in 30 days but you
should really think about this and I
don't know what it is about putting
terms and conditions in front of people
but they don't want to read them so
almost nobody read this which is a
little disappointing for our graphic
designer that we work with but it did
teach this lesson that it was really
incumbent on us to make this more
approachable and consumable another
super difficult thing to deal with was
using the classifications and labeling
the data that exists in the Y or exists
currently right if you look at this it's
probably too small for you to see but if
you look at this up close you'll see
that there's only four ethnicities
available in the label faces in the wild
attribute classifiers black white Asian
and Indian I thought Asians and Indians
were kind of but whatever and you know
we're in Hawaii we're doing this project
in Hawaii there's a lot of Southeast
Asians right if you just say Asia and
you're like I don't know and there's
like it it's everybody almost and yeah
and there's no Hawaiian right so you
know there's so you kind of like leaving
out a giant portion of the population
and then also I'd probably not need to
go into it but you know art collections
that you're using this for also have a
bias
kind of taking a wider view you know I
found this when we were doing our
research there's obvious bias here I
don't need to go into it but the dataset
has a bias the engineers have a bias
there's bias out there people anyway and
then so this is Hawaii
Hawaii's weird and this is why it's my
concern why is super unique in the
United States we're number one in terms
of the percentage of non-white
population even looking at this chart
there's no category for Hawaiians
there's also no category for mixed mixed
race or ethnicity so so why is that
right because the bar graph tool didn't
really allow for that and then also
locally our Police Department just
bought 1200 axial on body cameras which
sounds a lot like letting my axe on body
cameras and I'm a little worried about
that because you know
excellent has an AI division
headquartered in Scottsdale Arizona and
you know if bias are the thing maybe
this is something we should be concerned
about why looks very different than
Scottsdale Arizona and a worthwhile so I
think in the words of so I'm gonna leave
you with a couple of coats so we're
worthwhile what's what exercise would be
to delete the word technology from
vocabulary in order to see how quickly
capitalism's objectives are exposed okay
marketing is powerful right AI I believe
is one of those terms which if not more
clearly defined it's essentially just
marketing right and AI means anything
that's all powerful and inevitable and
as long as we we allow that term to be
just thrown around without properly
defining it we're basically giving in to
this sort of desire to just say it's
inevitable it's all powerful it's gonna
happen let's just let it right so
where's it so there's an opportunity for
us and I'm gonna just kind of go through
this real quick but this is actually
happening like people are getting
tickets in China because their picture
goes across the intersection right we
don't really have to let that happen
but we do also this quote by having
anymore is also inefficiency is
precisely what shelters us from the APA
inhumanity of Taylorism and market
fundamentalism so when it efficiency is
the result of the deliberative
commitment by a democratically run
community
no need to eliminate it even if the
latest technologies can accomplish that
in no time I love this guy's books I
highly recommend them essentially
sometimes in efficiencies a good thing
we should think about it one think about
it when we eliminate it and then finally
so technology is not and never can be a
thing in itself isolated from economies
and it kind of economics and society
this means that technological
inevitability does not exist and I can't
I love this quote it's it's everything
that that that we love about the work
that we do is learning about the
communities and learning about the
technology that they use and
understanding when it's appropriate to
apply the technology and when it's
appropriate to step back and just sort
of listen so I encourage you all to take
this you know what did I end on a high
note so today to take what I've said
it's like go forth to your local
communities and see how you can create
your own lens and use your lens to shape
the future with these technologies
because none of this is inevitable so
thank you very much
[Applause]