[Music]
yeah so this talk is that it's not
actually that's scary it just has some
unusual things some flickering moments a
few unsettling pictures nothing graphic
nothing where you'll be looking at it
and you'll say like wow this is
disturbing I and I know why it's
disturbing no it's it'll be more like I
don't know why but this looks this is
weird to me and we'll be asking why and
we'll also be talking about some things
some of the kinds of things that we
don't usually talk about and all sorts
of stuff kind of wants to pour down
river when you open locks like that and
so I want you to be here for it but if
you can't be here for it then take care
of yourselves otherwise hello my name is
Oishi and today I want to talk with you
about a hobby I want to share some of
the things that I've learned at the
intersection of computational
neuroscience and machine learning for
years I've been fascinated by how we
think how we perceive how the machinery
of our bodies results in qualitative
experiences and why we have the kinds of
experiences we do why they're shaped
like this and not like that why we
suffer for years
I've also been fascinated by AI and I
think we all have we're watching these
machines begin to approximate the tasks
of our cognition sometimes in unsettling
ways and so today I want to share with
you some of what I've learned some of it
is solid research some of it is solid
speculation and all if it speaks to
truth that I have come to believe which
is that we are computations our worlds
created on an ancient computer powerful
beyond imagining so let's begin
part-1 hallucinations this person is
Michaela pareo Nieto and he has
something to show us it starts with
these simple patterns and splashes of
light and dark kind of like images from
the first eyes
these give way to lines and colors and
then curves and more complex shapes
what's happening is that we are diving
through the layers of the inception
image classifier and it seems that there
are whole worlds in here shaded multi
chromatic hatches the crystalline farm
fields of an alien world the cells of
plants to understand where these visuals
are coming from let's take a look inside
the job of an image classifier is to
reshape its input which is the square of
pixels into its output which is a
probability distribution so the
probability that this image is of a cat
the probability of a dog a person a
banana a toaster it performs this
reshaping through a series of
convolutional filters now convolutional
filters are basically Photoshop filters
each neuron in a convolutional layer has
a receptive field some small patch of
the previous layer that it's looking at
and each convolutional layer applies a
filter specifically it applies an image
kernel a kernel is a matrix of numbers
where each number represents the weight
of the corresponding input neuron so
each pixel in a neurons receptive field
is multiplied by its weight and then we
sum all of them to produce the output
neurons value we apply that same filter
across every neuron in a layer and the
values in that filter are learned during
training so we feed the classifier
labelled image that's something where we
know what's in it
it outputs predictions and then we math
to figure out how wrong those
predictions were and then we math again
to figure out how to change the values
in this fill
to produce a better result so the term
for that is gradient descent the deep
dream process which is what's creating
these visuals inverts that so this
visualization is recursive to compute
the next frame we feed the previous we
feed the current frame into the network
we run it through the networks many
layers until we find that layer that
were interested in and then we mask what
could we do to the input layer to make
that layer activate more and then we
adjust the input image in that direction
so the term for this process is gradient
ascent finally we scale the image up
very slightly before feeding it back
into the network for the next frame that
kind of keeps the network from fixating
on the same details in the same places
and it also creates this really wild
zooming effect every hundred frames or
so we move to a deeper layer or a layer
that's off to the side inception has a
whole lot of layers and they are not all
arranged in a neat linear stack and that
gives us this so we started with these
rudiments of light and shadow and now
down here we kind of have a city of
kodamas situation happening but then
we're about to enter the spider
observation area in which spiders
observe you but it's okay because soon
the spiders will become corgis and the
corgis are going to become the 70s later
we've got this space of nearly human
eyes which will transform into dog slugs
and then dog bird slugs deeper
unfortunately we had a saxophonist
teleporter accident
and finally the flesh zones
with a side of lizard so when I first
saw this I was like should I tell the
story when I tell the story when I first
saw this I I thought it looked like
nothing so much as US President Donald
Trump and and I resolved to never tell
anyone that certainly not on stage until
my best friend was watching the same
video and she said you know this this
just kind of reminds me of and I think
that actually says more about the state
of our neural networks than this one I
think the lizard juxtaposition has
something to do with it
but I do want you to notice and think
about what it means that all of the
flesh in this network is so very pale so
this is pretty trippy yeah why is that
what does it mean for something to be
trippy to figure that out let's take a
look inside ourselves meet Scully Scully
doesn't need all this trust we're just
looking at Scalia's visual system which
starts here in the retina now Scully is
retina your retina our retinas they're
actually pretty weird light comes into
them and it immediately hits a membrane
there's a layer of ganglions which are
not actually particularly photoreceptive
that they are a little then there's a
layer of more stuff that does important
things presumably and at the back are
the photoreceptors the rods and cones
which sends luminance and color so when
light comes in it has to go through
these four layers of tissue hit a
photoreceptor that photoreceptor is
going to get excited it'll send out a
signal to its ganglions which then have
to send it to the brain somehow through
the optic nerve which is drilled through
the center of your eye which means the
sent the sensors in our eyes are mounted
backwards and there's a hole in the
center of them and it's all okay because
we patch it up later in software
a couple of other problems with our eyes
too one we have about 120 million
photoreceptors and there's 10 times your
ganglions than that so it can't be a
one-to-one mapping and to the bandwidth
of our optic nerve is 10 megabits which
is not you know like a lot I don't know
when the last time you tried to watch
video over a 10 megabit connection is
but it's much slower than Wi-Fi our
cameras are about 100 megapixels it's
not gonna work and so our retinas do
what you would do if you were given does
design constraints they compress the
data each ganglion is connected to a
receptive field that's a hundred or so
photoreceptors that is that are divided
into a central disc and a surrounding
region so when there's no light on any
of them the ganglion doesn't fire when
the whole field is illuminated the
ganglion fires very weakly when only the
surround is illuminated and the center
is dark half the ganglions in your
retina fire release strongly and the
other half don't fire at all in the same
situation but those ganglions behave the
opposite way they fire when the center
is bright and the surround is dark so
these different species of ganglions
they're not actually different species
they're just ganglions that naturally
behave this way they're scattered
they're distributed evenly throughout
your retina and if you think about what
this does creates an edge detection
filter so we're doing processing even in
our eyeballs in order to down sample the
image coming in from our photoreceptors
a hundred times while retaining vitally
important information namely where the
boundaries of objects are okay so then
the signal is going to go into the brain
it's gonna hit the optic chiasma where
the data streams from our left and right
eyes cross where we extract stereo
vision it's gonna get processed by the
thalamus which is a switching center for
all kinds of signals in your brain it's
responsible amongst other things for
running our eyes autofocus each step in
the signal pathway is doing a little bit
of processing
extracting a little bit of information
and that's all before we even get to
hear the visual cortex all the way
around back our visual cortex is
arranged into a stack of neuronal layers
the signal stay is actually pretty
spatially coherent throughout the visual
cortex so there's some slice of tissue
in the back of your brain that's
responsible for pulling faces out of
this particular chunk of your visual
field I mean more or less your brain is
very squishy each neuron in a layer of
our visual cortex has a receptive field
some chunk of the entire visual field
that it's looking at and neurons in a
given layer tend to respond to signals
in the same way and that operation
distributed over a layer of neurons it
extracts features from the visual signal
early layers extract simple features
like lions and curves and edges and then
later layers extract more complex ones
like gradients and surfaces and objects
Aya's faces and movement it's no
accident that we see very similar
behavior in Inception because
convolutional neural networks like
Inception were inspired by the design of
our visual cortex of course our visual
cortex is different from inception in
many ways inception is a straight shot
through it has branches but no cycles
our visual cortex contains feedback
loops these pyramidal neurons that
connect deeper layers to earlier ones
those feedback loops allow the results
of deeper layers to inform the behavior
of earlier layers so we might turn up
the gain for edge detection along the
edge of what is later detected to be an
object this lets our visual system adapt
and focus not optically but
attentionally it gives us the ability to
ruminate on visual input well because we
become consciously aware of it improving
our predictions over time you know this
feeling I think you think you see one
thing
and then you realize that something else
and these loopback pyramidal cells in
our visual cortex are covered in
serotonin receptors different kinds of
pyramidal cells respond to serotonin
differently but generally they find it
exciting and don't we all you might be
familiar with serotonin in its starring
role as the target of typical
antidepressants which are serotonin
reuptake inhibitors
when serotonin gets released into your
brain they make it stick around longer
thereby treating depression some side
effects may occur most serotonin in your
body is actually located in your gut
where it controls bowel movement its
signals to your gut that it's got food
in it and it should go on and do
whatever it does to food and that seems
to be with the molecule signals
throughout your body resource
availability and for animals like us
with complex societies resources can be
very abstract social resources as well
as energetic ones that your pyramidal
cells respond excitedly to serotonin
suggests that we focus on that which we
believe will nourish us now it's not
correct as a blanket statement to say
that pyramidal cells are excited by
serotonin there are different kinds of
serotonin receptors and their binding
produces different effects so 5-htt 1a
receptors tend to be inhibitory somewhat
drowsiness inducing 5-htt 3 receptors in
the brain they're associated sensations
of queasiness and anxiety and in the gut
they make it run backwards so
anti-nausea drugs are frequently 5ht 3
antagonists there is another serotonin
receptor one that the pyramidal cells in
your visual cortex find particularly
exciting
this is the 5-htt 2a receptor this is
the primary target of every known
psychedelic drug this is what enables
our brains to create psychedelic
experiences so
you go to a show you eat a little piece
of paper and that piece of paper makes
its way down into your stomach where it
dissolves releasing molecules of fly
surgit acid diethylamide into your gut
now LSD doesn't bind to 5 HT 3 receptors
particularly so if you feel butterflies
in your stomach it's likely just because
you're excited because you know what's
gonna happen and what's gonna happen is
this LSD will diffuse into your blood it
has no trouble acrossing the blood-brain
barrier because it's tiny but powerful
like you it will diffuse deep into your
brain into your visual cortex where it
finds a pyramidal 5-ht to a receptor and
locks into place it will stay bound
there for around 221 minutes that's 4
hours which is a very long time they
think that a couple of proteins snap
into place and form a lid overtop of the
receptor trapping LSD inside which would
help explain why it's so very potent
with typical doses about a thousand
times less than most other drugs so
while it's rattling around in there the
little molecule of LSD is stimulating a
feedback loop in your visual cortex it's
sending the signal pay attention what
you're looking at may be nourishing the
pattern finding machinery in your brain
then starts to run over time and at
different rates in one moment the
pattern in a tapestry seems to extend
into the world beyond it in the next
it's the trees that are growing and
breathing the perception of movement of
visual hypothesis that's been allowed to
grow wild with deep dream we asked what
would excite some layer of Inception and
then we adjusted the input image in that
direction but there's no comparable
gradient ascent process in the
biological psychedelic experience that's
because we're not looking at a source
image we're looking at the output of the
network we are the output of the network
the output of your visual cortex is a
signal carrying visual perceptions these
like proto qualia which will be
integrated by other circuits in your
brain into your next moment of conscious
experience
inception poor thing never gets that far
we never even run it all the way to the
classification stage we never ask what
it sees in all this but we could we
could perform the amplification process
on a final result rather than an
intermediate one maybe we ask what would
it take for you to see this banana as a
toaster or say don't these skiers look
like a dog so these are adversarial
examples images that have been tuned to
give classifiers Frank hallucinations
the confident belief that they're seeing
something that just isn't there and
they're not completely wild these robot
delusions I mean that sticker does look
quite a lot like a toaster and it's very
shiny and those skiers do kind of look
like a dog if you squint and I mean
there's the head there's the body and if
you were far away and tired and drunk
you might think that it was a big dog
but she probably wouldn't conclude that
it's a big dog there are current
properties of our visual cortex not to
mention the whole rest of our brains
mean that our sense of the world is
stateful it's a continuously refined
hypothesis whose state is held by the
state of our neurons laying the
groundwork for capsule networks sera
sabor Nicholas frost and Geoffrey Hinton
write a parse tree is carved out of a
fixed multi-layer neural network like a
sculpture is carved from rock our
perceptions are a process of continuous
refinement which may point the way
towards more robust recognition
architectures recurrent convolutional
neural networks that ruminate upon
images making better classifications or
at least providing some kind of signal
that something is off about an input
there are adversarial examples for the
human visual system after all and we
call them optical illusions and they
usually feel pretty weird to look at in
this image we can feel our sensory
impression of
scene flipping between three
alternatives a box a little box in front
of a big one a box in a corner and a box
missing one in this monkey illusion
there is something scintillating about
the color of the dots which are all the
same and are all brown if we design
convolutional neural networks with
recurrence they could exhibit such
behavior as well which maybe doesn't
sound like such a good thing on the face
of it like let's make our image
classifier it's vacillating and
uncertain and then put them in charge of
driving cars around but we drive cars
around and it's our ability to hem and
haw and reconsider our own perceptions
at many levels that gives our perceptual
system such tremendous robustness
paradoxically being able to second-guess
ourselves allows us greater confidence
in our predictions we are doing science
in every moment the cells of our brain
continuously reconsidering and refining
shifting hypotheses about the state of
the world and this gives us the ability
to adapt and operate within a pretty
extreme range of conditions even while
we're tripping face or well asleep
dreams
these are not real people like the
unpeople of Monica showed us yesterday
these are the photos of fake celebrities
dreamt up by a generative adversarial
Network a pair of networks which are
particularly creative the networks get
better through continuous mutual
refinement and it works like this on one
side we have the Creator this is a deep
deep learning network not unlike
inception but trained to run in Reverse
this network we feed with noise
literally just a bunch of random numbers
and it learns to generate images but it
has no way to learn to play this game
and the technical parlance it lacks a
gradient without another opponent the
adversary the adversary is an image
classifier again like Inception
but trained to run on only two classes
real and fake it's job is to distinguish
the creator's forgeries from true faces
we feed this network with the ground
truth with actual examples of celebrity
faces and the adversary learns and then
we use those results to train the
creator so if it makes a satisfying
forgery it's doing well if it's
forgeries are detected we back propagate
the failure so that it can learn I
should tell you that the technical names
for these networks are the generator and
the discriminator I changed the names
because names are important and also
meaningless they don't change this
underlying structure of the training
methodology which is a game these two
neural circuits are playing with each
other and competition is inspiring when
we spar our opponent creates for us the
tactical landscape that we must traverse
just as we do the same for them together
our movements ruminate on a space of
possibility is much larger than any
fixed example set agains can train
remarkably well on relatively small
amounts of data
and it seems likely that this kind of
adversarial process could be helpful for
neural circuits of all kinds it was not
without its quirks games are not
particularly great at global structure
this is fallout cow it is a cow with an
extra body just as you may have spent a
night walking through a house that is
your house but with many extra rooms
these networks are also not super great
at counting
so this monkey has eight eyes because
sometimes science goes too far do
something for me next time you think
you're awake which i think is now count
your fingers just to be sure
I've been awkward if I didn't have five
now if you find that you have more or
fewer than you expected please don't
wake up just yet we're not quite done
another interesting thing about this
training technique is that the generator
is being fed noise a vector of noise
some random point in a very high
dimensional latent space and so it
learns a mapping from this space onto
its generation target in this case faces
if we take a point in that space and we
just kind of drag it around we got this
which is also pretty trippy no I mean
this resembles things I've seen things
someone who isn't me has seen on acid
this resembles the sorts of things that
you may have seen in long-forgotten
dreams now I don't have a Magic School
Bus riot to take us on to understand why
that is but I do have a theory when we
see a face there's a bunch of neurons in
our brain that light up and begin
resonating the signal that is the
feeling of looking at that particular
face taken together all of the neurons
implicated in face detection produce a
vector embedding some mapping from faces
to positions in a high dimensional space
and so as we drag around the generators
vector here
we are also dragging around our own
which is a novel and kind of unsettling
sensation so that's a wild theory but
it's not totally without neurocognitive
precedent here we have a rat in a cage
so we've hooked an electrode up to a
particular neuron in the rat's brain and
these pink dots are the locations where
it's firing if we speed this whole thing
up then we're gonna start to see a
pattern emerge this neuron is a grid
cell so named because the Centers of its
firing fields produce a triangular grid
there's lots of different grid cells in
your brain
each aligning to a different grid and
they collect data from your visual
system from head Direction cells which
encode the position of your head and
together these cells can they construct
an encoding of our position in 2d
euclidean space this operates even in
our sleep if earlier you discovered that
you were dreaming and you want to see
the end of this talk which are having
trouble not waking up oh neuro knots
recommend spinning around this detaches
you're perceived body the one with 12
fingers and three extra bedrooms from
your physical body which is lying in bed
this positioning system is something
which on some level you always knew
existed after all you know where you are
in space you have a sense of space as
you move through it and it's likely even
necessary
if we believe that cognition is
computation that our qualitative sense
of position has a neurocognitive
precursor some signal in the web that
tells us where we're at in many senses
of the word
part three sticks and stones they say
you can't tickle yourself because you
know it's coming
specifically when your brain sends an
action command to your muscles it's
called an efference when an efference is
sent your brain makes a copy now makes a
copy sound so planned so engineered your
brain is this big messy evolved signal
processing mesh so another way to think
of efference copies is as reflections we
take the efference and we send it out to
the peripheral nerves where presumably
it's going to make some muscles contract
meanwhile from the efference copy we
predict how our body's state will change
and we use that to update our brains
model of our body's state now if we
didn't do this then we would have to
wait for sense data to come back to tell
us what happened I mean where is our
hand right now and then we would face
the same problem is trying to play a
twitchy video game over a crap
connection signals take about 10
milliseconds to travel from our brain to
the periphery and about 10 milliseconds
to come back it's just not that low
latency or high bandwidth our nervous
system and so to enable smooth
coordinated movements our brain has to
make predictions so life goes on but in
a moment we have another problem see we
still receive sense data from our nerves
and if we update our models again they
would fall out of sync and so we
attenuate this signal and keep the model
in sync this attenuation applies even to
our sense of touch when that touch is an
expected consequence of our own movement
now expected consequence that's quite a
complicated model and aspects of it are
likely distributed throughout our brain
but there is one place that is
particularly important in maintaining it
the cerebellum the cerebellum is very
special
it contains half the neurons in our
nervous system all action commands
from the brain to the body wrap through
it and all sensations from the body to
the brain as well it has long been
recognized as vitally important to motor
coordination like this people with
cerebellar damage have trouble
performing this action smoothly with
cerebellar damage
it's our movements become jerky and
laggy it's theorized that the cerebellum
acts as a Smith predictor our brains
controller for our latency distant
body's able to estimate the body's
current state integrate sensory feedback
to update that model and decompose
grouse action commands generated
elsewhere in the brain into a fine-tuned
continuously varying control signal and
once you've evolved it the thing like
that has many uses there's a growing
body of evidence implicating the
cerebellum in language which makes sense
uh pterence is a kind of movement and
language she said kind of gesticulating
is not limited to utterance the work of
moving words it's not so different from
the work of moving the body they are
both transformations from the space of
internal states of efference and ideas
to the space of world coordinates and
external sense impressions and back
again and what happens when this
predictor encounters a problem when
there is an irreconcilable discontinuity
in the model
[Music]
these are not extremely different
they're both visceral and guttural they
shake our bones and jokes too are shaped
like trauma
they're both shattering the
Illuminations of discontinuities
paradoxes which cannot be and yet are
things that we must revisit again and
again water smoothing the edges of
cutting stone the machinery of our
brains trying to make sense of a world
that resists it these last few months
have been very difficult for me the
world is heavy my heart is heavy we
build cages for kids to Diane we live in
drowning cities built by slaves
meanwhile I spend my days trying to make
numbers into bigger numbers there are
days when I open my email and every
subject line is a stone and I think I
should put these all into my dress and
walk into the sea but I don't because I
remind myself because I remember that I
am a process of creation I am a song
singing myself we our story is telling
ourselves a see understanding itself our
churning waves creating every moment of
exquisite agony and exquisite joy and
everything else it's you it's all you
you are everything everything you have
ever seen every place you have ever been
every song you have ever sung every god
you have ever prayed to every person you
have ever loved
and this boundaries between you and them
and the sea and the stars are all in
your head
thank you
[Applause]