neural networks are good for learning
lots of different types of patterns
to give an example of how this would
work i imagine you had a four pixel
camera
so not not four megapixels but just four
pixels and it was only black and white
and you wanted to go around and take
pictures of things and
determine automatically then whether
these pictures were of a solid all-white
or all-dark image
a vertical line
or a diagonal line
or a horizontal line
this is tricky because you can't do this
with simple rules about the brightness
of the pixels
both of these are horizontal lines but
if you tried to make a rule about which
pixel was bright and which was dark you
wouldn't be able to do it
so to do this with the neural network
you start by taking all of your inputs
in this case are four pixels and you
break them out into input neurons
and you assign a number to each of these
depending on the brightness or darkness
of the pixel
plus one is all the way white minus one
is all the way black and then gray is
zero right in the middle
[Music]
so these values once you have them
broken out and listed like this on the
input neurons it's also called the input
vector or array it's just a list of
numbers
that represents your inputs right now
it's a useful notion to think about the
receptive field of a neuron all this
means is what set of inputs
makes the value of this neuron as high
as it can possibly be
for input neurons this is pretty easy
each one is associated with just one
pixel
and when
that pixel is all the way white
the value of that input neuron is as
high as it can go
the black and white checkered areas show
pixels that an input neuron doesn't care
about
if they're all the way white or all the
way black it still doesn't affect the
value of that input neuron at all
now to build a neural network
we create
a neuron
the first thing this does is it adds up
all of the values
of the input neurons
so in this case if we add up all of
those values we get a 0.5
now to complicate things just a little
bit
each of the connections
are
weighted meaning they're multiplied by a
number that number can be one
or minus one
or anything in between so for instance
if something has a weight of minus one
it's multiplied and you get the negative
of it and that's added in if something
has a weight of zero then it's
effectively ignored
so here's what those weighted
connections might look like
and you'll notice that after the values
of the input neurons are weighted
and added the values can the final value
is completely different
graphically it's convenient to represent
these weights as
white links being positive weights black
links being negative weights
and the thickness of the line is roughly
proportional to the magnitude of the
weight
then after you add the weighted input
neurons
they get squashed
and i'll show you what that means
you have a sigmoid squashing function
sigmoid just means s-shaped
and what this does is you put a value in
let's say 0.5
and you run a vertical line up to your
sigmoid and then a horizontal horizontal
line over from where it crosses
and then where that hits the y axis
that's the output of your function so in
this case slightly less than 0.5 it's
pretty close
as your input number gets larger
your output number also gets larger but
more slowly and eventually
no matter how big the number you put in
the answer is always
less than one
similarly when you go negative the
answer is always greater than negative
one
so this ensures that that neuron's value
never gets outside of the range of plus
one to minus one
which is helpful for keeping the
computations in the neural network
bounded and stable
so after you sum the weighted values of
the neurons and squash the result you
get the output in this case 0.746
that is a neuron
so we can call this we can collapse all
that down and this is a neuron that does
a weighted sum and squash the result
and now instead of just one of those
assume you have a whole bunch there are
four shown here but
there could be 400 or 4 million
now to keep our picture clear
we'll assume for now that the weights
are either plus one
white lines
minus one black lines or zero in which
case they're missing entirely
but
in actuality all of these neurons that
we created are each
attached to all of the input neurons
and they all have some weight between
minus one and plus one
when we create this first layer of our
neural network
the receptive fields get more complex
for instance here each of those end up
combining two of our input neurons
and so the value the receptive field
the pixel values that make
that first layer neuron
as large as it can possibly be
look now like pairs of pixels
either
all white or a mixture of white and
black
depending on
the weights
so for instance
this neuron here
is attached to this input pixel
which is upper left and this input pixel
which is lower left and both of those
weights are positive
so it combines the two of those and
that's its receptive field the receptive
field of this one plus the receptive
field of this one
however if we look at this neuron
it combines
our
this pixel
upper right
and this pixel lower right
it has a weight of minus one for the
lower right pixel so that means it's
most active
when this pixel is black
so here is its receptive field
now
the because we were careful of how we
created that first layer
its values look a lot like input values
and we can turn right around and create
another layer on top of it the exact
same way with the output of one layer
being the input to the next layer
and we can repeat this
three times or seven times or 700 times
for additional layers
each time the receptive fields get even
more complex
so you can see here using the same logic
now they cover
all of the pixels
and more uh more special arrangement of
which are black and which are white
we can create another layer
again all of these neurons in one layer
are connected to all of the neurons in
the previous layer but we're assuming
here that most of those weights are zero
and not shown
it's not generally the case
so just to mix things up we'll create a
new layer but if you notice our
squashing function isn't there anymore
we have something new called a rectified
linear unit
this is another
popular neuron type
so you do your weighted sum of all your
inputs and instead of squashing
you do rectified linear units you
rectify it so if it is negative
you make the value zero if it's positive
you keep the value
this is obviously very easy to compute
and it turns out to have very nice
stability properties for neural networks
as well in practice
so after we do this because some of our
weights are positive and some are
negative connecting to those rectified
linear units we get
receptive fields and they're opposites
if you look at the patterns there
and then finally when we've created as
many layers with as many neurons as we
want we create an output layer
here we have four outputs that we're
interested in
is the image solid
vertical diagonal or horizontal
so to walk through an example here of
how this would work let's say we start
with this input image
shown on the left
dark pixels on top white on the bottom
as we propagate that to our input layer
this is what those values would look
like
the top pixels the bottom pixels
as we move that to our first layer
we can see the combination of a dark
pixel
and a light pixel some together
get us zero
gray
whereas down here we have the
combination of a dark pixel
plus a light pixel with a negative
weight
so that gets us a value of negative one
there
which makes sense because if we look at
the receptive field here
upper left pixel white
lower left pixel black
is the exact opposite of the input that
we're getting
and so we would expect its value to be
as low as possible
minus one
as we move to the next layer we see the
same types of things combining zeros to
get zeros
um combining a negative
and a negative with a negative weight
which makes a positive
to get a zero
and here we have combining two negatives
to get a negative
so again you'll notice the receptive
field of this is exactly the inverse of
our input so it makes sense that its
weight would be negative
or its value would be negative
and we move to the next layer
all of these of course these zeros
propagate forward
here
this is a negative has a negative value
and it gets has a positive weight so it
just moves straight forward because we
have a rectified linear unit
negative values become zero
so now it is zero again two but this one
gets rectified and becomes positive
negative times a negative is positive
and so when we finally get to the output
we can see they're all zero except for
this horizontal which is positive and
that's the answer our neural network
said this is
an image of a horizontal line
now
neural networks usually aren't that good
not that clean
so there's a notion of
with an input what is truth
in this case the truth is this has a
zero for all of these values but a one
for
horizontal it's not solid it's not
vertical it's not diagonal
yes it is horizontal
an arbitrary neural network will give
answers that are not exactly truth it
might be off by a little or a lot
and then the error
is the magnitude of the difference
between the truth and the answer given
and you can add all these up to get the
total error for the neural network
so the idea
the whole idea with learning and
training
is to adjust the weights to make the
error as low as possible
so the way this is done is
we put an image in we calculate the
error at the end
then we look for how to adjust those
weights higher or lower to either make
that error go up
or down and we of course adjust the
weights in the way that make the error
go down
now the problem with doing this is each
time we go back and calculate the error
we have to multiply all of those weights
by all of the neuron values at each
layer
and we have to do that again and again
once for each weight
this takes forever
in computing terms
on computing scale and so it's not a
practical way to train a big neural
network
you can imagine instead of just rolling
down to the bottom of a simple valley we
have a very high dimensional valley and
we have to find our way down and because
there are so many dimensions one for
each of these weights that the
computation just becomes prohibitively
expensive
luckily there was an insight
that
lets us do this in a very reasonable
time
and that's that
if we're careful about how we design our
neural network we can calculate the
slope directly the gradient we can
figure out the direction that we need to
adjust the weight
without going all the way back through
our neural network and recalculating
so just to review the slope that we're
talking about is when we make a change
in weight the error will change a little
bit
and that
relation of the change in weight to the
change in error
is the slope
mathematically there are several ways to
write this
we'll favor the one on the bottom it's
technically most correct
we'll call it d-e-d-w for shorthand
every time you see it just think
the change in error
when i change a weight
or the change in the thing on the top
when i change the thing on the bottom
um this is uh does get into a little bit
of calculus we do take derivatives
that's how we calculate slope if it's
new to you
i strongly recommend a good semester of
calculus just because the concepts are
so
universal
and uh a lot of them have very nice
physical interpretations which i find
very appealing
but don't worry otherwise just gloss
over this and pay attention to the rest
and you'll get a general sense for how
this works
so in this case
if we change the weight by plus one the
error changes by minus two which gives
us a slope of minus two
that tells us the direction that we
should adjust our weight and how much we
should adjust it to bring the error down
now to do this you have to know what
your error function is
so assume we had an error function that
was the square of the weight
and you can see that our weight
is right at -1
so the first thing we do is we take the
derivative
change in error divided by change in
weight d e d w
the derivative of weight squared is two
times the weight
and so we plug in our weight of minus
one and we get a slope d e d w of minus
two
now the other trick that lets us do this
with deep neural networks is chaining
and to show you how this works imagine a
very simple trivial neural network with
just one hidden layer one input layer
one output layer and one weight
connecting each of them
so it's obvious to see that the value y
is just the value x times the weight
connecting them w1
so if we change w1 a little bit we just
take the derivative of y with respect to
w1
and we get x the slope is x if i change
w1 by a little bit then y will change by
x times the size of that adjustment
similarly for the next step we can see
that e
is just the value y
times the weight w2
and so when we calculate d e d y it's
just w
2. because this network is so simple we
can calculate from one end to the other
x times w1 times w2
is the error e
and so if we want to calculate how much
will the error change if i change w1 we
just take the derivative of that with
respect to w1
and get x times w2
so this illustrates you can see here now
that what we just calculated is actually
the product
of
our first derivative that we took the
the dydw1
times
the derivative for the next step d e d y
multiplied together
this is chaining
you can calculate the slope of each tiny
step
and then multiply all of those together
to get the slope of the full chain the
derivative of the full chain
so in a deeper neural network what this
would look like is if i want to know how
much the error will change
if i adjust a weight that's deep in the
network i just calculate the derivative
of each tiny little step all the way
back to the weight that i'm trying to
calculate
and then multiply them all together
this computationally is many many times
cheaper
than what we had to do before of
recalculating the error for the whole
neural network for every weight
now in the neural network that we've
created there are several
types of back propagation we have to do
there are several operations we have to
do for each one of those
we have to be able to calculate the
slope
so for the first one is just a weighted
connection between two neurons a and b
so let's assume we know the change in
error with respect to b
we want to know the change in error with
respect to a
to get there we need to know db da
so to get that we just write the
relationship between b and a
take the derivative of b with respect to
a we get the weight w
and now we know how to make that step we
know how to do that
little nugget of back propagation
another element that we've seen
is sums
all of our neurons sum up a lot of
inputs
to take this back propagation step
we do the same thing we write our
expression
and then we take the derivative
of our endpoint z with respect to our
step that we're uh propagating to a
and dz da in this case is just one
which makes sense if we have a sum of a
whole bunch of elements
we increase one of those elements by one
we expect the sum to increase by one
that's
the definition of a slope of one
one to one relation there
another element that we have that we
need to be able to back propagate is the
sigmoid function
so this one's a little bit more
interesting mathematically
we'll just write it
shorthand like this the sigma function
um it is entirely feasible to uh go
through and take the derivative of this
analytically
and calculate it
it just so happens that this function
has a nice property
that to get its derivative you just
multiply it by
1 minus itself
so this is very straightforward to
calculate
another element that we've used is the
rectified linear unit
again to figure out how to back
propagate this we just write out the
relation
b is equal to a if a is positive
otherwise it's zero
and
piecewise for each of those we take the
derivative so db da is either one if a
is positive or zero
and so with all of these
little back propagation steps
and the ability to chain them together
we can calculate the effect of adjusting
any given weight
on the error
for any given input
and so to train
then
we start with a fully connected network
we don't know what any of these weights
should be
and so we assign them all random values
we create a completely
arbitrary random neural network
we put in an input that we know the
answer to we know whether it's solid
vertical diagonal or horizontal so we
know what truth should be
and so we can calculate the error
then
we run it through calculate the error
and using back propagation go through
and adjust all of those weights a tiny
bit in the right direction
and then we do that again with another
input and again with another input for
if we can get away with it
many thousands or even millions of times
and eventually all of those weights will
gravitate they'll roll down that many
dimensional valley to a nice low spot
in the bottom
where it performs really well and does
pretty close to truth on most of the
images
if we're really lucky it'll look like
what we started with with intuitively um
understandable
receptive fields for those neurons and a
relatively sparse
representation meaning that most of the
weights are
small or close to zero and it doesn't
always turn out that way but
what we are guaranteed is that it'll
find a pretty good representation of you
know the best that it can do adjusting
those weights
to get as close as possible to the right
answer for all of the inputs
so what we've covered is just a very
basic introduction to the principles
behind neural networks i haven't told
you quite enough to be able to go out
and build one of your own
but if you're feeling motivated to do so
i highly encourage it
here are a few resources that you'll
find useful
you'll want to go and learn about bias
neurons
dropout is a useful training tool
there are several resources available
from andre carpathi who is an expert in
neural networks and great at teaching
about it also there's a fantastic
article called the black magic of deep
learning that just has a bunch of
practical from the trenches tips
on how to get them working well
neural networks are famously difficult
to interpret
it's hard to know what they're actually
learning when we train them
so let's take a closer look and see
whether we can get a good picture of
what's going on inside
just like every other supervised machine
learning model
neural networks learn relationships
between input variables and output
variables
in fact we can even see how it's related
to the most iconic model of all
linear regression
simple linear regression assumes a
straight-line relationship between an
input variable x
and an output variable y
x is multiplied by a constant
m
which also happens to be the slope of
the line
and it's added to another constant b
which happens to be where the line
crosses the y axis
we can represent this in a picture
our input value x
is multiplied by m our constant
b is multiplied by one
and then they get added together to get
y
this is a graphical representation of y
equals m x
plus b
on the far left the circular symbols
just indicate that the value is passed
through
the rectangles l labeled m and b
indicate that whatever goes in on the
left comes out multiplied by m or b on
the right
and the box with the capital sigma
indicates that whatever goes in on the
left gets added together and spit out on
the right
we can change the names of all the
symbols for a different representation
this is still a straight line
relationship we've just changed the
names of all the variables
the reason we're doing this is to
translate our linear regression into the
notation we'll use in neural networks
this will help us keep track of things
as we move forward
at this point we have turned a straight
line equation into a network
a network is anything that has nodes
connected by edges
in this case
x sub 0 and x sub 1
are our input nodes
v sub 0 is an output node
and our weights connecting them are
edges
this is not the traditional sense of a
graph meaning a plot or a grid like in a
graphing calculator or graph paper
it's just the formal word for a network
for nodes connected by edges
another piece of terminology you might
hear is a directed acyclic graph
abbreviated as d-a-g or dag a directed
graph is one where the edges just go in
one direction
in our case
input goes to output but output never
goes back to input
our edges are directed
acyclic
means that you can't ever draw a loop
once you have visited a node there's no
way to jump from edges to nodes to edges
to nodes to get back to where you
started
everything flows in one direction
through the graph
we can get a sense of the type of models
that this network is capable of learning
by choosing random values for the
weights
w sub 0 0 and w sub 1
and then seeing what relationship pops
out between
x sub 1
and v sub 0.
remember that we set x of 0 equal to 1
and are holding it there always
this is a special node called a bias
node
it should come as no surprise that the
relationships that come out of this
linear model are all straight lines
after all we've taken our equation for
the line and rearranged it but we
haven't changed it in any substantial
way
there's no reason we have to limit
ourselves to just one input variable
we can add an additional one
now here we have an x of 0
an x sub 1 and an x sub 2.
we draw an edge between x sub 2 and our
summation
with the weight
w
sub 2 0
x sub 2
times w sub 2 0
is again
u sub 2 0
and all of our u's get added together to
make a v sub 0.
and we could add more inputs as many as
we want
this is still a linear equation but
instead of being two dimensional we can
make it three dimensional or higher
writing this out mathematically could
get very tedious so we'll use a shortcut
we'll substitute the subscript i
for the index of the input it's the
number of the input we're talking about
this allows us to write u sub i zero
where our
u sub i
equals x sub i
times w sub i zero
and again our output v sub zero is just
the summation over all values of i
of u sub i zero
for this three dimensional case we can
again look at the models that emerge
when we randomly choose our w sub i
zeros our weights
as we would expect
we still get the three dimensional
equivalent of a line a plane in this
case
and if we were to extend this to more
inputs we would get the m dimensional
equivalent of a line which is called an
m-dimensional hyperplane
so far so good
now we can start to get fancier
our input
x sub 1 looks a lot like our output v
sub zero
in fact there's nothing to prevent us
from taking our output
and then using it as an input to another
network just like this one
now we have two separate identical
layers
we can add a subscript roman numeral i
and a subscript roman numeral i i or two
to our equations depending on which
layer we're referring to
and we just have to remember that our x
sub 1
in layer 2
is the same as our v sub 0 in layer 1.
because these equations are identical
and each our layer each of our layers
work just the same we can reduce this to
one set of equations adding a subscript
capital l to represent which layer we're
talking about
as we continue here we'll be assuming
that all the layers are identical and to
keep the equations cleaner we'll leave
out the capital l
but just keep in mind that if we were
going to be completely correct and
verbose we would add the l
subscript onto the end of everything to
specify the layer it belongs to
now that we have two layers
there's no reason that we can't connect
them in more than one place
instead of our first layer generating
just one output we can make several
outputs
in our diagram we'll add a second output
v sub 1
and we'll connect this
to a third input into our second layer
x sub 2.
keep in mind that the x sub 0 input to
every layer will always be equal to 1.
that bias node shows up again in every
layer
now there are two nodes
shared by both layers
we can modify our equations accordingly
to specify which of the shared nodes we
are talking about
they behave exactly the same so we can
be efficient and reuse our equation
but we can specify subscript j
to indicate which output we're talking
about
so now if i'm connecting the ith input
to the jth output
then
i and j will determine which weight
is applied
and which u's get added together to
create the output v sub j
and we can do this as many times as we
want
we can add as many of these shared nodes
as we care to
the model as a whole only knows about
the input x sub 1 into the first layer
and the output
v sub 0 of the last layer
from the point of view of someone
sitting outside the model the shared
nodes between layer 1 and layer 2 are
hidden
they are inside the black box
because of this they're called hidden
nodes
we can take this two layer linear
network
create a hundred hidden nodes
set all of the weights randomly
and see what model it produces
even after adding all of this structure
the resulting models are still straight
lines
in fact it doesn't matter how many
layers you have or how many hidden nodes
each layer has
any combination of these linear elements
with weights and sums will always
produce a straight line result
this is actually one of the traits of
linear computation that makes it so easy
to work with
but unfortunately for us it also makes
really boring models
sometimes a straight line is good enough
but that's not why we go to neural
networks we're going to want something a
little more sophisticated
in order to get more flexible models
we're going to need to add some
non-linearity
we'll modify our linear equation here
after we calculate our output v sub 0
we subject it to another function
f
which is not linear
and we'll call the result
y sub zero
one really common non-linear function to
add here is the logistic function it's
shaped like an s so sometimes it's
called a sigmoid function too
although that can be confusing because
technically any function shaped like an
s is a sigmoid
we can get a sense of what logistic
functions look like by choosing random
weights for this one input
one output one layer network
and meeting the family
one notable characteristic of logistic
functions is that they live between zero
and one
for this reason they're also called
squashing functions
you can imagine taking a straight line
and then squashing the edges and bending
and hammering it down so that the whole
thing fits between zero and one no
matter how far out you go
working with logistic functions brings
us to another connection with machine
learning models logistic regression
this is a bit confusing
because regression refers to finding a
relationship between an input and an
output
usually in the form of a line or a curve
or a surface of some type
logistic regression is actually used as
a classifier most of the time
it finds a relationship between a
continuous input variable and a
categorical output variable
it treats observations of one category
as zeros
treats observations of the other
category as ones
and then finds the logistic function
that best fits all those observations
then to interpret the model we add a
threshold often around 0.5 and wherever
the curve crosses the threshold there's
a demarcation line
everything to the left of that line is
predicted to fall into one category and
everything to the right of that line is
predicted to fall into the other
this is how a regression algorithm gets
modified to become a classification
algorithm
as with linear functions there's no
reason not to add more inputs
we know that logistic regression can
work with many input variables and we
can represent that in our graph as well
here we just add one
in order to keep the plot three
dimensional but we could add as many as
we want
to see what type of functions this
network can create we can choose a bunch
of random values for the weights
as you might have expected the functions
we create are still s-shaped
but now they're three-dimensional
they look like a tablecloth laid across
two tables of unequal height
more importantly if you look at the
contour lines projected down onto the
floor of the plot
you can see that they are all perfectly
straight the result of this is that any
threshold we choose for doing
classification
will split our input space up into two
halves
with the divider being a straight line
this is why logistic regression is is
described as a linear classifier
whatever the number of inputs you have
whatever dimensional space you're
working in
logistic regression will always split it
into two halves using a line or a plane
or a hyperplane of the appropriate
dimensions
another popular non-linear function is
the hyperbolic tangent
it's closely related to the logistic
function and can be written in a very
symmetric way
we can see when we choose some random
weights and look at examples
that hyperbolic tangent curves look just
like logistic curves
except that they vary between -1 and
plus 1.
just like we tried to do before with
linear functions we can use the output
of one layer as the input to another
layer
we can stack them in this way and can
even add hidden nodes the same way we
did before
here we just show two hidden nodes in
order to keep the diagram simple but you
can imagine as many as you want there
when we choose random weights for this
network and look at the output we find
that things get interesting
we've left the realm of the linear
because the hyperbolic tangent function
is non-linear
when we add them together we get
something that doesn't necessarily look
like a hyperbolic tangent
we get curves
wiggles
peaks and valleys and a much wider
variety of behavior than we ever saw
with single layer networks
we can take the next step and add
another layer to our network
now
we have a set of hidden nodes between
layer 1 and layer 2 and another set of
hidden nodes between layer 2 and layer
3.
again we choose random values for all
the weights and look at the types of
curves it can produce
again we see wiggles and peaks
valleys and a wide selection of shapes
if it's hard to tell the difference
between these curves and the curves
generated by a two-layer network
that's because they're mathematically
identical
we won't try to prove it here but
there's a cool result
that shows that any curve you can create
learning a menu using a many layered
network you can also create using a two
layer network as long as you have enough
hidden nodes
the advantage of having a many layered
network is that it can help you create
more complex curves
using fewer total nodes
for instance in our two layer network we
used a hundred hidden nodes
in our three layer network we used 11
hidden nodes in the first layer and nine
hidden nodes in the second layer
that's only a fifth of the total number
we used in our two layer network but the
curves it produces show similar richness
we can use these fancy wiggly lines to
make a classifier as we did with
logistic regression
here we use the zero line as the cutoff
everywhere that our curve crosses the
zero line there's a divider
in every region that the curve sits
above the zero line we'll call this
category a
and similarly everywhere the curve is
below the zero line we have category b
what distinguishes these nonlinear
classifiers from linear ones is that
they don't just split the space into two
halves
in this example regions of a and b are
interleaved
building a classifier around a
multi-layer non-linear network gives it
a lot more flexibility
it can learn more complex relations
this particular combination of
multi-layer network
with hyperbolic tangent non-linear
function has its own name a multi-layer
perceptron
as you can guess when you have only one
layer it's just called a perceptron and
in that case you don't even need to add
the nonlinear function to make it work
the function will still cross the x-axis
at all the same places
here is the full network diagram of a
multi-layer perceptron
this representation is helpful because
it makes every single operation explicit
however it's also visually cluttered
it's difficult to work with
because of this it's most often
simplified to look like circles
connected by lines
this implies all the operations we saw
on the previous diagram
connecting lines each have a weight
associated with them
hidden nodes and output nodes perform
summation and nonlinear squashing
but in this diagram all of that is
implied
in fact our bias nodes
the nodes that always have a value of
one in each layer
are omitted for clarity
so our original network reduces to this
the bias nodes are still present and
their operation hasn't changed at all
but we leave them out to make a cleaner
picture
we only show two hidden nodes from each
layer here but in practice we used quite
a few more
again to make the diagram as clean as
possible we often don't show all the
hidden nodes we just show a few and the
rest are implied
here's a generic diagram then
for a three layer
single input single output network
notice that if we specify the number of
inputs the number of outputs and the
number of layers
and the number of hidden nodes in each
layer
then we can fully define a neural
network
we can also take a look at a two input
single output neural network
because it has two inputs
when we plot its outputs it will be a
three dimensional curve
we can once again choose random weights
and generate curves to see what types of
functions this neural network might be
able to represent
this is where it gets really fun
with multiple inputs multiple layers
and non-linear activation functions
neural networks can make really crazy
shapes
it's almost correct to say that they
could make any shape you want
it's worth taking a moment though to
notice what its limitations are
first notice that all of the functions
fall between plus and minus one
the dark red and the dark green regions
kiss the floor and the ceiling of this
range but they never cross it this
neural network would not be able to fit
a function that extended outside of this
range
also notice that these functions all
tend to be smooth
they have hills and dips and valleys and
wiggles and even points and wells
but it all happens relatively smoothly
if we hope to fit a function with a lot
of jagged jumps and drops this neural
network might not be able to do a very
good job of it
however aside from these two limitations
the variety of functions that this
neural network can produce is a little
mind-boggling
we modified a single output neural
network to be a classifier when we
looked at the multi-layer perceptron
now there's another way to do this
we can use a two output neural network
instead
outputs of a three layer
one input
to output neural network like this
we can see that there are many cases
where the two curves cross and in some
instances they cross in several places
we can use this to make a classifier
wherever the one output is greater than
another
it can signify that one category
dominates another
graphically
wherever the two output functions cross
we can draw a vertical line
this chops up the input space into
regions
in each region one output is greater
than the other
for instance wherever the blue line is
greater we can assign that to be
category a
then wherever the peach colored line is
greater those regions are category b
just like the multi-layer perceptron
this lets us chop the space up in more
complex ways than a linear classifier
could
regions of category a and category b can
be shuffled together arbitrarily
when you only have two outputs the
advantages of doing it this way over a
multi-layer perceptron with just one
output are not at all clear
however
if you move to three or more outputs the
story changes
now we have three separate outputs and
three separate output
functions we can use our same criterion
of letting the function with the maximum
value determine the category
we start by chopping up the input space
according to which function has the
highest value
each function represents one of our
categories
we're going to assign our first function
to be category a and label every region
where it's on top
as category a
then we can do the same with our second
function
and our third
using this trick we are no longer
limited to two categories
we can create as many output nodes as we
want
and learn and chop up the input space
into that many categories
it's worth pointing out that the winning
category may not be the best by very
much
in some cases you can see they can be
very close
one category will be declared the winner
but the next runner up may be almost as
good a fit
there's no reason that we can't extend
this approach to two or more inputs
unfortunately it does get harder to
visualize
you have to imagine several of these
lumpy landscape plots on top of each
other
and in some regions one will be greater
than the others
in that region that category associated
with that output will be dominant
to get a qualitative sense for what
these regions might look like you can
look at the projected contours on the
floor of these plots
in the case of a multi-layer perceptron
these plots are all sliced at the y
equals zero level
that means if you look at the floor of
the plot
everything in any shade of green will be
one category and everything in any shade
of red will be the other category
the first thing that jumps out about
these category boundaries is how diverse
they are
some of them are nearly straight lines
albeit with a small wiggle
some of them have wilder bends and
curves and some of them chop the input
space up into several disconnected
regions of green and red
sometimes there's a small island of
green or island of red in the middle of
a sea of the other color
the variety of boundaries is what makes
this such a powerful classification tool
the one limitation we can see looking at
it this way is that the boundaries are
all smoothly curved
sometimes those curves are quite sharp
but usually they're gentle and rounded
this shows the natural preference that
neural networks with hyperbolic tangent
activation functions
have for smooth functions and smooth
boundaries
the goal of this exploration was to get
an intuitive sense for what types of
functions and category boundaries neural
networks can learn when used for
regression or classification
we've seen both their power and their
distinct preference for smoothness
we've only looked at two nonlinear
activation functions
logistic and hyperbolic tangent both of
which are very closely related
there are lots of others and some of
them do a bit better at capturing sharp
non-linearities
rectified linear units or relu's for
instance
produce surfaces and boundaries that are
quite a bit sharper
but my hope was to seed your intuition
with some examples of what's actually
going on under the hood
when you train your neural network
here are the most important things to
walk away with
neural networks learn functions and can
be used for regression
some activation functions limit the
output range but as long as that matches
the expected range of your outputs it's
not a problem
second neural networks are most often
used for classification
they've proven pretty good at it
third
neural networks tend to create smooth
functions when used for regression
and smooth category boundaries when used
for classification
fourth
for fully connected vanilla neural
networks
a two-layer network can learn any
function that a deep network can learn
however a deep network might be able to
learn it with fewer nodes
fifth making sure that inputs are
normalized that is they have a mean near
zero and a standard deviation of less
than one
this helps neural networks to be more
sensitive to their relationships
i hope this helps you as you jump into
your next project
happy building
welcome to how convolutional neural
networks work
convolutional neural networks or
convnets or cnns
can do some pretty cool things
if you feed them a bunch of pictures of
faces for instance they'll learn some
basic things like edges and dots bright
spots dark spots
and then because they're a multi-layer
neural network
that's what gets learned in the first
layer the second layer are things that
are recognizable as eyes noses mouths
and the third layer are things that look
like faces
similarly if you feed it a bunch of
images of cars down to the lowest layer
you'll get things again that look like
edges
and then higher up look at things that
look like tires and wheel wells and
hoods
and at a level above that things that
are clearly identifiable as cars
cnn's can even learn to play video games
by forming patterns of the pixels as
they appear on the screen
and learning what is the best action to
take when it sees a certain pattern
a cnn can learn to play video games in
some cases far better than a human ever
could
not only that if you take a couple of
cnns and have them set to watching
youtube videos
one can learn objects
by again picking out patterns and the
other one can learn types of grasps
this then coupled with some other
execution software
can let a robot learn to cook
just by watching youtube
so there's no doubt cnns are powerful
usually when we talk about them we do so
in the same way we might talk about
magic
but they're not magic what they do is
based on some pretty basic ideas
applied in a clever way
so to illustrate these we'll talk about
a very simple toy convolutional neural
network
what this one does
is takes in an image a two-dimensional
array of pixels you can think of it as a
checkerboard and each square on the
checkerboard is either light or dark
and then by looking at that the cnn
decides whether it's a picture of an x
or of an o
so for instance
on top there we see an image with
an x drawn in white pixels on a black
background and we would like to identify
this as an x
and the o
we'd like to identify as an o
so how a cnn does this
is uh has several steps in it
what makes it tricky
is that the x is not exactly the same
every time
the x or the o can be shifted it can be
bigger or smaller can be rotated a
little bit
thicker or thinner
and in every case we would still like to
identify whether it's an x or an o
now the reason that this is challenging
is because for us
deciding whether these two things are
similar is straightforward we don't even
have to think about it
for a computer it's very hard
what a computer sees
is this checkerboard this
two-dimensional array
as a bunch of numbers
ones and minus ones a one is a bright
pixel a minus one is a black pixel
and what it can do is go through pixel
by pixel and compare whether they match
or not
so to computer to a computer
it looks like there are a lot of pixels
that match but some that don't quite a
few that don't actually
and so it might look at this and say ah
i'm really not sure whether these are
the same
and so it would because the computer is
so literal i would say
uncertain i can't say that they're equal
now one of the tricks that convolutional
neural networks use is to match parts of
the image
rather than the whole thing
so if you break it down into its smaller
parts or features
then it becomes much more clear
whether these two things are similar
so examples of these little features are
little
mini images in this case just three
pixels by three pixels
the one on the left is a diagonal line
slanting downward from left to right the
one on the right is also a diagonal line
slanting in the other direction
and the one in the middle is a little x
these are little pieces of the bigger
image
and you can see as we go through if you
choose the right feature and put it in
the right place it matches the image
exactly
so okay we have the bits and pieces
now to take a step deeper
there the math behind matching these
is called filtering
and the way this is done is a feature
is lined up with the little patch of the
image
and then one by one the pixels are
compared
they're multiplied by each other
and then add it up and divide it by the
total number of pixels
so to step through this to see why it
makes sense to do this you can see
starting in the upper left-hand pixel in
both the feature and the image patch
multiplying one by a one gives you a one
and we can keep track of that by putting
that in the position of the pixel that
we're comparing
we step to the next one minus one times
minus one is also a one
and we continue to step through
pixel by pixel
multiplying them all by each other and
because they're always the same the
answer is always one
when we're done we take all these ones
and add them up and divide by nine
and the answer is one
so now we want to keep track of where
that feature was in the image and we put
a one there say when we put the feature
here we get a match of one
that is filtering
now we can take uh that same feature
and move it to another position and
perform the filtering again and we start
with the same pattern the first pixel
matches the second pixel matches the
third pixel does not match
minus one times one equals minus one so
we record that
in our results
and we go through and do that through
the rest of the image patch
and when we're done we notice we have
two minus ones this time
so we add up all the pixels to add up to
five divide by nine
and we get a point five five so this is
very different than our one
and we can record the 0.55 in that
position where we were where it occurred
so by moving our filter around to
different places in the image
we actually find different values for
how well that filter matches or how well
that feature is represented at that
position
so this becomes a map
of where the feature occurs
by moving it around to every possible
position
we do convolution
that's just the repeated application of
this feature this filter over and over
again
and what we get is a nice
map across the whole image of where this
feature occurs
and if we look at it it makes sense this
feature is a diagonal line slanting
downward left to right
which matches
the downward left to right diagonal of
the x so if we look at our filtered
image we see that all of the high
numbers ones and .77s are all right
along that diagonal that suggests that
that feature
matches along that diagonal much better
than it does elsewhere in the image
to use a shorthand notation here we'll
do a little x with a circle in it to
represent convolution
the act of trying every possible match
and we repeat that with other features
we can repeat that with our x filter in
the middle and with our upward slanting
diagonal line on the bottom
and in each case the map that we get of
where that feature occurs
is consistent with what we would expect
based on what we know about the x and
about where our features match
this
act of convolving an image
with a bunch of filters a bunch of
features
and creating a stack of filtered images
is we'll call a convolution layer
a layer because it's an operation that
we can stack with others as we'll show
in a minute
in convolution one image becomes a stack
of filtered images we get as many
filtered images out as we have filters
so convolution layer is one trick that
we have
the next big trick that we have is
called pooling
this is how we shrink the image stack
and this is pretty straightforward
we start with a window size usually two
by two pixels or three by three pixels
and a stride usually two pixels just in
practice these work best
and then we take that window and walk it
in strides across each of the filtered
images
from each window we take the maximum
value
so to illustrate this we start with our
first filtered image
we have our 2 pixel by 2 pixel window
within that pixel the maximum value is
1.
so we track that and then move to
our stride of 2 pixels we move 2 pixels
to the right and repeat
out of that window the maximum value is
0.33
etc 0.55
and when we get to the end we have to be
creative we have
don't have all the pixels representative
so we take the max of what's there
and we continue doing this across the
whole image and when we're done
what we end up with is a similar pattern
but smaller
we can still see our high values are all
on the diagonal
but instead of seven by seven pixels in
our filtered image we have a four by
four pixel image so it's half as big as
it was about
this makes a lot of sense to do
if you can imagine if instead of
starting with a nine by nine pixel image
we had started with a nine thousand by
nine thousand pixel image shrinking it
is convenient for uh working with it
makes it smaller
the other thing it does
is pooling doesn't care where in that
window that maximum value occurs
so that makes it a little less sensitive
to position
and the way this plays out is that if
you're looking for a
particular feature in an image it can be
a little to the left a little to the
right maybe a little rotated and it'll
still get picked up
so we do max pooling with all of our
stack of filtered images
and get in every case
smaller set of filtered images
now that's our second trick third trick
normalization
this is just a step to keep the math
from blowing up
and keep it from going to zero
all you do here
is everywhere in your image
that there is a negative value
change it to zero
so for instance if we're looking back at
our filtered image we have these what
are called rectified linear units that's
the little computational unit that does
this but all it does is steps through
everywhere there's a negative value
change it to zero
another negative value change it to zero
by the time you're done you have a very
similar looking image
except there's no negative values
they're just zeros
and we do this with all of our images
and this becomes another
type of layer
so in a rectified linear unit layer a
stack of images becomes a stack of
images with no negative values
now what's really fun the magic starts
to happen here
when we take these layers convolution
layers rectified linear unit layers and
pooling layers and we stack them up so
that the output of one becomes the input
of the next
you'll notice that what goes into each
of these and what comes out of these
looks like an array of pixels or
an array of an array of pixels
and
because of that we can stack them nicely
we can use the output of one for the
input of the next and by stacking them
we get these operations building on top
of each other
what's more we can repeat the stacks we
can do deep stacking you can imagine
making a sandwich that is not just
one patty and one slice of cheese and
one lettuce and one tomato but a whole
bunch of
layers double tripper triple quadruple
deckers as many times as you want
each time
the image gets
more filtered as it goes through
convolution layers
and it gets smaller as it goes through
pooling layers
now the final layer
in our toolbox is called a fully
connected layer
here every value gets a vote
on what the answer is going to be
so we take our now much filtered and
much reduced in size stack of images we
break them out we just rearrange and put
them into a single list because it's
easier to visualize that way
and then each of those connects to one
of our answers that we're going to vote
for
when we feed this in x
there will be certain values here that
tend to be high they tend to predict
very strongly this is going to be an x
they get a lot of vote
for the x outcome
similarly
when we feed in a picture of an o
to our convolutional neural network
there are certain values here at the end
that tend to be very high and tend to
predict strongly when we're going to
have an o at the end
so they get a lot of weight a strong
vote
for the o category
now when we get a new input and we don't
know what it is and we want to decide
the way this works
is the input goes through all of our
convolutional
our
rectified linear unit our pooling layers
and comes out to the end here we get a
series of votes
and then based on the weights
that each value gets to vote with
we get a nice average vote at the end
in this case this this particular set of
inputs votes for an x with a strength of
0.92
and an o with a strength of 0.51
so here definitely x is the winner
and so
the neural network would categorize this
input as an x
so in a fully connected layer
a list of feature values
becomes a list of votes
now
again what's cool here is that a list of
votes
looks a whole lot like a list of feature
values so you can use the output of one
for the input of the next
and so you can have intermediate
categories that aren't your final votes
or sometimes these are called hidden
units in a neural network and you can
stack as many of these together as you
want also
but in the end they all end up voting
for an x or an o and whoever gets the
most votes wins
so if we put this all together
then
a two-dimensional array of pixels in
results in a set of votes for a category
out at the far end
so there are some things that we have
glossed over here
you might be asking yourself where all
of the magic numbers come from
things that i pulled out of thin air
include the features in the
convolutional layers those convenient
three pixel by three pixel diagonal
lines of the x
also
the voting weights
in the fully connected layers i really
waved my hands about how those are
obtained
in all these cases the answer is the
same there is a trick called back
propagation
all of these are learned you don't have
to know them you don't have to guess
them
the deep neural network does this on its
own
so the underlying principle behind back
propagation is that the error in the
final answer is used to determine how
much the network adjusts
and changes
so in this case
if we knew we were putting in an x
and we got a 0.92 vote for an x
and that would be an error of 0.08
and we got a 0.51 vote for o
we know that that would be an error of
0.49
actually an error of 0.51 because it
should be 0. then if we add all that up
we get an error of what should be 0.59
so
what happens with this error signal
is it helps drive a process called
gradient descent
if there is another bit of something
that
is pretty special sauce to deep neural
networks it is the ability to do
gradient descent
so for each of these magic numbers each
of the feature pixels each voting weight
they're adjusted up and down by a very
small amount
to see how the error changes
the amount that they're adjusted
is determined by how big the error is
large error they're adjusted a lot
smaller just a tiny bit no error they're
not adjusted at all you have the right
answer stop messing with it
as they're adjusted you can think of
that as
sliding a ball slightly to the left and
slightly to the right on a hill you want
to find the direction where it goes
downhill you want to go down that slope
down that gradient to find the very
bottom because the bottom is where you
have the very least error that's your
happy place
so after sliding it to the left and to
the right you find the downhill
direction and you leave it there
doing that many times over lots of lots
of iterations lots of steps helps all of
these values across all the features
and all of the weights
settle in to what's called a minimum
and it
uh and it at that point the network is
performing as well as it possibly can if
it adjusts any of those a little bit its
behavior its error will go up
now there are some things called hyper
parameters and these are knobs that the
designer gets to turn decisions the
designer gets to make these are not
learned automatically
in convolution
figuring out how many features should be
used
how big those features should be how
many pixels on the side
in the pooling layers choosing the
window size and the window stride
and in fully connected layers choosing
the number of hidden neurons
intermediate neurons
all of these things are decisions that
the designer gets to make
right now there are some
common practices that tend to work
better than others but there is no
principled way there's no hard and fast
rules for the right way to do this
and in fact a lot of the advances in
convolutional neural networks are in
getting combinations of these they work
really well
now in addition to this
there are other decisions the designer
gets to make like how many of each type
of layer
and in what order
and for those that really like to go off
the rails can we design new types of
layers entirely and slip them in there
and get new fun behaviors
these are all things that people are
playing with to try to eke out more
performance and address stickier
problems with cnns
now what's really cool about these we've
been talking about images
but you can use any two-dimensional or
even for that matter three or four
dimensional data
but what's important is that in your
data
things closer together are more closely
related than things far away
what i mean by that is if you look at an
image
two rows of pixels or two columns of
pixels are right next to each other
they're more closely related than rows
or columns that are far away
now what you can do
is you can take something like sound
and you can chop it up into little time
steps
and for each piece of time
the
time step right before it and right
after is more closely related than time
steps that are far away and the order
matters
you can also chop it up into different
frequency bands
base mid-range treble you can slice it a
whole lot more finally than that
and again those frequency bands
are the ones closer together are more
closely related
and
you can't rearrange them the order
matters
once you do this with sound it looks
like a picture it looks like an image
and you can use convert convolutional
neural networks with them
you can do something similar with text
where
the position in the sentence becomes the
column
and the row
is words in a dictionary
in this case
it's hard to argue whether order matters
that order matters it's hard to argue
that words in the dictionary are that
some are more closely related than
others in all cases and so the trick
here is to take a window that spans the
entire column top to bottom
and then slide it left to right that way
it captures all of the words but it only
captures a few positions in the sentence
at a time
now the other side of this limitation of
convolutional neural networks is that
they're really designed to capture
local spatial patterns spatial in the
sense of things that are next together
next to each other matter quite a bit
so if the data can't be made to look
like an image
then they're not as useful
so an example of this is say some
customer data
if i have each row it's a separate
customer
each column is a separate piece of
information about that customer
such as their name their address what
they bought what websites they visited
then this doesn't so much look like a
picture i can take and rearrange those
columns and rearrange those rows
and this
still means the same thing it's still
equally easy to interpret
if i were to take an image
and rearrange the columns and rearrange
the rows it would result in a scramble
of pixels and it would be difficult or
impossible to say what the image was of
there i would lose a lot of information
so as a rule of thumb
if your data is just as useful after
swapping out any of the columns for each
other then you can't use
convolutional neural networks
so the take home is that convolutional
neural networks are great at finding
patterns and using them to classify
images
if you can make your problem look like
finding cats on the internet
then they're a huge asset
applications of machine learning have
gotten a lot of traction in the last few
years
there's a couple of big categories that
have had winds one
is
identifying pictures
the equivalent of finding cats on the
internet and any problem that can be
made to look like that
and the other is sequence to sequence
translation this can be speech to text
or one language to another
most of the former are done with
convolutional neural networks most of
the latter are done with recurrent
neural networks uh particularly long
short-term memory to give an example of
how long short-term memory works we will
consider the question of what's for
dinner
let's say for a minute that you are a
very lucky apartment dweller and you
have a flatmate who loves to cook dinner
every night he cooks one of three things
sushi
waffles
or pizza
and you would like to be able to predict
what you're going to have on a given
night so you can plan the rest of your
days eating accordingly
in order to predict what you're going to
have for dinner
you set up a neural network
the inputs to this neural network are
a bunch of items like the day of the
week the month of the year whether or
not your flatmate was in a late meeting
variables that might reasonably affect
what you're going to have for dinner
now if you're new to neural networks i
highly recommend you take a minute and
stop to watch the how neural networks
work tutorial there's a link down in the
comments section
if you'd rather not do that right now
and you're still not familiar with
neural networks you can think of them as
a voting process
and so in the neural network that you
set up there's a complicated voting
process and all of the inputs like day
of the week and month of the year go
into it
and then you train it
on your history of what you've had for
dinner
and you learn
how to predict what's going to be for
dinner tonight
the trouble is that your network doesn't
work very well
despite carefully choosing your inputs
and training it thoroughly
you still can't get much better than
chance predictions on dinner
as is often the case with complicated
machine learning problems it's useful to
take a step back
and just look at the data
and when you do that you notice a
pattern
your flat mate makes
pizza
then sushi then waffles then pizza again
in a cycle
it doesn't depend on the day of the week
or anything else it's in a regular cycle
so knowing this we can make a new neural
network
in our new one the only inputs that
matter are what we had for dinner
yesterday
so if we know if we had pizza for dinner
yesterday it'll be sushi tonight sushi
yesterday waffles tonight and waffles
yesterday pizza tonight
it becomes a very simple voting process
and and it's right all the time because
your flatmate is incredibly consistent
now if you happen to be gone on a given
night let's say yesterday you were out
you don't know what was for dinner
yesterday
you can still predict what's going to be
for dinner tonight by thinking back two
days ago
think what was for dinner then
so what would be predicted
for you last night
and then you can use that prediction in
turn to make a prediction for tonight
so we make use of not only
our actual information from yesterday
but also
what our prediction was yesterday
so at this point it's helpful to take a
little detour and talk about vectors
a vector is just a fancy word for a list
of numbers if i wanted to describe the
weather to you for a given day i could
say
the high is 76 degrees fahrenheit the
low is 43 the wind's 13 miles an hour
there's going to be a quarter inch of
rain and the relative humidity is 83
percent
that's all a vector is the reason that
it's useful is vectors list of numbers
are computers native language
if you want to get something into a
format that it's natural for a computer
to compute to do operations on to do
statistical machine learning lists of
numbers are the way to go
everything gets reduced to a list of
numbers before it goes through an
algorithm we can also have a vector for
statements like it's tuesday
in order to encode this kind of
information what we do is we make a list
of all the possible values it could have
in this case all the days of the week
and we assign a number to each
and then we go through and set them all
equal to zero except for the one that is
true right now
uh this format is called one hot
encoding and it's very common to see a
long vector of zeros with just one
element being one
it seems inefficient but for a computer
this is a lot easier way to ingest that
information
so we can make a one hot vector for our
prediction for dinner tonight we set
everything equal to zero except for the
dinner item that we predict so in this
case we'll be predicting sushi
now we can group together
our
uh we can group together our inputs and
outputs into vectors separate lists of
numbers
and it becomes a useful shorthand for
describing this neural network
so we can have our dinner yesterday
vector
our predictions for yesterday vector and
our prediction for today vector
and the neural network is just
connections between every element in
each of those input vectors to every
element in the output vector
and to complete our picture we can show
how the prediction for today will get
recycled the dotted line there means
hold on to it for a day and then reuse
it tomorrow
and it becomes our yesterday's
predictions tomorrow now we can see how
if we were lacking some information
let's say we were out of town for two
weeks we can still make a good guess
about what's going to be for dinner
tonight
we just
ignore the new information part and we
can unwrap or unwind this vector in time
until we do have some information to
base it on and then just play it forward
and when it's unwrapped it looks like
this
and we can go back as far as we need to
and see what was for dinner and then
just trace it forward and play out our
menu over the last two weeks until we
find out what's for dinner tonight
so this is a nice simple example that
showed recurrent neural networks now to
show how they don't meet all of our
needs
we're going to write a children's book
it'll have sentences of the format
doug saw jane period
jane saw spot period spot saw doug
period and so on
so our dictionary is small just the
words doug
jane
spot
saw and a period
and the task of the neural network is to
put these together in the right order to
make a good children's book
so to do this we replace our
food vectors with our dictionary vectors
here again it's just a list of numbers
representing each of the words so for
instance
if doug was the most recent word that i
saw
my new information vector would be all
zeros except for a one in the dug
position
and we similarly can represent our
predictions and our predictions from
yesterday
now
after training
this neural network and teaching it what
to do we would expect to see certain
patterns
for instance anytime a name comes up
jane dug or spot
we would expect that to vote heavily for
the word saw or for a period
because those are the two words in our
dictionary that can follow a name
similarly if we had predicted a name on
the previous time step we would expect
those to vote also for the word saw or
for a period
and then by a similar method anytime we
come across the word saw or a period
we know that a name has to come after
that so it will learn to vote very
strongly for a name
jane doug or spot
so in this form in this formulation we
have a recurrent neural network
for simplicity i'll take the vectors and
the weights and collapse them down to
that little symbol with the dots and the
arrows the dots and the lines connecting
them
and there's one more symbol we haven't
talked about yet
this is a squashing function and it just
helps the network to behave
how it works is you take all of your
votes
coming out and you subject them to this
squashing function for instance if
something received a total vote of 0.5
you draw a vertical line up where it
crosses the function you draw a
horizontal line over to the y axis and
there is your squashed version out
for small numbers the squashed version
is pretty close to the original version
but as your number gets larger
the number that comes out is closer and
closer to one
and similarly if you put in a big
negative number then what you'll get out
will be very close to minus one
no matter what you put in
what comes out is between minus one and
one
so this is really helpful when you have
a loop like this where the same values
get processed again and again day after
day
it is possible you can imagine if in the
course of that processing say something
got voted for twice it got multiplied by
two
in that case it would get twice as big
every time and very soon blow up to be
astronomical
by ensuring that it's always less than
one but more than minus one
you can multiply it
as many times as you want you can go
through that loop and it won't explode
in a feedback loop this is an example of
negative feedback or attenuating
feedback
so you may have noticed our neural
network in its current state is subject
to some mistakes
we could get a sentence for instance of
the form doug
saw
doug
period
because
doug strongly votes for the word saw
which in turn strongly votes for a name
any name which could be doug
similarly we could get something like
doug saw jane
saw spot saw
because each of our predictions only
looks back one time step it has very
short-term memory
then it doesn't use the information from
further back
and it's subject to these types of
mistakes
in order to overcome this we take our
recurrent neural network and we expand
it and we add some more pieces to it
the critical part that we add to the
middle here
is memory
we want to be able to remember what
happened many times steps ago
so in order to explain how this works
i'll have to describe a few new symbols
we've introduced here
one is another squashing function this
one with a flat bottom
one is an x in a circle and one is a
cross in a circle
so the cross in a circle
is element by element addition
the way it works is you start with two
vectors of equal size
and you go down each one
you add the first element of one vector
to the first element of another vector
and then the total goes into the first
element of the output vector
so 3
plus 6 equals 9. then you go to the next
element 4 plus 7 equals 11.
and so your output vector is the same
size of each of your input vectors
just a list of numbers same length but
it's the sum element by element of the
two
and very closely related to this you've
probably guessed the
x in the circle is element by element
multiplication
it's just like addition except instead
of adding you multiply for instance 3
times 6 gives you a first element of 18.
4 times 7 gets you 28.
again the output vector is the same size
of each of the input vectors
now element-wise multiplication lets you
do something pretty cool
um
you imagine that you have
a signal and it's like a bunch of pipes
and they have a certain amount of water
trying to flow down them
in this case we'll just assign the
number to that of 0.8 it's like a signal
now
on each of those pipes we have a faucet
and we can open it all the way close it
all the way or keep it somewhere in the
middle to either let that signal come
through or block it
so in this case an open gate an open
faucet would be a one and a closed
faucet would be a zero
and the way this works with element wise
multiplication we get 0.8 times one
equals 0.8
that signal passed right through into
the output vector but the last element
0.8 times 0
equals 0.
that signal the original signal was
effectively blocked
and then with the gating value of 0.5
the signal was passed through but it's
smaller it's attenuated
so
gating lets us control what passes
through and what gets blocked
which is really useful
now in order to do gating
it's nice to have a value that you know
is always between zero and one
so we introduce another squashing
function
this will represent with a circle with a
flat bottom
and this is it's called the logistic
function it's very similar to the other
squashing function the hyperbolic
tangent
except that it just goes between zero
and one instead of minus one and one
now when we introduce all of these
together what we get
we still have the combination of our
previous predictions and our new
information
those vectors get passed
and we make predictions based on them
those predictions get passed through but
the other thing that happens is a copy
of those predictions
is held onto
for the next time step the next pass
through the network
and some of them here's a gate right
here
some of them are forgotten some of them
are remembered
the ones that are remembered are added
back into the prediction
so now we have not just prediction but
predictions plus the memories that we've
accumulated
and that we haven't chosen to forget yet
now there's an entirely separate neural
network here that learns
when to forget what
based on what we're seeing right now
what do we want to remember what do we
want to forget
so you can see this is powerful this
will let us hold on to things for as
long as we want
now you've probably noticed though
when we are
combining our predictions with our
memories we may not necessarily want to
release all of those memories out as new
predictions each time
so we want a little filter to keep our
memories inside and let our predictions
get out
and that's we add another gate for that
to do selection
it has its own neural network so its own
voting process so that our new
information and our previous predictions
can be used
to vote on what all the gates should be
what should be kept internal and what
should be released as a prediction
we've also introduced another squashing
function here since we do an addition
here it's possible that things could
become greater than one or smaller than
minus one so we just squash it to be
careful to make sure it never gets out
of control
and now
when we bring in new predictions we
make a lot of possibilities
and then we collect those with memory
over time and of all of those possible
predictions
at each time step
we select just a few to release as the
prediction for that moment
each of these things
when to forget
and when to let things out of our memory
are learned by their own neural networks
and the only other piece we need to add
to complete our picture here is yet
another set of gates
this lets us actually ignore
uh possible predictions possibilities as
they come in
this is an attention mechanism
it lets things that aren't immediately
relevant
be set aside so they don't cloud the
predictions in memory going forward
it has its own neural network and its
own
logistic squashing function and its own
gating activity right here
now long short term memory has a lot of
pieces a lot of bits that work together
and it's a little much to wrap your head
around it all at once so what we'll do
is take a very simple example and step
through it just to illustrate how a
couple of these pieces work
it's admittedly an overly simplistic
example
and feel free to poke holes at it later
when you get to that point then you know
you're ready to move on to the next
level of material
so
we are now in the process of writing our
children's book
and for the purposes of demonstration
we'll assume that this lstm has been
trained
on our children's books examples that we
want to mimic
and all of the appropriate votes and
weights in those neural networks have
been learned
now we'll show it in
action so so far our story so far is
jane saw spot period
doug
so doug is the most recent word that's
occurred in our story
and also not surprisingly for this time
step
the names doug jane and spot were all
predicted as viable options
this makes sense we just wrapped up a
sentence with a period the new sentence
can start with any name so these are all
great predictions so we have our
new information
which is the word doug
we have our recent prediction which is
doug jane and spot
and we pass these two vectors together
to all four of our neural networks which
are learning to make predictions
to do it ignoring to do forgetting and
to do selection
so
the first one of these
makes some predictions
given that the word doug just occurred
this is learned that the word saw
is a great
guess to make for a next word
but it's also learned that having seen
the word doug
that it should not see the word doug
again very soon seeing the word doug at
the beginning of a sentence so it makes
a positive prediction for saul and a
negative prediction for doug it says i
do not expect to see doug in the near
future
so that's why doug is in black
so this example is so simple we don't
need to focus on attention or ignoring
so we'll skip over it for now
and this prediction of saw
not doug is passed forward
and again for the purposes of simplicity
let's say there's no memory at the
moment so
saw and doug get passed forward
and then the selection mechanism here
has learned
that when the most recent word was a
name
then what comes next is either going to
be
the word saw or a period so it blocks
any other names from coming out so the
fact that there's a vote for not doug
gets blocked here and the word saw
gets sent out
as the prediction for the next time step
so we take a step forward in time now
the word saw is our most recent word and
our most recent prediction they get
passed forward to all of these neural
networks and we get a new set of
predictions
because the word saw just occurred
we now predict that the words doug jane
or spot might come next
we'll pass over ignoring and attention
in this example and we'll take those
predictions forward
now the other thing that happened
is our previous set of
possibilities
the word saw and not doug that we were
maintaining internally
get passed to a forgetting gate
now the forgetting gate
says hey my last word
that came uh that occurred was the word
saw
based on my past experience then i for
can forget about you know i know that it
occurred i can forget that it happened
but i want to keep
any predictions having to do with names
so it forgets saw
holds on to the vote for not doug
and now at this
element by element addition we have a
positive vote for doug a negative o for
doug and so they cancel each other out
so now we just have votes for jane and
spot
those get passed forward
our selection gate
it knows that the word saw just occurred
and based on experience a name will
happen next and so it passes through
these predictions for names and for the
next time step then we get predictions
of only jane and spot
not doug
this avoids the doug saw doug period
type of error and the other errors that
we saw
what this shows
is that long short-term memory can look
back
two three many time steps
and use that information to make good
predictions about what's going to happen
next
now to be fair to vanilla recurrent
neural networks they can actually look
back several time steps as well but not
very many
lstm can look back many time steps and
has shown that successfully
this is really useful in
some surprisingly practical applications
if i have text in one language and i
want to translate it to text to another
language lstms work very well
even
though translation is not a word to word
process
it's a phrase to phrase or even in some
cases a sentence to sentence process
lstms are able to represent those
grammar structures that are specific to
each language and
what it looks like is that they
find the higher level idea and translate
it from one mode of expression to
another just using the bits and pieces
that we just walked through
another thing that they do well is
translating speech to text
speech is just
some signals that vary in time
it takes them and uses that then to
predict
what text what word is being spoken and
it can use
the history the recent history of words
to make a better guess for what's going
to come next
lstms are a great fit
for
any information that's embedded in time
audio
video
my favorite application of all of course
is robotics
robotics is nothing more than
uh an agent taking in information from a
set of sensors
and then based on that information
making a decision
and carrying out an action
it's inherently sequential and actions
taken now can influence what is sensed
and what should be done many times steps
down the line
if you're curious what lstms look like
in math
this is it this is lifted straight from
the wikipedia page
i won't step through it but it's
encouraging that something that looks so
complex
expressed mathematically
actually makes a fairly straightforward
picture and story
and if you'd like to dig into it more i
encourage you to go to the wikipedia
page
also there are a collection of really
good
tutorials and discussions
other ways of explaining lstms that you
may find helpful as well
i'd also strongly encourage you to visit
andre carpathi's blog post showing
examples of what lstms can do in text
you can be forgiven if when reading on
the internet you can substitute magic
for deep learning and it fits perfectly
in all of the articles
it's hard to know what it can't do we
don't get to talk about that very much
so the goal of this talk is just to talk
about a really simple nuts and bolts
level
the summary case you want to take a nap
deep learning is not magic
but it's really good at finding patterns
so
if this
is our brain
this is deep learning
an owl can fly
uh a fighter jet can fly
but there are a whole lot of things that
an owl can do
and arguably it's a much more complex
thing
although what the fighter jet does it
does really really well
so deep learning is the fighter jet
highly
specialized highly
engineered
we're today going to talk about the
basics
the wright brothers airplane
if you understand the principles that it
works on then you can see easy to branch
out into the finer engineering details
but there's a whole lot of things that
go on into fighter jet that we're not
going to talk about in detail
but this is nice we can talk about this
at a comfortable level
this is a neuron
like all neurons it's got a big body in
the middle a long tail and some arms
that branch off
here's an artist's conception of a
neural network or a bunch of neurons
again
big body long tails
arms
this is an actual picture of neurons in
some brain tissue
here the bodies look like dots or blobs
you can see long tails some of which
branch and the arms are pretty much
invisible
and again a picture of some brain tissue
here the neurons are small dots and you
can barely see any of the tails at all
this is just to give you a sense of how
tightly these things are packed together
and how many of them there are
big numbers with lots of zeros and the
crazy part is that a lot of them are
connected to many more of their
neighbors this
is one of our very first pictures of a
neuron
santiago ramona kahal
found a stain that he could introduce
into a cell turn the whole thing dark
under his
19th century
light microscope was able to see this
and then draw it with a pen and paper
this is old school
what you see here though
bodies
long tails
lots of arms
um these have we're going to turn it
upside down because that's how they're
typically represented in neural networks
and these these pieces actually have
names the bodies are called the soma
the long tails are called axons
and the arms are called dendrites
we're going to draw a cartoon version of
them
this is what they look like in uh
in powerpoint
now the way that neurons work
is that a dendrite you can think of it
as feelers or whiskers and they look for
electrical activity they pick it up and
send it to the body
the soma takes this and adds it together
and accumulates it and then depending on
how fast it's accumulating it it will
activate the axon and send that signal
along
down the tail
the more dendrite activity there is
the more
axonal activity there is
and if you get all of the dendrites
really firing then that axon is just as
active as it can possibly be
in a very simplistic way
a neuron adds things up
now a synapse is where the axon from one
neuron touches the dendrite of another
that's an artist's conception of it
you can see in ramona tahoe's drawings
these little
nubs or buttons
they're actually called boutons on
the dendrites
and these are sites
where the axon of another neuron touches
that
so you can imagine there's a little
connection there
will represent that connection by a
circle and the diameter of that circle
is the strength of that connection big
circle strong connection
and it can connect strongly or weakly or
somewhere in between
we can put a number on this connection
between zero and one
so a medium connection
we'll uh call it a 0.6
when the axon of the input neuron
the upstream neuron
is active
then it
activates the dendrite of the output
neuron
it passes that on with a modest strength
if that connection is strong
then it passes the signal on very
strongly that connection is a one then
when the axon is active the next
dendrite is very active
likewise if that connection is very weak
say a 0.2
then when the axon is active the
dendrite of the output neuron is only
weakly activated
no connection at all is a zero
now this starts to get interesting
because many different input neurons can
connect to the dendrites of a single
output neuron
and each connection has its own strength
we can redraw this by taking out all of
the parallel dendrites and just drawing
each axon and the single dendrite that
it connects to and the connection
strength represent like this with the
dots
we can substitute in numbers for that
the weights
uh although most often
uh oh we can also separate substitute in
line thicknesses to show how strongly
these things are connected most of the
time neural networks are drawn like this
and this is what we have we went from
the
super complex slice of brain tissue
with many subtleties in its operation
and interconnection
to
a nice circle stick diagram
where each one of those sticks
represents a weight
in its current form
it can still do some pretty cool things
the input neurons input neurons can
connect to many output neurons
so actually what you get here is many
inputs many outputs
and the connection between each is
distinct and has its own weight
this is uh it's good for making pretty
pictures it's also great for
representing combinations of things
and the way this is done
let's say you have five inputs
labeled a b c e
in this case this output neuron has a
strong connection to a
c and e
very weak connections to b and d
that means when the a c and e input
neurons are active
all together that really strongly
activates the output neuron
b and d don't matter because they only
connect weakly
so a way to think about this output
neuron is in terms of the inputs that's
strongly activated so we call this the
ace neuron
and here we have
an atomic example of
what happens here this output neuron
represents a combination of the input
neurons
this is neural networks in a nutshell
you can do this with any kind of input
so you have a really low tech 4 pixel
camera
each of those four inputs is uh one of
the pixels upper left lower left
upper right or lower right
in this particular neural network with
strong connections to the upper left and
upper right pixel
we have a neuron and output neuron that
represents this bar in the top half of
the image
so we can combine letters we can combine
pixels to make
small images
if you're doing text processing the
input neurons can represent individual
words
so in this case
we're pulling words out of text
this output neuron is strongly connected
to the anterior neurons for eye and ball
so we can call it the eyeball neuron
similarly we can have a sunglasses
neuron
and infant neurons can connect to many
outputs we could have an eyeglasses
neuron just as easily
so going a little deeper into this this
is a somewhat trivial example to show
how these things work in practice
so there's a guy at the shawarma place
and makes shawarma like nobody else and
so you want to make sure and go when
he's working there
and
taking a step back we actually have some
domain knowledge here we know he's got
two schedules
working in the morning
off in the evening and off in the
morning working in the evening
now if we were to instrument this with
sensors
we would have the working in the morning
off in the morning
working in the evening
off in the evening
and it might be useful to represent
his working patterns in terms of a
couple of output neurons that combine
those
so this is the network that we would
expect to end up with
working in the morning off in the
evening is one pattern
off in the morning working in the
evening is the other path and you can
see based on their connection strengths
how they combine those inputs
here would be the weights associated
with those
now the question is how do we learn this
if we have to go in and fill it all in
by hand we haven't learned anything it's
just a fancier way of programming
and a lot of
hard work especially if you're dealing
with many millions of input neurons
so we want to learn this automatically
so to start with might be a little
counter-intuitive we create our neural
network
we have our input neurons
all we choose is the number of output
neurons in this case we'll choose two
because we happen to know we're learning
two patterns
and then we randomly assign weights
randomly number generate numbers for
each of these it's a neural network this
completely you roll the dice you throw
the sticks and whatever falls out that's
what you start with
and then we start to gather data we go
stand on the other side of the street
and we observe that the shawarma guy on
this particular day worked in the
morning
and then went home did not work in the
evening
that means
this input
is active we'll say it's at a level one
often the morning is at a level zero
because we didn't observe it
working in the morning is at zero
and
off in the evening is out of one because
we observed that too
so the next step is for each of the
output neurons we calculate the activity
so in this case
an appropriately simple way to do this
is just take the average of the inputs
so here
this weight is 0.3 and this weight is
0.1
so the average of those is 0.2
these neurons don't contribute anything
because those inputs aren't active
similarly we can take the weight between
this input and that output 0.8
and 0.4
the average of those is 0.6
the upper neuron on the right has a
higher activity that's the one we care
about we ignore all the others there's a
million others we ignore the rest and
focus on this one for this time step
first thing we figure out is how wrong
is it
well if it was perfect if our neural
network was perfect that would have an
activity of one it would be uh perfectly
aligned with our inputs but it only has
an activity of 0.6 so the error is 0.4
the bigger that error is that's a signal
for how much we need to adjust our
weights when that error gets very small
it means that the weights really
represent what's going on and we don't
need to make any more changes
now the trick here
gradient descent if there is an element
of magic
in deep learning it's gradient descent
what you do
is you go through and adjust each of
these weights
all through
you adjust it a little bit up and a
little bit down and see which way
decreases the error
the idea the concept in gradient descent
is
weight is a quantity that you can shift
a little bit side to side
as you do this error will change you can
think of it as taking this ball
if you shift it a little bit to the left
it has to climb the hill if you shift it
a little to the right it has to fall
down the hill and you like the direction
you choose the direction in which it
gets lower you want to bring that error
down as low as it can get
and you take small incremental steps to
keep everything numerically stable
so we go through and we do these for all
of the neurons uh sorry for all of the
weights that attach input neurons to our
output
and we find that yes we want to increase
this one
because these aren't active we actually
have a bias toward low weights so it
doesn't hurt to decreasing so we'll go
ahead and decrease that weight and
decrease that weight and increase that
one
when we do that sure enough our new
activity is seven and so our error went
from a point four to a point three it's
a little bit better at representing what
we saw
so that was one data point
we go back and we do the same thing the
next day
it just so happens that this day
he's off in the morning working in the
evening
we adjust the weights and we do that day
after day after day
and eventually the weights will stop
changing
or they'll slow down changing quite a
bit they'll get stable
and we get
the system of weights that we originally
saw because we had knowledge of the
problem
so this is the
wright brothers airplane version
of how
training by back propagation using
gradient descent works back propagation
is a very specific way to do this that
is computationally cheap and slick and
fast and you get your your jet engine
instead of the flapping of wings so
this is the underlying mechanism by
which it works
so what we just looked at was a single
layer we have inputs we have outputs
every output is a combination of things
that were on the previous layer
now there's no reason
then that we can't turn around and take
that output layer and make it inputs for
the next layer
and do that again
and again
if a network has more than three layers
or so we call it deep
uh some have more than 12.
in some recent research in microsoft
their deep neural networks would lay
more than a thousand dollars there's no
theoretical reason to limit the number
of layers that you have
it just depends on the specifics of your
problem
now
what does deep get you why is deep
special
if your input neurons are say letters in
the alphabet
your first layer outputs
sorry this is a deep neural network with
all of the connections emitted for
clarity
so these are your inputs this is your
first layer of outputs
they're combinations
of those letters
each level you go up you get
combinations of what happened the level
before so by the time you get to your
second level of outputs you're getting
perhaps words
in the english language if that's what
you're training on
the layer above that you get
combinations of words
short phrases
and so forth and you can do this as deep
as you like
so there's a variety of things you can
learn with deep neural networks a very
popular one is images
if you take as your inputs pixels
and show instead of looking at uh
shawarma guy schedule you're looking at
individual pictures as your training
data set
what you start to learn after a while is
these little representations of short
lines and dots and patches
these are the primitives of an image
if you train on image of faces
then your first
layer output sorry your second layer
outputs start to look like eyes and
noses and mouths and chins and your
third layer output start to look clearly
recognizable as faces
similarly if you train on automobiles
your second layer outputs start to look
like wheels and doors and windows and
your third layer output
look like automobiles
so that's pretty cool we didn't have to
go in and twiddle any of those weights
this just learned that from seeing a
bunch of pictures
you can do it on color images too here's
an uh output of an eight some of the
output neurons of an eight layer neural
network and as you get deeper you can
see things that are clearly recognizable
and quite complex you get spiders and
rocking chairs and sailing and chips and
teddy bears
you can also plug information in about
music artists
so here's some research where
output neurons were learned based on
information about artists and then their
uh
representation was plotted based on how
similar they were in that in the those
output neurons
and so we see things like
kelly clarkson
and beyonce are similar over here which
is also not too far from taylor swift
and avril lavigne
whereas if we go up here we get weezer
flat keys modest mouse presidents of the
united states of america all in the same
neighborhood
this is a network that didn't know
anything still doesn't know anything
about music
but
because of the data that it gets on the
input neurons it's able to group these
things appropriately it finds patterns
and then finds things that most closely
fits those patterns
turns out you can take atari 2600 games
take the pixel representations feed
those in as input neurons learn some fun
features and then pair it with something
else called reinforcement learning that
learns appropriate actions
and when you do this for a certain class
of games it can learn to play them far
better than any human player ever has or
is ever likely to
and it turns out that you can take a
robot and let it watch youtube videos
about how to cook
and it loses uses a pair of deep neural
networks one to interpret the video one
to learn to understand its own movements
and then uses those pairs that with some
other execution software to cook
based on the video representations that
it's seen
so
while it's not magic it's pretty cool
you can do some stuff
so as you're going through reading
literature reading popular articles
about this you can kind of play bingo
there are some buzzwords some popular
algorithms you can think of a lot of
these as the you know model numbers for
the various fighter jets that are out
there
but when you see any of these terms if
you like you can mentally do the
substitution of
deep learning and apply what you know
about the wright brothers airplane and
most of it will still be accurate
so the bottom line it's good at learning
patterns it doesn't do anything but it's
pretty good at learning patterns
i'm excited to get to talk about two of
my favorite topics at once
machine intelligence and robots
they go together pretty well but they're
not the same thing you can definitely
have one without the other
first some disclaimers i'm not going to
give you the answer to human level
intelligence
i would if i had it but i don't
next these are my own personal opinions
they're definitely not those of any
current or former employer
and they don't reflect those of many
experts in the field
take them with a huge grain of salt
if they are useful you're welcome to
them
and if they're not please discard them
also
this story that i'm going to tell you
is not rigorous
it doesn't have any equations it's
conceptual
and i just intended to start a
discussion and foster
ideas throughout this presentation we'll
be talking about intelligence
the working and definition that i
propose is that it's a combination of
how much one can do and how well one can
do it
notionally you could be extremely good
at only one thing and not be that
intelligent
also you could do many things but do
them all very poorly and also not be
that intelligent
intelligence is the combination of being
able to do many things and to do them
well
this is a functional definition of
intelligence
there are many other potential
definitions some of them can be measured
experimentally and some can't
particular definition has the advantage
that if we wanted to reduce it down to a
measurable set of tasks we could
because it is at least in theory
observable this allows us to have a
scientific discussion about machine
level intelligence
it lets us form hypotheses that we could
potentially falsify
and it lets us compare the relative
intelligence of two separate agents
in short it's practical and useful
undoubtedly some will find it
philosophically unsatisfying
that's a separate conversation but one
that i would be happy to have in another
forum
using fake math we can say that
intelligence equals performance times
generality
it's only fake because we haven't
defined performance or generality yet
but assuming we do you can imagine
plotting them
and putting them on a set of axes like
this
although
we haven't defined it quantitatively for
human performance
what i would like to propose
is that
human performance is the level at which
a human expert can do something
this can be measured in lots of
different ways
it can be
error rates
or
the time required to execute a task
the time required to learn a task
the number of demonstrations required
before one learns a task
the amount of energy
expended when one performs a task
the subjective judgment of performance
from a panel of human judges
the speed that someone does something
there are many aspects of performance
and i'm not going to try and specify or
quantify them all here i only list them
to illustrate that we are considering
performance in a broad sense
we're considering performance in a broad
sense rather than in a narrow machine
learning accuracy leaderboard sense
if we consider human level performance
to be something of a baseline we can
place it on our x-axis and then chop up
the rest of the axes in equal increments
we'll make this a logarithmic scale to
enable us to compare a very wide range
of performances
equal steps along this axis represent
equal multiplicative factors of change
human level generality is the set of all
the tasks that humans can do
and have undertaken
these include things like
writing stories cooking pies
building cities
gathering and transmitting information
all around the world
and even exploring the origin of the
universe
it's a very broad set of activities
we can represent human level generality
on the y-axis
roughly this is the set of all tasks
that a human or group of humans can do
we'll make the y-axis logarithmic as
well
so an equal interval is a factor of 10
in performance either
multiplied or divide depending on
whether you're moving up or down
so human intelligence can be notionally
represented by the area
that's entailed
by this point
just want to point out there's no reason
to believe that machines might not
exceed human performance in some areas
humans have a number of limitations that
are built into the way in which we've
achieved our intelligence through
evolution
things that may have been very useful at
one point or may be useful broadly but
now may not be useful in pushing the
limits of intelligence
things like
limited attention
instinctive drives
how every part of us fatigues
and a host of cognitive biases all of
which put some distance between us
and perfectly rational or perfectly
efficient or optimal behavior
machines by comparison have a more
nurturing environment in which to take
root
they don't have to evolve we're trying
everything we can to encourage them
now you can imagine
on the performance generality axis
another agent
that
can do a much larger set of tasks than
humans although do them all
more poorly than a human could so it
might look like this
the area under that triangle the overall
intelligence would still be comparable
to that of a human
so we would call that human level
intelligence also
you can also imagine an agent who can do
a subset of the tasks that humans do but
do them so very much better that the
area under that rectangle is also about
the same as the humans
so again human level intelligence
now if we take the set of all of these
agents that have about the same area
under their intelligence rectangle
then we get this curve representing
human level intelligence
any agent who falls along that curve
would be comparable to a human any agent
down and to the left of it subhuman
intelligence and any agent up and to the
right
superhuman intelligence
now let's look at a few agents that you
might be familiar with and see where
they fall in this scheme
chest pain
chest plane computers have been around
for going on 30 years now um which is a
world champion chess playing computers
have been around for almost 30 years
ibm's deep blue beat gary kasparov in
1989
this was a task that people assumed hugh
computers would have a very tough time
with
it involved planning
strategy thinking mental models of your
opponent
it seemed to take in the very peak of
human cognition
and it seemed unlikely that such a thing
could be done
by a fancy calculator but
it did
and now a chess program running on your
phone can do about the same as deep blue
did
the current state of the art is a
program called stockfish which has an
elo rating which is like a chess skill
score
of 34 47.
compare this to the top human
uh top-rated human player of all time
magnus carlsen
who achieved a 2882
the program and the human are not even
comparable they're not even close
it's worth noting that stockfish is an
open source project with code freely
available and has a number of
contributors scattered around the globe
now in terms of generality
stockfish
understands the rules of chess and in
fact it understands them very well
it has a bunch of hints and tips and
tricks built in by humans that are
specific to chess
it uses a point system to evaluate
pieces
based on where they are and what the
stage of the game is
it uses a full table of end games once
there are just a few pieces left on the
board
the number of possibilities for how the
game will play out are few enough that
they can be completely enumerated
so there's no search there's no solving
there's just essentially looking up in a
giant table and figuring out what to do
next
there are hand-tuned strategies for each
phase of the game
and then what the computer does is
uses tree search to evaluate future
moves
each choice of move is a branch on this
tree and it can look and say for each
move what's the likely outcome
for each of those outcomes what are the
possible moves my opponent can make for
each of those outcomes then what are my
possible responses
and by fully going down this branching
tree and looking at all the
possibilities
the program can then figure out
what its best choices now one of the
things that makes stockfish so clever
and good at its games it's very good at
pruning this tree and ignoring moves
that are unlikely to lead to any good
end and not exploring them very far
but all told
this
program is pretty useless at anything
that is not chess
it is the opposite of general
so on our plot
we would put chess well above human
level performance
but also very low on the generality axis
now compare that to uh
go
also a board game
if you're not familiar with it they're
19 by 19 grid and opponents take turns
placing white and black stones
the rules are actually in some ways
simpler than chess at each turn you pick
a junction on which to place a stone
now the strategy however uh some argue
is even
more complex and more importantly more
subtle
it's what is undoubtedly true is there
are more possible board configurations
where chess has an eight by eight board
go has a 19 by 19 board
where each piece in chess has a small
prescribed number of motions
any stone and go can be placed on any
open junction
so when doing a tree search
the number of moves
explodes
much more quickly than in chess
now despite this two years ago already
alphago
a program
built by researchers at deepmind
beat
lee sidol a professional nine dan player
professional nine dan would put him in
the all-stars of the go world
a later version alphago master was
clocked with an elo rating of 48.58
compare this to the highest rated human
player park jung-won who's rated at
66-69
so
not only has go beaten the best players
of the world but now already by a very
healthy margin
now as we mentioned
the program understands the rules of go
or knows the rules of go
and use the tree search to evaluate
moves
because there are so many possible board
configurations however it can't memorize
them all so it uses convolutional neural
networks to learn common configurations
and most importantly it can
see patterns that are not exactly
repeated but are similar to things that
have seen before
this is an important innovation
and we'll come back to convolutional
neural networks in a few minutes
it also uses reinforcement learning on a
library of human games to learn which
moves are good
reinforcement learning
in
very straightforward terms is
looking at a configuration
an action
and the outcome
and learning the pattern
for a given configuration if i take
action a good things happen if i take
action b bad things tend to happen
after i learn those patterns the next
time i see configuration a
i can take the action that leads to good
things
um so using reinforcement learning on a
library of human games then kind of
bootstrapped alphago and let it learn
from the history of human play and human
insights to get started
but like stockfish it's useless at
anything that's not go
so while it has a few tricks that allow
it to be more general it is still very
narrow in its
applications so on our plot we might put
it here again far exceeding human level
performance
but still very
low on the generality axis
now let's take a jump to a different
category entirely
um
image classification
there is a wonderful data set called
imagenet
which has many thousands of images
classified by hand by humans
into a thousand pre-defined categories
these categories include household items
cars and doors and chairs includes
animals giraffes rhinoceroses
it also includes for instance
many different species of dog
so
it's not a trivial thing to take an
image and accurately categorize it and
in fact a typical human score on this is
about five percent error about one out
of every 20 images a human will classify
incorrectly
so it's a tough
task
in 2011 uh there was began a large scale
visual recognition challenge in which
teams got to put together their
algorithms for classifying these images
and in 2011 the very best one got 26
percent error so about one in every four
image was wrongly classified still three
out of every four was correct which was
pretty amazing performance
each year this error rate decreased by
close to half
which is an amazing rate of progress
finally in 2015 the error rate got lower
than human so we had a computer program
classifying images better than a human
does
now in 2017 one of the more recent
competitions more than half of the teams
got fewer than five percent wrong
so
now machines are routinely beating
humans at this task
uh
pretty impressive
in terms of generality this task is
definitely
harder more general more challenging
than a board game
there are more variations more
possibilities
to get at it it uses convolutional
neural networks
which
are a deep neural network architecture
specifically designed for finding
patterns
in two-dimensional arrays of data
like
a pixels
or
squares on a chessboard
or on a go board they're very good at
finding patterns for that could be
visually represented
this is
good
but it has been shown to break easily
outside of the set of images that you
train it on
to give an example of this if you look
at the images on the right in the left
hand column we see soap dispensers
praying mantis a puppy
these are all images that were correctly
categorized by a convolutional neural
network
with the addition of a little bit of
distortion shown in the middle column
which you're seeing that correctly is
just a little bit of noise but the gray
has no change at all
you get the images on the right
visually to us
they look identical or
very similar you might be able to see a
little bit of warping and distortion
there
but for whatever reason uh convolutional
neural networks confidently predicted
all of these to be ostriches
so this is not to say that they are not
powerful and good but they see something
different than we are seeing they are
not learning to see in the same way that
we see
since then the fragile nature of
convolutional neural networks has been
demonstrated through other ways
in some images changing a single pixel
the right pixel to the right value can
change how that image is classified
others have found that you don't even
have to go into the digital domain you
can take carefully designed stickers and
affix them to something
and have that object be confidently
classified as a banana whatever it
is and in my favorite demonstration
a physical plastic turtle
was rotated and from every direction the
convolutional neural network confidently
predicted that it was a turtle
then after just repainting a different
pattern
not symbolic or representative of
anything
but carefully chosen that same
convolutional neural network
categorized it as a handgun
these examples show
that
uh at least as currently done our image
classification generality is not quite
where we would like it to be
so
definitely higher than human performance
but
classifying imagenet is a much narrower
task than it might appear on the surface
so we'll put it pretty low on the
generality axis
now here's a really fun example of video
game performance
so again
the folks at deepmind put together
uh deep q learning or deep reinforcement
learning architecture to play video
games we'll talk about more about what
that is in a second but what they did is
they took 49 classic
atari games
and
let the algorithm just look at the
pixels and make random moves
and the algorithm didn't know if this
was supposed to be a move after a move
right or a jump or a shoot it just took
moves
and then used reinforcement learning to
learn from the outcome and then learn
this pattern of oh when i see this and i
do this
either something good happens or
something bad happens or something
neutral happens
after doing that for long enough
it learned the patterns
that let it choose the right thing to do
and in 29 of these 49 games it did at or
above
human expert level play
and that was super impressive so this is
not just looking at a picture and saying
this is a cat this is looking at a
picture in the moment and saying for
this particular instance the right thing
to do is jump
and then by jumping
that changes the image and then having
to respond to the new configuration and
doing that again and again and again and
doing this better than a human
now
the other part of this is there were 20
games then at which it did more poorly
than a human
so after using convolutional neural
networks to learn the pixel patterns for
which this is perfectly suited because
the pixels are big and coarse there's no
noise they don't change
the patterns are clear so what the
algorithm is seeing is very close to
what we as humans see
and after uni using reinforcement
learning to learn what actions to take
in each situation
20 of these games it wasn't able to
match human performance on
and the pattern among those games is
they tended to require longer term
planning
the one of them was ms pac-man and if
you've ever played that you know you're
trying to eat all the dots in a maze
while avoiding ghosts who are chasing
you
it involves planning routes out several
turns ahead anticipating where ghosts
are going to be
a lot of things that you can't get from
a single snapshot
and without thinking ahead several steps
in its current state this algorithm
didn't do that
and in fact the game that did the
poorest on a game called montezuma's
revenge required much more extensive
planning going to one location and
grabbing an object so you could go to
another location and open a door
and the computer just was not able to
make those connections
so we'll add video games to our plot
here
again more general than image
classification there's more going on the
task is broader
and performance is about human level
now you may notice a pattern here these
fit roughly into a line or a curve and
we'll see this pattern continue
looking at machine translation taking
text and changing it from one language
to another
if you ever gone to an online translator
and typed in a phrase or copied a phrase
from a language you weren't familiar
with to one that you were
you will probably notice that the
translation is surprisingly good at
getting some of the sense
which is
even five years ago was complete science
fiction to be able to do this in a
reliable way
you'll probably also
notice that the result is nothing that a
native language speaker would ever be
likely to say
so it's
it's okay it's definitely in the right
direction
but it's far from
perfect now what's really impressive to
me about this is that the state of the
art and language translation
takes
over a hundred languages and instead of
having specific models to translate from
each language to each language
all of these languages are translated to
a sum
uber intermediate representation
which is then able to be translated back
into any one of these hundred plus
languages so all to all language
translator
so the sheer scope of that is really
impressive
now in order to do this
it uses long short term memory
lstm which is a neural network
architecture and it actually uses
several
deep neural networks
in concert
one to
carefully ignore parts of the input
one to choose what to remember one to
choose what to forget when to choose
what to pass on
there's quite a bit of computation
involved
and this architecture uses several
levels of those even
so the the amount of effort and
computing power
thrown at this in general um
is
if we use one of our metrics as
efficiency it's a little bit could be
considered a little bit of a hit
in addition to the inaccuracies
it is worth noting that this uses an
attention mechanism so
i called out attention earlier as a
possible limitation of human performance
but it also proves to be a really useful
tool
when dealing with a massive amount of
information
too much
to
look at in depth
and so by pre-filtering and focusing
down on what's most likely to be helpful
then in an algorithm can be much more
efficient in how it handles it
so
for machine translations
amazing
performance still short of human
and for the
wildly
ambitious scope
it gets a
little step up on the generality axis a
little bit of a hit on their performance
axis
translation is still a very small part
of all the things that humans do
but
i would definitely say this is more
general than playing video games
now looking at recommenders
so if you think back to your last
experience with an on time online real
online retailer
probably
the recommendations that you got were
maybe one in 10 was really really
relevant
some of the others were close but near
misses and some of the others were
obviously way out in left field
so
this is
still pretty good like this is a tough
task if you can imagine like uh back
when there were video stores going to a
video store with your friend and trying
to guess
what your friend even a friend you knew
very well
would want to watch on a given night you
know you would be hard-pressed to do
better than like one and three or one
and four
so you know one in ten is not terrible
or just ballpark
it's common to assume among these
algorithms that order doesn't matter
so it just looks at everything you've
ever bought
today yesterday last year
and it doesn't think about how these
things are related or how many you might
have or how many you might need or how
something that you bought previously
might be related to what you might need
tomorrow
it just looks at what people have bought
in the past
what they've bought together
it also doesn't
adapt to the fact that your selections
might change with time so
even if you bought one jar of mayonnaise
a year ago and then another one six
months ago and another one a few months
ago it might not
track the fact that your preference has
changed
one of my favorite examples of these
came up from
jack raynor twitter user who said dear
amazon i bought a toilet seat because i
needed one
necessity not desire
i do not collect them i am not a toilet
seat addict
no matter how temptingly you email me
i'm not going to think
oh go on then just one more toilet seat
i'll treat myself
so recommenders
they do
okay
relative to humans and i would argue
that the knowledge of the world required
to do really well
is pretty deep so we'll boost it up on
the generality scale but it takes a hit
on performance
now
we get to robots finally
something physical bumping around in the
world self-driving cars
their performance is impressive
taken overall per mile driven
self-driving cars accident rates are
lower than humans
and this is pretty amazing when you
consider
all of the things that a car has to deal
with
construction
pedestrians bicycle riders
changing weather conditions changing
road conditions
they're not perfect but they're
surprisingly good
um now in terms of generality
there are a few things
that make self-driving cars less general
than they might at first appear and in
fact the biggest tricks to making them
successful is to reducing the difficulty
of the task and so reducing the
necessary generality of the solution
so one of the things that happens is
especially during training humans still
have to take over in some challenging
situations
when the human gets uneasy or the car
signals it doesn't know what to do then
it falls back to the human
and while driving is still a pretty
complicated task
it's still very simple compared to say
walking on rough terrain
while eating a bagel
and walking a dog who's pulling on the
leash
there's a lot more to consider there and
it's a lot tougher than
a car who is statically stable on four
wheels on a road that's flat and mostly
straight and mostly marked
and
where the rules are well prescribed
to further simplify the task and narrow
the scope of what needs to be learned
self-driving car's driving style tends
to be cautious
they definitely do not tend to speed
they tend to not follow closely or turn
aggressively
or do anything else that many human
drivers do
this is absolutely good practice
and should be lauded and as a model for
all of us to follow
but what that means is that the
raw driving skill
required by a self-driving car
is in general
less than that required by a human
and it should also be noted that
solutions are custom engineered for
driving
the selection of sensors the algorithms
used to process them the way everything
is put together is
not updated on the fly
it's gathered
evaluated by humans
and
then very
carefully and deliberately
the heuristics the rules behind how
that's interpreted and processed are
then updated and tested and released
again
this makes sense for deploying anything
that is has such a high consequence as a
car
but
from a machine learning side it means
that the solution is actually not as
general as it seems it's
very specific to a given
car with a given set of sensors and
sometimes even to a given environment
some of some self-driving cars at least
early failures
had to do with being deployed in
climates that they weren't familiar with
for instance
so until
their set of training data encompasses
all of the conditions in which they will
be
deployed
they will be even narrower than human
drivers
so
all these things considered on the task
of driving in general
i chose to rate self-driving cars at
lower performance than human
still its physical interaction and its
interaction with other people in other
cars so it's
quite a lot going on and definitely more
complex than machine translation or any
even recommendations
now
humanoid robots
the apex of
cool applications
um if you have not yet done it
get on the internet and search for
robots doing backflips and check it out
when you see something like this it is
easy to make the jump
to believe that robotics has been solved
like when a robot can do physical feats
of acrobatics that i can't do
then i mean it's done i'm ready to call
it and um
yeah it's just uh
it puts a smile on my face that i can't
wipe off
now in terms of generality
um do another search for robots falling
down
and
you'll be treated to a montage of
really
humorous shorts of robots trying to do
very simple things
like open a door or lift an empty box
or even stay standing up
and they really
struggle with this
um because
the systems are so complex because the
hardware and the sensors have so much
going on
and because most of these are deployed
as as research projects
most of these activities are
fairly hard-coded and pretty fragile
they make a lot of assumptions about
the nature of the hardware what's going
on the nature of the environment and if
any of these assumptions are violated
the robot's performance
fails
so as a result uh plotting them here
the generality in the sense that the
types of things that they have
experienced taken together as a whole
are
now getting to be a non
negligible fraction of things that
humans can do you know maybe it's 0.1
maybe it's 0.01 somewhere in between
but
an amazing set of things that can be
quite hard
but performance is still sometimes
laughably low compared to human level
um we can
you know compare humanoid robots as
agents
to humans and see that it's much less
more interestingly here
our trend now is quite clear
there's a fat line here that runs
roughly parallel
offset from the human level intelligence
line
as solutions
tend to get higher performance they also
tend to get less general and vice versa
but it's rare that we get big steps
toward the human intelligence line
now
this is this is what i think is cool
this is the whole point of this talk
there is one example that i would like
to show of this before i
jump to the conclusion
which is again from deep mind a program
called alpha zero
so alpha zero is like alphago
except
everything it knows about go
has been taken out
it doesn't know the rules of any game
now it just sees visual patterns
tries actions and learns to see what is
successful and what's not
the way it was used is you can think of
a brand new alpha zero instance as being
an infant in terms of the gameplay and
two alpha zero infants were created and
they started to play each other one was
allowed to learn the other one was not
so the one that learned gradually got
just a little bit better stumbling into
some good moves by accident until it
became an okay beginner player of the
game
then
it cloned itself
one of the two learned the other did not
and they played and played until the one
became an intermediate level player of
the game
and it repeated this process of cloning
and playing itself with one learning and
the other not
and used
its
intermediate steps
as scaffolding to get better and better
and it turns out
that when
using this approach
with go
within four hours it was as good as the
best human player
and within eight hours
it had beat
the previous best computer its
uncle
alphago
um
because it did not build any rules of
the game it was also able to learn chess
and beat the current best chess playing
program stockfish and another board game
called shogi
and be the current best shogi playing
program as well
all of which beats humans by a wide
margin
so this is cool because it
does both better performance and it's
more general it's not specific to any
board game and presumably if there were
other board games that had
two-dimensional grids and a set of rules
that was not wildly wildly different it
could learn to play those as well so
generality and performance so what we
have now is a point that is both
farther to the right
higher performance
and farther up higher generality than
the original that it
was from so this is a real increase in
area under that rectangle an increase in
intelligence this is the direction that
we want to go
so it's worth taking a step and
taking a moment and thinking about what
is it that allowed us to step in this
direction
well alpha zero made many fewer
assumptions about what was going on
and it also was able to practice as many
times as it needed to through self-play
in general
assumptions are what prevent generality
they enable performance so if i build in
knowledge of the rules of chess i'm able
to take advantage of those much more
quickly
but it also prevents me from doing
anything that's not chess
so if i turn that around making fewer
assumptions
i mean it takes me longer to do
something but it means i might be able
to learn to do more
things
so some common assumptions
sensor information is noise free we have
ideal sensors
that makes sense
if we're playing chess
when we sense that a piece is on a given
square we expect that it will be
but if we're dealing with say a
self-driving car
maybe there's a smudge of mud on the
camera maybe the calibration of the
lidar is off a little bit
we can't assume ideal sensors when we're
interacting with the physical world
there are too many things we can't
control for
another common assumption is determinism
that's when i take an action i know that
it will have the same outcome every time
again makes great sense when i have a
board game make sense if i'm classifying
images if i say an image is an image of
a cat i know that it will be labeled as
a cat image right or wrong
however if i'm a humanoid robot
and i
make an action to reach for a doorknob
the motor might not perform the way i
expect
my
feet might slip on the ground i might
have unanticipated challenges to my
balance the action may not turn out
exactly as i expect
and i need to be able to adapt to this
another really common assumption
unimodality
all of the sensors are the same type
so this is
an assumption in convolutional neural
networks for instance
it's great at bringing in a
two-dimensional array
of information that is all the same type
it's all pixels where it's all board
squares
a general solution
needs not to make this assumption
another assumption
stationarity
this is a very common one it's that the
world doesn't change the things that i
learned
yesterday
are still true today the things that i
learned five minutes ago are still true
right now
now we have to make some change sorry we
have to make some assumptions about
continuity
otherwise what i learned yesterday
doesn't do me any good at all
but we also need to allow for the fact
that the world has changed a little bit
maybe the lubrication in my ankle joint
is a little low so it's going to respond
differently than it did yesterday
maybe there are clouds covering the sun
so the lighting conditions i learned to
operate in yesterday have changed as
well and i'll need to to be able to
adapt to that
another common assumption is
independence which is the world is not
changed by what i do to it
physical interaction violates this
entirely if i am a robot operating in a
household and i
bump into a chair and i scoot it six
inches sideways then whatever map i've
made of that house
will need to be changed a little bit i
have changed it myself if i pick up a
mug and move it from this table to that
table i have changed the position of
that mug
the things that i do
change the world and i need to keep
track of that and any algorithm i use
needs to be able to account for that
another common assumption ergodicity
everything i need to know about how i
operate i can sense right at this moment
this is a common assumption also known
as a markov assumption
but it's also
commonly broken in physical interaction
for instance if i can sense position
that's great but that doesn't tell me
anything about velocity and sometimes i
need to know velocity to know how to
respond to something
another assumption that is very common
is that the effects of my actions become
apparent very soon
this is something that does not hold
true for instance in chess
where the opening move will affect
whether or not i win
many many time steps away
there are different tricks for handling
this in chess for instance assigning
point values to intermediate
positions of pieces on the board
but in physical interaction it's much
more difficult to do this
to know that
given a set of actions that i take right
now
which is most likely to result in
something that's desirable five minutes
from now or a day from now
all of these assumptions are very common
in the algorithms currently being used
that we call
ai
these algorithms
are not sufficient for achieving human
level intelligence
these assumptions will prevent them from
doing that
so one thing that all of these
assumptions have in common
is that they do not hold when you're
working with humanoid robotics or in
fact any
robot that's physically interacting with
the world
so
my proposal
is that focusing on
interact physical interaction
is a great way
to force us to confront these
assumptions to find out which ones we
can bend to find out which we can avoid
all together and to drive us
to create algorithms that are less
fragile and able to accommodate a much
more general set of tasks
that will then take us
one step closer to human level
intelligence
when you hear about artificial
intelligence about a half of the time
what people are talking about is
convolutional neural networks
understanding how they work is really
helpful in getting a peek behind the
curtain at the magic of artificial
intelligence so we're going to walk
through it in some detail
convolutional neural networks
takes images
and from them
they learn the patterns the building
blocks that make them up
so for instance in the first level of
this network you might learn things like
line segments that are at different
angles
and then at subsequent layers those get
built into things like faces or element
of cars
depending on the images that you train
the network on
you can pair this with reinforcement
learning algorithms to get algorithms
that play video games learn to play go
or even control robots
so to dig into how these work we'll
start with a very simple example much
simpler than all of these
a convolutional neural network
that can look at a very small image
and determine whether it's a picture of
an x or an o
just two categories
so for example this image on the left is
an eight by eight pixel image of an x
we want our network to classify it as an
x
similarly with the image of the o we
want the network to classify it as an o
now this is not entirely straightforward
because we also wanted to handle cases
where these inputs are of different
sizes or they're rotated or they're
heavier or they're lighter and every
time we'd like it to give us the correct
answer
a human has no problem looking at these
in deciding what to do but for a
computer this is much harder when trying
to decide if these two things are equal
what it does is it goes through
pixel by pixel black pixels might be a
minus one white pixels might be a plus
one and it'll compare them pixel by
pixel find the ones that match and here
the red pixels are the ones that don't
match
so a computer looking at this would say
uh no these are not the same they have
some matches but they have a lot that
don't match
so the way convolutional neural networks
do this one of the tricks they use is
that they match pieces of the image
so you can look at these pieces and
shift them around a little bit but as
long as the
tiny bits still match then the overall
image is still considered a pretty good
match
so these tiny bits might look like this
we'll call them features you can see the
one on the left looks like a diagonal
arm of the x that's leaning left the one
in the middle looks like the center of
the x where it crosses
and the one on the right looks like a
diagonal arm
leaning right
and you can see how these different
pieces these different features of the
image
match different patches
within the overall image
so the math behind finding this match
applying features is called filtering
it's pretty straightforward but it's
worth walking through
the way that it's done is you line the
feature up on the image patch you're
concerned with
multiply pixel by pixel
add up the values and then divide by the
total number of pixels this is one way
to do it
here for instance
we start with this feature of the arm of
the x leaning left
we align it with
this arm on the image
and we start with the upper left pixel
and we multiply both values one by one
equals one
now because we started with the upper
left pixel we can keep track of our
answers here so this pixel when
multiplied equals one
the upper center pixel is minus one
in both of the feature and the image so
minus 1 times -1 is also equal to 1 so
that
when you multiply them and you get a 1
that indicates a perfect and a strong
match
and we can continue doing this
throughout
the entire feature and the entire image
patch
and because they are a perfect match
every single one of these
matches will come back as a one
and so to find the overall match we just
add up all these nine ones divide by the
total number which is nine and we get a
a match of one
now we can create another array to keep
track
of how well the feature
when placed in this position
matched our image
so this average value is one we'll put a
one right there to keep track of that
you can see what it looks like if we
were to move this feature then and align
it to a different patch so let's say we
move it down to the center of the x
and we go pixel by pixel and find what
matches and after a few pixels we
actually find one that doesn't match we
end up with a minus 1 times a plus 1
giving us a minus 1 back
this indicates a non-match of these
pixels
and as we go through the rest of the
feature we can see that there are a
couple of pixels that don't match
so here when we add these up and divide
by nine we get a number that's less than
one point five five
so it indicates a partial match but not
a perfect one
it turns out you can go through and do
this for every possible location in the
image you can chop it up into every
possible image patch
compare the feature to each one
and here's what you would get in this
particular case
this is what convolution is it's taking
a feature and applying it across every
possible patch across a whole image
and you can see here
why it's called filtering
what we have is a map
of where this feature
matches the image
you can see a strong bunch of plus ones
on the diagonal line from the lower
right to the upper left
and then
lesser values everywhere else
so it's a filtered version of the
original image that shows where the
feature matches
you can do this
we can represent this with this notation
uh we just invented this little
convolution
operator for shorthand
and we can do this with our other
features as well
we can see where our center x matches
not surprisingly it matches strongest in
the center of the image
we can see where our leaning right arm
matches and not surprisingly it matches
along this diagonal from the lower left
to the upper right
we have three filtered versions of the
original image
so this is what
a convolution layer
in a convolutional neural network does
it has a set of features in it
and it can be three or thirty or three
hundred or three thousand
but it has a set of features
and it takes the original image
and returns
a set of filtered images one for each of
the features
so this is how we'll represent it
that's the number one ingredient in
convolutional neural networks that is
the magic
special sauce the special trick
that gets from
a non-exact match
and let's the algorithm is able to pull
out okay well it's not a perfect match
but it's still a pretty good match
because it does this convolution and
moves the feature across the image and
finds everywhere that it might match
another piece of this is called pooling
so we took our original image and now we
have a stack of images
what this step does is it shrinks it
down a little bit
we start by picking a window size
usually two or three pixels
picking a stride usually two pixels has
been shown to work well
and then walking this window
across the filtered images
and then from each window taking the
maximum value that you see so this is
called max pooling
so to see how this works we start with
one of our filtered images
we have our window which is two pixels
by two pixels
and within that the maximum value is one
so we create another little array to
keep track of all of our results and we
put a 1 in it
and then we step it over by our stride
which is 2 pixels
look at the window choose the maximum
value in this case it's 0.33
record it and go again
and we keep doing this recording the
maximum value each time all the way
through the image
and when we're done
we have which if you squint looks like a
shrunken version of the original
we still have this strong set of plus
ones on the diagonal from upper left to
lower right
and then everywhere else it's less than
that
so it maintains kind of the original
signal but shrinks it down kind of picks
off the high points
and this gives us a smaller image but
still similar to the original
and we can just represent it with this
little shrinking arrow we can do this
with each of our filtered images and
again you see that very roughly the
pattern of the original is maintained
so in a pooling layer
a stack of images becomes a stack of
smaller images
now the last ingredient we need is
normalization
so this keeps the math from breaking by
taking and tweaking these values just a
little bit it takes everything that's
negative and changes it to zero
this keeps things from
becoming unmanageably large as you
progress through subsequent layers
this function is called a rectified
linear unit
it's a fancy name for something that
just takes anything that is negative and
makes it zero so 0.77 not negative it
doesn't touch it but a minus .11 it's
negative just bumps it up to zero
and by the time you've gone through all
of your images and done this all of your
pixels and done this this is what you
have so everything that was negative is
now zero
so just a nice little bit of
normalization some conditioning to keep
things numerically well behaved
a stack of images becomes a stack of
images with no negative values
now
you can notice that the output of one
layer looks like the input to the next
there are always arrays of numbers
an image and an array of number are the
same thing they're interchangeable so
you can take the output of the
convolution layer feed it through the
rectified linear unit layer feed that
through the pooling layer
and when you're done
you have something that has had all of
these operations done to it
and in fact you can do this again and
again
and
this
recipe you can imagine making like a
scooby-doo sandwich of all of these
different layers again and again and in
different orders
um and uh some of the most successful
convolutional neural networks are
kind of like
accidentally
discovered
groups of these that just happen to work
really well so they get used again and
again
so over time each convolution layer
filters through a number of features
each rectified linear unit layer changes
everything to be non-negative and each
pooling layer shrinks it
so by the time you're done you get a
very tall stack of filtered images with
no negative values that has been
shrunken down in size
now by the time we've gone through
several iterations of this
we take and run it through a fully
connected layer this is more of a
standard neural network where every
input
gets connected to
everything
in the next layer
with a weight
every single value you can think of it
as a voting process
so every single pixel value that's left
in these filtered shrunken images gets a
vote on what the answer should be
and this vote depends on how strongly it
is tends to predict an x or an o when
this pixel
is high
is this output usually an x or is it
usually an o
so you can see for this particular input
the input was an x here's what the
imaginary convolved and filtered values
are
and
over time we would learn that these
things that are high
when it sees an x get a strong vote for
the x
category similarly
if you have an input that's an o
these
final pixel values that tend to be
really high when the right answer is row
is o gives a strong vote for the o
category
the thicknesses of these lines represent
the weights
the strength of the vote between these
pixels and these answers
so now if we get a new input that we've
never seen before these might be the
final pixel values we can use these
votes and do a weighted voting process
for both of these
add them up
and in this case you know it's a 0.92
total for x
and a 0.51 total for o
0.92 is obviously more than 0.51 we
declare x the winner so this input will
have been categorized as an x
so this is a fully connected layer so it
just takes a list of feature values in
this case our filtered shrunken pixels
and it becomes a list of votes for each
of our output categories in this case an
x or an o
now these can also be stacked you can
have like little they call them hidden
layers but like secret hidden categories
in between here so one votes on the
first layer votes on the first set of
hidden categories and then those vote on
the next layer and so forth until you
get to your final ones we'll dig into
this more in just a little sec
but
these
all stack onto the end
now to go into uh the next level of
detail on these neural networks
set aside our x and o detector for a
while there we had eight by eight
pixel images so 64 pixels in all
consider now a two by two pixel image so
just a four pixel camera
and what we would like to do
is categorize the images that it takes
as either being a solid image all light
or all dark
a vertical image
a diagonal image
or a horizontal image
now the trick here is that simple rules
can't do it so both of these are
horizontal images but
the pixel values are completely opposite
in both of them so you can't say well if
the upper left pixel is white
and the upper right pixel is white then
it must be horizontal because that's
violated by the other one
now of course you could do more
complicated rules to do this the point
is that when you go to larger images you
can't make simple rules that capture all
the cases that you want
so
how do we go about it
we take these four input pixels and
break them out we call them input
neurons but they just take these pixels
and turn them into a list of numbers
the numbers correspond to the brightness
minus one is black plus one is white
zero is middle gray and everything else
is in between
so this takes this little image and
turns it into a list of numbers
that's our input vector
now each of these you can think of it as
having a receptive field
this is
an image
which makes the value of this input as
high as possible
so if you look at our very top input
neuron
the image that makes that number as high
as possible is an upper left pixel
that's white
that makes the value of that one and it
doesn't care what the other pixels are
that's why they're checkered
so you can see that each of these has
its own corresponding receptive field
the image that makes the value as high
as it can go
now we're going to build a neuron
so when uh people talk about artificial
neural networks and a neuron we are
going to build it bit by bit
the first thing you do to build a neuron
is you take all of these inputs you add
them up
so in this case this is what we'd get so
the neuron value at this point is 0.5
now the next thing we do is we add a
weight we mentioned the weighted voting
process before so what that looks like
is each of these inputs gets assigned a
weight between plus and minus one
and
it gets the value gets multiplied by
that weight before it gets added
so now we have a weighted sum
of these input neurons
and we will represent this visually by
showing positive weights in white
negative weights in black and the
thickness of the line being
approximately
proportional to the weight and when the
weight is zero we'll leave it out to
minimize visual
clutter so now we have a weighted sum of
the inputs
the next thing that we need to do is
squash the result
so because we're going to do this a lot
of times it's nice if we always
guarantee the answer is between plus and
minus 1 after each step that keeps it
from growing numerically large
a very convenient function is this
s-shaped so sigmoid squashing function
this particular one is called the
hyperbolic tangent there is confusingly
something else that's called the sigmoid
it's a little bit different but the same
general shape but the characteristic of
this
is that you can
put in your input
you know draw a vertical line see where
it crosses the curve track that over to
the
using a horizontal line to the y axis
and you can see what the smashed version
the squashed version of your number is
so in this case 0.5 comes out to be just
under 0.5
0.65 comes out to be about 0.6 and as
you go up this curve you can see that no
matter how large your number gets
what you get out will never be greater
than one
and similarly it'll never be less than
minus one so it takes this
infinitely long number line and squashes
it so that it all falls between plus and
minus one
so we apply this function to the output
of our weighted sum
and then we get our final answer
so this
weighted sum and squash
is almost always what people are talking
about when they talk about an artificial
neuron
now we don't have to do this just once
we can do it as many times as we want
with different weights
and this collection of weighted
sum and squash neurons
is you can think of it as a layer
loosely inspired by the biological
layers of neurons in the human cortex
so
each of these has a different set of
weights here
to keep our picture really simple we'll
assume these weights are either plus one
white lines
minus one black lines or zero missing
entirely
so in this case now
we have our
layer of neurons
we can see that the receptive fields
have gotten more complex
if you look at the neuron in the first
layer on the top you can see how it
combines the inputs from the upper left
pixel and the lower left pixel
both of the weights are positive those
lines are white
and so what comes out its receptive
field is if both of those pixels on the
left are white
then it has the highest value it can
possibly have
if we look at that layer of neurons and
look at the one on the bottom we can see
that it takes its inputs
from both of the pixels on the left oh
sorry on the right
but it has a negative weight
collecting connecting it to the lower
right neuron
so its receptive field its what
maximally activates it
is a white pixel in the upper right and
a black pixel in the lower right
now we can repeat this because the
outputs of those that first layer of
neurons looks a whole lot like our input
layer still a list of numbers between -1
and 1 and so we can add additional
layers and we can do this as many times
as we want
each time
each neuron in one layer is connected to
each neuron in the other layer by some
weight
so in this case you can see how the
receptive fields might get still more
complex
and now we're starting to see patterns
that look like
the things that we're interested in
solids verticals diagonals horizontals
by combining these elements
now there's one more thing that we can
do remember our rectified linear unit
we can have different neurons here
instead of a weighted sum and squash we
can just have something that takes the
input and spits out
0 if it's negative
and
the original value if it's positive
and so for instance if we have
if we have an input whose receptive
field is the one on the very top and the
second layer all solid white and we
connect it with a positive weight
to the rectified linear unit neuron on
top
then of course what would maximize that
is all solid white input
but if we look at the
neuron just below that
that's connected to it with a negative
weight
then
that that flips everything around and
what maximally activates that
is an input that's all solid black
now we're really starting to get the set
of patterns
that we can imagine using to decide
what our images
so we connect these again to a final
output layer this output layer is the
list of all the possible answers that we
expect to get out of our classifier
originally it was x's and o's now it's
four categories solid vertical diagonal
and horizontal
and
each of these
inputs
into them have a vote
but you can see that very few of them
are connected this network assumes that
most of those votes are zero
so to see how this plays out
let's say we start with an input that
looks like the one on the left
with uh this is obviously a horizontal
image with a dark bar on top and a white
bar on the bottom
we propagate that to the input layer
and then we propagate that to the first
hidden layer
and you can see for instance the neuron
on the very top
it combines
two input neurons that one is light and
one is dark
so you can imagine it's summing a plus
one
and a minus one
and getting a sum of zero so that's why
it's gray
its value is zero
now if you look at the neuron in the
very bottom
in that first hidden layer
you can see that it sums also an input
that is
negative and one that's positive
but
it's connected to one by a negative
weight and the other by a positive
weight so it actually what it sees its
weighted sum is minus one and minus 1.
so what it is getting you can see is the
opposite
of its receptive field so that means
it's maximally activated but negatively
so that's why it's black
we move to the next layer and you can
trace these things through so anything
zero plus zero is going to get you zero
if you look at the neuron on the very
bottom of this second hidden layer
you can see that yes it's adding up a
negative and a negative both connected
by positive weight so it's also going to
be negative
which makes sense because you can see
that its receptive field is the exact
opposite of what the input is right now
so it's maximally activated
just negative
and then when we track this to our next
layer
you can see that
following that
bottom pair of neurons
because it's a negative value
it goes through the rectified linear
unit and becomes zero so that's gray but
if you look at the very bottom neuron
there
it has it's connected with a negative
weight so it becomes positive
so that rectified linear unit really
likes it
so it gives it a maximum value so
everything is zero except for that
neuron on the bottom
and then finally
what that means is that the only output
that is non-zero is this horizontal one
so this network
would classify
the input image
as being horizontal because of this
now there's some magic here where did we
get those weights where did we get the
filters in between um
this is where we start to get down
to the
when we talk about learning adaptation
you know the learning and machine
learning it is all about optimization
these are
learned through a bunch of examples
over time
so we're going to set that aside for
just a minute we'll come back to how
those get learned we need to talk about
optimization first
so consider drinking tea
there is a temperature range where it is
a delightful experience it's warm and
delicious and comfortable
if your tea is too much hotter than that
it's very painful and not good not fun
at all and if your tea is cooler than
that it's lukewarm and it's really meh
it's really not worth your time
so this area at the top is the peak this
is the best this is what we're trying to
find
in optimization we're just trying to
find the best experience the best
performance
now if we want to find that
mathematically first thing we do is we
flip it upside down
just because this is how optimization
problems are formulated
but it's the same type of thing instead
of maximizing tea drinking pleasure we
want to minimize minimize tea drinking
suffering we want to find the bottom of
that valley the lowest possible
suffering
there's a few different ways we could do
this
the first is to look at
every point on this curve and just pick
the lowest one
now the trick with that is we don't
actually know what this curve is
beforehand so in order to pick the
lowest one we have to do exhaustive
search which in this case would be make
a cup of tea have someone drink it ask
them how they like it make another one
ask them how they like that one do it
again and again for every possible
temperature
and then
pick the one with the lowest suffering
the most enjoyment
this is effective very effective also it
can be very time consuming for a lot of
problems and so
we search for a shortcut
now
because
this is a valley we can use our physical
intuition and say hey well what if we
just had a marble and we let it roll to
the bottom of this valley we wouldn't
have to explore every single piece of it
so this is what's behind gradient
descent
the way it works is we start not knowing
anything about this function we make a
cup of tea someone tells us how they
like it
and then we change the temperature a
little bit we make another cup of tea
just a little bit cooler
we ask someone how they like that and we
find out they actually like it just a
little bit less
that tells us what direction we need to
go we need to make our next cup of tea
warmer
and
the change the difference between how
much they like those two tells us the
slope tells us the steepness gives us a
sense of how much warmer we can expect
to make that next cup of tea
so we make another one
and we repeat the process
and then we
again scoot a little ways off to the
side make another cup of tea and figure
out again which direction we need to go
are we do we need to go warmer to make a
better cup or cooler to make a better
cup
and we repeat this until we get to the
bottom
of the curve
you'll know you're at the bottom
when you change the temperature just a
little bit and the tea drinker says yeah
it's exactly the same i like that just
as much as the last one
that means that you're there kind of at
the flat bottom of the valley
so um
gradient descent is the first level
trick
for brewing fewer cups of tea there's
another thing you can do which is to use
curvature this is kind of an advanced
method is you can
make
your original cup of tea and then make
one a little bit warmer and one a little
bit cooler and you can look to see how
that curve of your function goes
and if it's very steep and getting
steeper then you know you can take a
giant step because you're probably not
anywhere close to the bottom
and then you can do it again and if that
curvature is starting to bottom out then
you can take a smaller step because you
the signal that you're getting closer to
the bottom and it helps you to do this
in fewer steps
as long as your curve is relatively well
behaved
which is not always the case
so uh ways that this can break
imagine we're doing this on a hot day
and actually it turns out that if we
were to cool our tea way down we'd get a
really nice iced tea
which turns out to be even more popular
with our tea drinkers
but gradient descent would never find
this gradient descent always rolls down
to the bottom of the nearest valley it
doesn't hop around to see if there are
any valleys hiding anywhere else
another problem is
let's say there's a wiggle on our curve
there is something happening in the
environment we have noisy buses driving
by and it affects how people enjoy their
tea
we might not be able to find this very
lowest dip because we might get stuck in
a dip further up the curve
similarly if we ask our tea drinkers to
rate their tea drinking experience on a
scale from one to ten
we get these discrete jumps in our
function
and if you imagine a marble rolling
downhill
it downstairs it doesn't always work
well and it can get stuck on a step
without making it all the way to the
bottom
now all of these things happen
in real machine learning problems
another one imagine you have really
picky tea drinkers and if the tea is
anything but perfect they hate it hate
it hate it
and so you have these plateaus on either
side and there's no signal to tell you
that if you
move in a little bit you'll find that
deep valley
so
for cases like this
there of course we can always fall back
to exhaustive exploration
it will find the best answer in every
single one of those cases
but a lot of times we just don't have
the time like if i have to brew and
measure the pleasure drinking pleasure
of 10 million cups of tea to get a good
answer to this is not going to happen in
my lifetime
so
luckily there are some things in the
middle that are
more sample efficient than exhaustive
exploration
but a little bit more robust than
gradient descent things like genetic
algorithms simulated annealing things
that
their defining characteristic is they
have a little bit of random jumping
around they're a little bit of
unpredictability and so they make it
harder to slip things by them
they all have their strength and
weaknesses
there tend to be good for different
types of problems or different types of
pathologies in your loss function
but
all of them
help avoid getting stuck in the local
minima the little small valleys that
gradient descent will get stuck in
they get away with this by making fewer
assumptions
and they can take a little longer to
compute than gradient descent but not
nearly so long as exhaustive exploration
you can think of gradient descent as
being like a formula one race car and if
you have a really nice well-behaved
track it is fast
but if you put a speed bump in the truck
you're done
um
genetic algorithms simulated annealing
evolutionary algorithms those are like
you know a four-wheel drive pickup truck
you can take a fairly rough road with
those and get where you're going
you won't get there in record time
perhaps but you'll get there
and then exhaustive exploration is like
traveling on foot
there is nothing that will stop you from
getting anywhere you can travel little
or literally anywhere but it just might
take you a really really long time
so to illustrate how this works
imagine we have a model that we would
like to optimize
we have a research question how many m
m's are in a bag of m ms
so answering this is easy you buy a bag
of m ms you eat it
53 you can count those m ms
so
great we know how many were in the first
bag
now when i did this i made a mistake and
i bought another bag
and i tried that one and i got a
different answer so now
i can answer 53
or i can answer 57
either way i'm only right half the time
because
i can't capture both bags with one
answer and i could answer somewhere in
the middle but
that's never right i have never opened a
bag that had 55 m ms in it so it's
unclear that that's the right answer
either
and
the situation does not improve with the
more bags of m ms that i ate it just
gets a little bit
out of control and so
i change my goal
from
answering the answer answering the
question right to answering the question
in a way that is less
wrong so in order to do that i have to
get really specific about what i mean by
how wrong i am
and to do that i have
this distance function this deviation
which is the difference between my
actual guess
and
the actual number of m m's in a bag
so we call for for bag number i this
deviation is d sub i
it's just the difference between the
guess and the actual number
and then i have to take this deviation
and turn it into a cost
so one common way to do this is to
square it it's nice as the further away
things get kind of the more costly it is
perceived as and it goes up faster so if
there's a bag that's off by twice as
much as another the cost is four times
so i really it penalizes the things that
are way off things that are close it
doesn't penalize so much
and if we don't care if we don't want to
overly
penalize the things that are way out
there we could use the absolute value of
the deviation so if it's off by twice as
much it'll just be twice the cost
but really we could use anything we
could use the square root of the
absolute value we could use 10 to the
power of the absolute value of this
deviation anything that goes up the
further away you get from zero
we'll stick with squared
deviation this is super common it has
some nice properties and
makes for a good example
so for the total cost of any guess that
we make if i guess
n estimated
bags m m's in a bag
then the loss function this fancy curly
q l of that guess
is just adding up
the square of the deviation associated
with each bag of m ms d1 through dm
squared
so each deviation is actually the number
of m ms in that bag
minus the guess
so i square that
and we can write that with fancy
summation notation like this so this is
my loss function this is the total cost
this is how wrong i am when i make a
guess
nst
so
because we have computers you can write
a little bit of code and you can do
exhaustive exploration and i can say if
i guess anything between 40 and 70
how wrong would i be with this data and
you can plot it and
visually we can look at this and we can
say hey look there's the lowest value
and we can say what is the value
of the guess
that gives me the lowest loss that's
what that argument that
um notation means right there
and this best guess is just about 55 and
a half m ms
problem solved
so
this is an example of numerical
optimization where we calculate this
loss function and then we can do
essentially because it's simulated we
can do exhaustive exploration and just
pick off the lowest value
now for this particular example there's
a fun other way to find it
we know at the bottom of this curve the
slope is 0. it's the only place in the
whole curve where that's true where it's
flat
we can use a little bit of calculus to
find that
feel free to tune out if calculus is not
your thing
but it's not too bad so we find the
slope
of the loss function with respect to our
guesses and we set it equal to 0 and we
solve it to find
what for what guess is that true
so we take our loss function this sum of
the square of the differences
of the count and the guess and we take
the derivative of that with respect to
our guess
and the
derivative of a sum is the same as the
sum of the derivatives
we take the derivative of that just
bring down the exponent so two times
that
summed because all that's equal to zero
we can divide it by two and it'll still
be true
so now the sum
of our deviations is 0.
so
to further simplify this
it's the sum of all of the counts of the
actual bags times the sum of our guess
once for each bag if we have m bags then
it's m times our guess
and then we can move that to the other
side of the equal sign and divide both
sides by the number of bags m and what
we get is that our best guess
is the sum total
of the number of m ms we found in all
the bags divided by the number of bags
or the average count per bag
so this is a really slick result and
it's things like this that make people
so excited about optimization with a
little bit of math and calculus you can
get this nice theoretical result
um now it's worth noting that this is
only true if you use a
deviation squared as your cost function
so that's one reason people like it so
much is because it tends to give some
nice results like this
but there is this analytical shortcut to
find what the best guess is we're going
to come back to this in a few minutes
now
how does optimization change
how do we use it
in our neural network to find these
weights and these features
so what we want to do
we know what our error function is it's
how wrong our guesses are
so in this case we have a labeled data
set which means that a human has already
looked at this input on the left and
they said hey that's a horizontal image
the truth values are what we know
should be the right answer
zero votes for everything except
horizontal that should have a vote of
one
so let's say
initially we've got a neural network
that
all the weights are random
and it gives us
nonsense results it says well yeah
everything is
has some number associated with it but
it's nothing like the right answer
well we can find the error for each
category and add it up and find a total
error and this is how wrong our neural
network would be for this one example
here's our loss here's our error
now
the idea with gradient descent is we're
not just adjusting one thing we're not
just adjusting our guess of the number
of m ms
we're adjusting many things we want to
go through and adjust every single
weight
in every single layer to bring this
error down a little bit
now
that is a little bit challenging to do
because in order to do that one thing
you can do
is
find a
analytical solution like we did before
to go through and
move our guess a little bit up and a
little bit down and find the slope is
really expensive when you consider that
this is not a one-dimensional problem
anymore it might have hundreds or
millions of different weights that we
need to adjust
so calculating that gradient that slope
requires
hundreds or millions of more passes
through the neural network to find out
which direction is downhill
enter
back propagation
so remember we found the nice
analytical
solution
to what we had going on
in the case of the
m m estimate
so we would love to be able to do
something like that again if we had an
analytical solution we could jump right
to the right answer
so
slope in this case it's
change in weight
or sorry it's change in error for a
given change in weight
that's the slope here so there's lots of
ways to write that
but delta error delta weight d air d
wait we'll use this partial error
partial weight just because it's most
correct but all these things mean the
same thing
if i change the weight
by one
how much will the error change what is
the slope
so in this case it would be -2 and we
would know that we need to increase the
weight in order to get closer to the
bottom this tells us not only the
direction we need to move but gives us a
sense of about how far we should go
doesn't tell us exactly where the bottom
is but it tells us which way it needs to
adjust
now if we do know the error function
example we can make
a
an analytic solution and we can find
that we can calculate that slope exactly
so in this case
the change in error
for a given change in weight is just the
derivative of our error function here
which is in this case is the weight
squared
so the derivative is two times the
weight
the weight is minus one and so the
answer is oh a slope of minus two that
tells us what we need to know about
which way to adjust
now with neural networks of course
they're a lot more complex than that
but
we can actually
analytically
compute the slope
of the function where we are we don't
know where the minimum is but we can
find the slope without having to
recalculate the value
of everything each time and this is how
it works
imagine the world's most trivial neural
network that has one input one output
one hidden layer with one neuron in it
so it's got an input connected by a
weight w1
to an intermediate value connected by a
weight w2
to an output value
so the intermediate value is just x
times that weight
so
the derivative
of y with respect to the weight is x
what that means is if i change
w1
and i move it by one
then the value of y
will change by the value
x whatever x is
we have the slope of this piece
of the function
similarly it's
straightforward we can just read off
that whatever the value of y is
multiply it by the weight w2 we get e
so if we want to find the slope of the
error function
for a given change in y
the answer is w2 if i changed y by one
unit
then the error changes by the amount w
two
now chaining
means that we can take these two things
and just
multiply them together
so
by inspection we can see that
in this little neural network
if we take x
multiply it by w1 multiply that by w2 we
get the error e
now what we'd like to know is if i
change
that w1 by a certain amount how much
does the error change
well in this case we just take that
whole expression
and take the derivative derivative with
respect to w1
and
fairly trivial bit of calculus it comes
out to be x times w2 and
what we can see then is we can
substitute in these steps
this
change in y with respect to w1 is the
same as x
w 2 is the same as the change in error
with respect to y
and what this breaks down is
if we want to step down the chain we
want to know how much a change in w1
affects the error
what is d e d w 1
we can actually break it down into steps
and say okay well if i change w 1 how
much does y change
and then if i change y how much does the
error change
this is chaining and this is what lets
us
if we know
which way we want to change the error
it lets us calculate how much we can
change this weight to help that happen
and there's nothing to prevent us from
doing this again and again if i have a
weight that's deep into my neural
network
and i want to know how much my error is
going to change if i tweak it
up or down
i want to know the slope of my loss
function with respect to that weight
then i can just break it down and say
okay well if i change the weight how
much does a change if i change a how
much does b change if i change b how
much does c change and chain it all the
way down
now it's called back propagation
because in order to calculate it we
actually need the value at the end we
have to start with the error value
in order to calculate each of these all
the way back down into the depths of the
network but still we can do that
now the way
the reason you have to go backwards
is that
let's say
we want to know what this should be
if i change the error if i change a how
much does the error change
it's like well let's assume that i
already know how much the error is going
to change if i change b
what is this back propagation step what
is the additional link i need to add to
this chain
it's like well it's how much does b
change if i change a
if they're connected by a weight
then how do i incorporate that weight
we know that
two neurons connected in this way are
represented by this
b is the weight times the value of a
and so we can just take a little
derivative here and get
the change in b with respect to a
is w
so this
step
in the back propagation change
can be represented by whatever that
weight is
cool
now we know that we have sums in our
neural network that's another thing we
have to handle
if i know how much my error changes with
a change in z
then how much would it change
with a change in one of the inputs to
this to z where that input goes into a
sum
well i have can write the expression for
z
adding up all the inputs
if i want to know how much z changes
with respect to a change in a i just
take the derivative
and it is turns out to be one
so this is a trivial back propagation
step
now the most interesting one of all
if
i know how much the error changes with
respect to a change in b
and then i want to know how much it
changes with the input and to that
sigmoid function
then i can just say okay well a sigmoid
function mathematically looks like this
and i can take the derivative of b with
respect to a
and um
one of the beautiful things about the
sigmoid function is that the derivative
actually looks like this
um it's just the value of the function
times one minus the value of the
function which is one of the reasons
that sigmoids perhaps are so popular in
deep neural networks
so this step is also straightforward to
calculate
in none of these steps have we had to
recalculate all of the values in
the
neural network
we've been able to rely on things that
have already been calculated what the
values are
at each of these neurons
that's what makes
back propagation so mathematically
efficient and is what allows us
to
efficiently train neural networks
that is why each
element in a neural network no matter
how exotic it is
needs to remain differentiable so that
we can go through this exercise of
finding what the link in the chain is
when we're doing the chain rule on our
derivatives so that we can compute the
back propagation we can back propagate
it
and again rectified linear units
if we know how much the output
affects a change in error we want to
know how that extends to the input we
can write the function of a rectified
linear unit we can take the derivative
of it and then use that
in our chain rule
so imagine now
we have this labeled example
we calculate the answer that this random
neural network that's not special at all
it'll give an answer that's completely
wrong
and then we back propagate the error and
adjust every one of those weights a
little bit in the right direction
and we do that again and again after you
do that a few thousand times this
stochastic gradient descent goes from
this fully connected totally random
neural network to something that is a
lot more efficient and that is able to
give answers that are much closer to the
right answer
so coming back up
to our convolutional neural networks
these are the fully connected layers
that's how they're trained
they can also be stacked
this back propagation
up applies not only to these fully
connected layers but also to the
convolutional layers and the pooling
layers we won't go through and calculate
the chain rule for them but you can do
that as well
and
going through this this whole
stack
of different layers gets trained on a
bunch of examples in this cases of
labeled x's and o's give it a bunch of
inputs that we know the right answer to
and we let it adjust all of those
connections
not only that it also adjusts all of the
pixels in the features for each
convolutional layer so it learns not
only the weights but also the features
and then over time those representations
become
something that lets it predict very well
what is an x and what is an o
on top of that there are other things
that we can use optimization for so
there's a bunch of decisions here that
we haven't addressed yet how do we know
how many features to put in each
convolutional layer
how do we know how big those should be
how many pixels on a side how do we
choose the
size and stride of our pooling windows
in our fully connected layers how many
layers do we have and how many hidden
neurons do we put in each
each of these decisions
are called hyper parameters they are
also values that we get to choose but
they're the next level up they kind of
control how everything happens below
and in order to
see how well they perform we have to
train the whole thing on all the images
start to finish
so but the same principles apply we can
adjust these
and choose them
to get the best
result possible
in a lot of cases it's worth pointing
out that there's just not enough
computation available in the world to
try out all the possible examples
and so what we have right now
are some recipes some things that
researchers have stumbled onto that seem
to work well
and they get reused
but there are a lot of places a lot of
combinations of these hyper parameters
that actually haven't been tried yet
and so there is always the possibility
that there are some combinations that
work even much better than what we've
seen so far
now
uh
we don't have to use convolutional
neural networks for just images
any
two-dimensional or three-dimensional
data
works well
the thing that matters
is that in this data
things that are closer together
are more closely related than things far
away
it matters if two things are in adjacent
rows or in adjacent columns
so in images this is plainly the case
the location of a pixel in an array of
pixels is part of the information if you
were to randomly jumble the rows and
columns that would lose the information
that's there
that's what makes this well suited to
convolutional neural networks
anything you can make look like an image
may also be suited to convolutional
neural networks
for instance if you're working with
audio
you have a really nice
x-axis your columns can be subsequent
time steps
you don't want to jumble those because
the time the order in which things occur
in time matters
and you can make your rows
the intensity in different frequency
bands going from low frequency to high
frequency
again the order matters there
and so being able to take sound then and
make it look like an image you can apply
this processing to it and find patterns
in the sound that you wouldn't be able
to find conveniently any other way
you can also do this with text with a
little bit of work you can make each of
your rows a different word in the
dictionary and then you can make your
columns the position and sentence or
position
location that occurs in time
now there are some limitations here
convolutional neural networks only
capture local spatial patterns so if
your data can't be made to look like an
image or if it doesn't make sense to
then they're less useful
so for example imagine you have customer
data
that has columns representing things
like names and ages addresses emails
purchases transactions browsing
histories and these customers are listed
if you were to rearrange the rows
or rearrange the columns
the information itself wouldn't really
be compromised it would all still be
there it would also be queryable
searchable and interpretable
convolutional neural networks don't help
you here they look for spatial patterns
so if the spatial organization of your
data is not important it will not be
able to find
what matters
so rule of thumb if your data is just as
useful
after swapping your columns with each
other
then you can't use convolutional neural
networks you shouldn't use convolutional
neural networks that's a big takeaway
from this
so they're really good at finding
patterns and using them to classify
images that is what is they are the best
at
now the takeaway from this is not that
you should go and code up your own
convolutional neural networks from
scratch
um you can it's a great exercise it's a
lot of fun but when you go to actually
use it there are a lot of mature tools
out there that are
helpful and just waiting to be applied
to this
the takeaway from this is that you will
be asked to make a lot of subtle
decisions
about how to prepare your data
and feed it in how to interpret the
results and how to choose these hyper
parameters for this
it helps to know
what's going to be done with your data
and what it all means so you can get the
most out of these tools
all right good luck