all right hello world this is cs50 live
my name is Colton Ogden joined today
again by C's buddies Nick Wong Oh Nick
what are we talking about today so today
we're continuing a little bit of our
subject from last stream we're taking a
step back for like a moment from images
and then we're hopping into neural
networks to kind of the biggest buzzword
of ML at the moment going a little bit
more level than images work on see ya
never gonna use a little bit easier data
set when I build kind of a sample
version of this this last weekend it
took I think like an hour or so not
great so we're going to use a very
simple data set and figure out how to
just what a neural network is you know
what is it neuron because they're not
quite the same as the ones in our heads
or in the rest of our bodies and it's
kind of complex there's a lot of weird
terms like backprop and gradient descent
and things like that none of these
things I know by the way yeah yeah I
didn't know them for quite some time we
talked about this before the stream and
I there's a lot of jargon but it'd be
cool to the tons of jargon elucidate
what it all means if anybody's a
familiar see vidya live is a show hosted
here at Harvard University cs50 is
Harvard's intro to computer science
taught by a David Malin who may make an
appearance in the chat and this is kind
of an offshoot of that where we build
things from scratch or talk about
concepts nick is a regular here on the
stream we've done not only programming
things but also operating system related
things kali linux was a cool thing i
talked about cursive at some point and
the first thing was actually a machine
learning related it was a classifiers a
very high level approach yeah we're
gonna use some api's we're gonna just
kind of skim over a lot of the math now
we're gonna dive a little bit more into
the math and then we're going to
actually end up motivating the
high-level approach again but this time
we're going to explain each of the kind
of lower-level concepts that we talked
about actually I think in that stream
yeah the that first stream was we used
tensor flow yeah even higher yeah yes
today we're gonna be using just numpy
just numbers with the goal of eventually
of continue because this is kind of like
a new sub series yeah we're we're doing
a sort of like building a neural network
talking about machine learning
yep with the eventual goal of being able
to generate images from prior streams
which we talked about last week yep and
we're going towards that I think that
next week that's where we're gonna start
hitting you with these like generative
neural networks this kind of like do we
have variational Auto encoding we have
kind of like distributional networks and
then we have Gans which are the
generative adversarial networks and
those are like state of the art at the
moment they're super cool none of the
things that generate the really cool
landscape yeah they produce all these
like very sharp images so even like kind
of like deep baked this is kind of a
similar concept where it's like you
can't necessarily tell that it's not a
real person like the presidential videos
where they like dancing or someone else
yeah wacky the only thing that I think
will probably explore we might even
start with next week because it's a
little easier to build from scratch is
style transfer networks so let's say I
give you like van Gogh painting and I
then hand you a modern-day picture of
something right I gotta transfer that
style onto that and it's beautiful I
mean they create this just like
incredible works of art and it's pretty
easy to build once we have kind of like
our basic understanding of like what's a
CNN what is convolution there's a lot of
steps along the way yes it's kind of
like warming but I think it's cool
there's a lot so if you're ever confused
luckily this on video so you guys go
through that if anybody wants to watch
on YouTube our first part where we built
yeah kami intensifier yes because we're
gonna eventually need to classify images
so much presumed to be able to generate
them migrating against those those
clothes today
exactly but yeah you can check that out
that's on YouTube that should be Avada
at the time of this recording actually
true Cooney says hi oh nice job on the
podcast : thank you very much a nice
show we are releasing a podcast very
soon probably today we just filmed it
it's on YouTube and all the major sort
of spot of our not full not all the
major Spotify about a major podcast
distribution channels including Spotify
but that's to be formaly now it's
probably on Facebook and and otherwise
but yeah definitely check that out
sweet wife Sarah so some nice music
thank you very much
cocoa thanks so much everybody for
tuning in I say let's just we have a lot
to cover - we do it exciting again why
don't we just don't let me just dive in
yeah so let me just do the old people
can actually see what I'm doing now we
changed our camera lens a lot wider so
so boom there we go
classic C matrix screensaver cool so
we're gonna hop out of that and we have
an empty file it's a start
programming oh my kind it again it kind
of looks like that does I guess yeah no
one ever really seen those before yeah
that's true yeah well I guess I gotta do
this but it's a little harder the signal
yeah Wow
yeah so we're starting out with a just
kind of a Rahab 5:02 kind of like keep
ourselves on track I do have a like
reference file in case I kind of forget
some of the math on the top of my head a
lot of this is very yeah there's a lot
of like very very down in the nitty
gritty details so we're gonna start with
kind of like my favorite import numpy oh
you want a little preview of every but
we got this overeager I know we're going
to start with numpy and for those that
are familiar in dumpee is a pretty
awesome library let's do all sorts of
beautiful things mathematically it also
has just kind of like inbuilt matrix
operations and matrices that make all
sorts of math really convenient we're
going to have kind of a simple X data
set which is going to be a numpy array
of literally just will put 0 0 0 1 and
for those that are kind of curious as to
what we're doing right now and why we're
doing it it's a great curiosity to have
essentially we're just creating kind of
like a toy data set and the reason that
it's really just kind of like a toy
thing is the function that we are kind
of testing is deterministic I can give
it an input tell you immediately what
the output is it's human computable it's
not particularly interesting but it
motivates using a neural net because
it's function is nonlinear and it's not
separable in any way by a linear
function now if you're familiar with
like basis functions you can actually
transform this into a basis in which it
is linear but we're going to pretend
like you can't do that in order to force
us to use a neural net area in those
so in case maybe if you're watching very
closely you'll notice that this is the
exclusive or function so given
exclusively 1 1 as its input it will
return to you 1 anything else returns 0
so you'll notice that two zeros that's a
zero and two ones that's also a zero and
this can be generalized to be kind of
like the parody of a of an input so if I
give it like 100 zeros and then one one
it'll also return one there are variants
on exclusive or kind of like
applications of it that also carry these
properties but we're gonna start with
the simplest possible data set you'll
notice we're ignoring like batching
we're not gonna split this into training
and validation and testing we're not
gonna do any of that there's no
unlabeled data this is very much just
like a toy set and we want to figure out
how to build a neural net on top of that
and speaking of neural net is kind of
like this tournament gets thrown around
all the time it's not necessarily
intuitive like what it is it's a network
of some sort but even that's not super
intuitively defined Network meaning
things can talk to each other and there
is some path by which they do that now
that path sounds kind of simple in
common in this high-level overview but
as we go into it you'll notice that
there's a lot of math involved and then
the neuro part is where we'll start
which is you know what's a neuron and
it's modeled after the idea of you know
our heads in our heads we have this
individual I can err on that all
communicate with each other and they do
something that was for a long time
thought to be very simple you gave them
some sort of stimulus and they either
fired an action potential or didn't
that's the biological neurons now that's
not quite true there are actually kind
of a hidden complexity to neurons where
they actually have sub kind of like
dendrites that poke out and those can
also fire many potentials and then
certain accumulations of these
potentials can then fire the overall
neurons action potential and what that
means and why it's interesting is now we
have a way to do nonlinear action
potentials we can do all sorts of wild
things in our heads it partially also
explains why our brains are so complex
or at least gives an understanding for
the intuition why now the neurons that
we're building are a lot simpler they're
not they're modeled in the same concept
but it's not quite analogous so the
neural sorry I just posted in our chat
said my brain is a neural network it
only contains zeros that's great I'm
sure that's not true but very funny joke
so our neural network and neural
networks in general they consist of each
neuron is essentially one wait and one
bias now to say it that way it sounds a
little funny and if you're familiar with
this at all hearing that it's a little
wack but that's kind of the intuition is
I have some weight which is just a
number that says hey anytime I'm given
an input I'm gonna apply this number to
that input and I have some bias so I
know somehow that I am off linearly from
whatever the actual answer is if you're
familiar like y equals MX plus B that
would be like the B term and then M
might be your weight now that's not
quite how this works it's all done
through matrix math so things get
generalized a little bit more broadly
but that's kind of the intuition is
every time a piece of or an input
feature is given the neuron that gets it
says I'm gonna multiply you by a weight
out of bias and I'm gonna pass you on to
the next layer we're gonna build a
single layer neural network which so you
may go that's Tilley and it is to a
degree a couple maybe like 40 years ago
not so silly but now it is kind of silly
we have GPUs we can calculate like
hundreds of neurons in one layer at a
time and things like that and this will
not restrict us as far as number of
neurons we have but it will restrict us
in number of layers we have and the
reason for that is there are a couple of
algorithms that I want to very much like
explain very clearly but if we were to
generalize this to a general network
with many layers that might be some of
that clarity might be lost in just the
syntax of the code so I'm gonna try and
avoid doing that so yeah what we're
gonna do is we're gonna just if I could
spell if you all have seen the streams
before I can't spell for the life of me
at least uh it's the tense the curse
though it's always in cursive live
stream coding I'm gonna get rid of this
window actually so we have some neuron
I'm going to call it neurons actually
and that's just the number of neurons in
our hidden layer we have our like input
dimension which
is our X dot shape now what is a hidden
layer so our right that's a great point
so in these networks I forgot to explain
the network part of network um the
things that are communicating with each
other are these different layers and so
if we imagine a network I like to
imagine it as like I'm standing on top
of it looking down or I'm standing at
the bottom looking up I don't know why
that's the way I imagine usually I see
it I was like a left or right yeah
critical layers let's do that maybe pull
up like a picture on Google or some yeah
uh let's illustrate this a little bit
very nice
changing background by the way thank you
thank you I hope you find of just pretty
pictures it helps me remember that the
world is normal outside school nothing
against school so a lot of people I
think imagine it like this and this is a
good image for what we're doing because
instead of having two hidden layers B
sub-1 Isis a Hitler is not anything
that's not the input-output exactly so
from the outside observer you can think
of it as like the black box part I just
have to configure my thing to work with
your input layer input would imagine I
kind of examine they I guess hidden
layer would be your brain yeah it's kind
of like most of your brain and then what
you think of is the output layer right
so it's kind of like as it goes through
you eventually then figure out something
and you're like oh and that's your
output layers like yeah yeah players
like Shrek accepting this yeah well say
that that's roughly true these are often
referred to as dense layers because and
there is kind of a cool Facebook comment
somewhere but I think a software
engineer probably an m/l engineer who
said that like there is no such thing as
a truly connect like a full true fully
connected layer these are often referred
to as fully connected layers their point
is that like in order to do that would
be almost impossible and you can go
google out if you'd like but essentially
these are referred to as like your dense
just neuron layers there's nothing super
crazy about them they basically are just
these like weights that can be applied
biases that can be applied and they all
talk to one another so every neuron from
one layer influences the neurons from
the next you can change that concepts in
order to get different behaviors there's
all sorts of things this is just kind of
like the very simple like what exactly
are we talking about news name so in
this image you have each circle
represents one neuron in the input layer
each search
does not represent a neuron instead each
circle represents one in particular
aspect or feature of the input and then
the circle of the output layer
represents our like prediction given
some input what do we what class do we
think it's in so going back to our I'm
gonna organize that a little bit better
so I can switch quickly between them
going back to this we have our input
dimensions we have our output dimensions
which is equal to our Y dot shape and
from there all we are trying to do is
figure out how to set up a hidden layer
of some sort
now initializing the weights for any
sort of like input dimension output
dimension hidden layer prediction layer
generally there's all sorts of like act
not activations initializations that you
can use with those we're gonna use just
num peas random uniform to do that so
what I'm gonna do is I'm going to
initialize my hidden weights it's going
to actually be equal to num P dot random
I can never never spell hi almost typed
random dot weights dot uniform spelling
is just killing me today and then I
believe it's sighs yeah sighs equals so
when we have this like hidden layers
weights we know we're going to multiply
them by whatever the input is and our
inputs shape you'll notice is like an N
by 2 so we'll probably transpose that in
order to get it to work properly but
let's imagine that we're actually doing
the input layer as or our hidden layer
which is what we're going to kind of
throw things into as some sort of number
of neurons so this is just neurons by
whatever the shape of our input is so
this is input dimensions one and there
are better ways of generalizing this but
I don't want to spend too much time on
like matrix math and drawing these out I
also don't have the ability to draw so I
I mean I well yeah in general I also
can't draw off with wife of me so we're
gonna just pretend that this is how this
is going to work and then we want our
output weights as well and this is again
going to be just a random uniform random
dot uniform with a size and again we
want to kind of like can
they're the matrix math of this it's a
little bit tricky if you're not super
familiar with how matrices work there's
all sorts of great tutorials online Khan
Academy has some awesome like linear
algebra tutorials I would recommend
going through them
a lot of machine learning is basically
just linear algebra at least at the like
mathematical level but to say that it is
only linear algebra only statistics this
is out on the kind of emergent behavior
and other properties that arise while
you're performing machine learning so
what we're gonna do here is because
we're now multiplying by whatever our
neural our input layer I'm sorry our
hidden layer handed to us in order to
output something is we're going to say
okay whatever our outputs dimensions
should be and again one by the neurons
and so now whatever we kind of throw
through this we should be able to figure
out the dimensions and everything should
roughly translate correctly for just
kind of our own benefit I'm going to
print these out they're not particularly
wild oops I don't know why press tab but
we'll see why they work okay yeah so we
printed out the hidden weights which is
this part of our code and then the other
half is the output weights and
essentially what we're looking at is we
have something that is roughly four by
two and then we have something that is
one layer by four so that should make
sense we have four neurons if I were to
change this to eight neurons you'll see
this grow and again we have this eight
by two which makes sense our input shape
second half is going to be just two and
then our output shape has this one by
eight and that should also make sense
comes cool so once we have all of these
things kind of set up we also need to
initialize our biases and biases you
don't start out biased kind of in a
weird metaphysical philosophers
philosophical sense you aren't born
biased at least not well supposedly and
so we're going to also initialize our
biases to just zeros
and the thing with bias is that
essentially you're adding bias to the
product of the weights and whatever
they're being multiplied with so if we
think about this kind of matrix style
our hidden weights have this shape of
well I forgot of whatever our neurons
number is by the input dimension so once
you multiply that by the input dimension
we then have some resulting shape if
you're familiar with a whole lot of
matrix math then if I take something
that is we'll say four by two and I
multiply that by something that is two
by n then each kind of stack I don't I
don't know why I'm kind of illustrating
it like this but I think this is a very
convenient way to imagine how this will
work each element of the input is then
going to go down that matrix and be
summed up into the corresponding column
and row pair if what I've just said
didn't make any sense let me find an
example to illustrate that it's a little
tricky to see I would like an example oh
here you go so I'm a very visual person
this is not necessarily the biggest
image I could have found but that might
help essentially you're going to take a
column from the thing being multiplied
and multiplied each corresponding entry
by the row of the thing multiplying and
then you're going to sum all those up
and put them in the corresponding like I
throw J column and you'll do that each
time and so you might into it that's
something that's I was gonna say n by M
but that's really difficult to hear the
difference so I'll say n by P so let's
say this is n rows P columns if I
multiply that by something that is P
rows you'll notice that P and P have to
be the same in order for this to work P
rows by Q columns then I'll end up with
something that's n by Q and that
hopefully makes sense if not yeah as
talks man and I have mentioned Khoda
Khan Academy maybe practice a little bit
with your matrix map things like that so
this would be 7 plus 18 plus 33 yes
yeah and so what Colton just did there
as he said ok 7 times 1/9 times 2 11
times 3 and so on we're gonna kind of
like fiddle around with how these work
it's not I even I started to like keep
track of them in my head at all times
it's not super intuitive it's not the
way humans really think so don't worry
about it if it's something that bothers
you but you should always be able to
like write it out and kind of like
imagine how these work I'm positive you
will see me mess these up pretty much at
least 3 times on this stream maybe more
you know we're gonna play golf with this
it makes it fun yeah it'll be kind of
entertaining I'm sure some of you will
be like wait no transpose that that's
another thing is we use a lot of matrix
terminology like a dot product
multiplication matrix exponentiation
and transpose and I think that the first
two are the really key ones like dot
product and Maitreya and and transpose
so if we were to transpose this matrix
then we can imagine you like flip it
across its diagonal so instead of being
this like 2 by 3 matrix it would now be
3 by 2 and you would just kind of rotate
it around now you will notice that if
you transpose this first matrix it is no
longer possible to compute this
multiplication this dot product because
the rows and columns do not match up so
things won't work in right symmetric
matrices always retain the earth sorry
square matrices always retain their
dimensionality because they're n by n so
if I transpose it it's still n by n but
they don't necessarily retain their
properties because they're flipped but
if it's a symmetric n square matrix then
transpose is equivalent to itself so
transposition is the identity function
for a symmetric square matrix what is
the best book to learn JavaScript oh I
seen a recent book in German read a
recent book in JavaScript but there's a
katana I feel like anybody now who's
going to YouTube and probably find like
an 8 hour video on like the basics of
JavaScript probably yeah especially I
recommend probably looking into newer
versions of JavaScript like es6 2015
javascript because that's kind of
becoming the norm also apologetic to
note because our facebook a not go live
so I was kind of like typing and
figuring stuff out on
I missed a lot of what you were
explaining no it's all good and I'm sure
we'll have to explain it a couple of
times actually in this stream so what
we're doing right now is we're saying
okay we have these kind of our neural
architecture our layers have these
weights and these biases and we need to
initialize them so we basically need to
initialize all of our neurons the hidden
layer and actually just in general I've
initialized them all to these uniform
weights so we'll actually print or we've
been printing out what those kind of
look like they're just uniformly
distributed numbers from 0 to 1 and
essentially they're very generally
speaking there are many ways to
initialize this I could use like the I
think there's gal or I could use a gamma
function here but I can also use like I
think it's normal
yeah I could use a normal function and
do this there are many different ways to
do it uniform is just a very kind of
like vanilla if you don't really know
how your like your inputs gonna behave
this is a pretty decent place to start
it's like a normal distribution random
number this is just a uniform
distribution but you could use a normal
distribution if you like and give it
some other parameters like if we wanted
this to be a normal distributed one I
believe I have to give it a location and
scale oh no they initialized to a
standard normal so I could equivalently
normalize or put these as normal
distributions and then if I print them
out and actually might be also
informative is for me to print this I'm
so proud of the easy for ministering
thank you I learn from the best
oh no that weights although it actually
wouldn't matter and then if we I know if
we do standard deviation on the same
thing and PDV it's really confusing and
numpy and are used different these
different abbreviations for standard
deviation and it's always very very
confusing so you'll notice I'm like a
pretty small data set our mean is
actually non-trivial e distant from zero
but a standard deviation is relative
again non-trivial e distant from one but
it's like pretty close so essentially
the larger these becomes so if we were
to use like sixteen hundred of these
neurons I'm gonna not print those
weights anymore and be huge you'll
notice that our mean is now very close
to zero our standard deviation is now
very close to 1
if you've taken a stats class there are
ways of measuring what very close means
but essentially this
should have a normal distribution is
what we're saying but if we switch this
back to a uniform distribution which I'm
going to start with but we can see how
swapping these out might change things
although in this example might not be
super easy the like expected value of a
uniform distribution from zero to one
should be about 0.5 and we see that here
the standard deviation I don't know if
it's out my head but it's around 0.33
sounds reasonable to me
so what are these weights eventually
going to be used for right so these
weights are our model so if you are
trying to say like oh I'm saving my
model or I want to use it later to make
a prediction this is what we're using is
essentially I have some set of weights
some set of biases and I want to know
how do these how does my model think the
input that it's given given these
weights and biases shouldn't be
classified and essentially what it does
is it's going to multiply these weights
by whatever the input is it's going to
pass them through the layers and then
once we get to the end it's going to say
based on what I know and what I'm
currently set at I'm going to spit out a
number okay so for then so what we would
end up wanting is some format of this we
want either 0 or 1 so it's a binary
classifier in this case but more
generally speaking if we give it a set
of classes we want it to spit out
something that's close to one of those
classes and we'll pretend as if it is
what it actually ends up spitting out is
a probability that it's either of those
classes and in this case it actually
spits out something a little strange
but we'll we'll kind of see how that
looks in a moment and these biases how
do these operate have what's the
relevance of the biases just basically
like a way to correct based on like I'm
totally right but I'm just like off
every time like every time I answer I
got like the pattern right but I'm off
by a little bit and so I'm just gonna
say okay we're gonna add in this bias
term that says you're biased and we're
gonna just correct back towards whatever
you should have been because you got the
pattern right you're just consistently
shifted by one or by some number and so
that's what that bias does is it kind of
just correct you back towards wherever
you should have been this is not to be
confused with a regularizer which says
every time your model gets like too
complex we punish you more
and we won't actually use a regularizer
not really in this case but if you've
looked at any sort of kind of like
introduction to ml usually people start
with like linear regression and then
they hop into the ridge regression and
ridge regression is literally just like
a regular eyes diversion of linear
regression so I say hey my mama gets too
complex you should have a higher loss
for that Oh
loss is another key term we should
define loss is a way of measuring how
poorly our model is doing the higher the
loss the worse we're doing loss in
absolute terms like the number 3 versus
the number 5 it that's not super
meaningful but in a relative sense it is
super meaningful so if I have a loss of
5 versus a loss of 3 versus a loss of
0.01 then the latter is the best version
of our model given our data loss deals a
lot with like a lot of the ways that
people come up with these loss functions
either deal with like like maybe for a
linear function is root mean squared
error so I take the square root of the
mean of the squared errors from some
given knowledge that I have on this and
that would be supervised learning I know
the answers of some of the data set and
I can tell you how far off your model is
on those answers there are other forms
of loss like the one we'll use here
which is categorical cross-entropy wow
that sounds awful when you say that love
and that's not nearly as terrible
sounding as it well as it's not nearly
as terrible as it sounds it basically
just says entropy is a measure of kind
of disorder and in this sense can
actually be interpreted as a form of
error and categorical and cross mean
that we have a categorical problem so we
have several categories we're trying to
distinguish between them and then cross
means across these categories what is
the entropy as where that comes in and
so we're essentially saying given the
categories that we have given all the
like classes that we're trying to form
then what is my like set of mistakes or
like how large are my mistakes and the
reason we don't use like a root mean
squared error is that we're essentially
we're predicting on something that's not
linear so using some sort of
non-existent thing will not say sorry
trying to fit a linear function to
something that is by nature not linear
might result in us not having what's
called a like convex function and that's
that's not good so because we then can
minimize we can't come to some global
minimum
I think it's like the swag says I took
too much adderall
nope I actually don't even know coffee I
just have too much energy um I wish I
could drink coffee because sometimes I
get really tired but oh well no coffee
for me when I do drink coffee I'm wild
leg is going and nothing can stop me I
also just can't sleep so you know it's a
wild time anyway I'm gonna just double
check myself I think I'm right but I
would like to just be sure cool and I
called these other things in my
reference sheet but it's okay
and yeah so sorry yes thank you they say
in defense things do involve a lot of or
the neural networks do involve a lot of
math in the background of how they work
can't help it unless you want to be a
complete black box I thoroughly agree
we're basically trying to avoid the
pattern of a lot of tutorials on the
internet which is like just use
tensorflow or a Karass or get away from
like the math we're trying to show that
the math is not necessarily super crazy
though it does require like some algebra
some linear algebra and a little bit of
like probability theory so you do need
some of those things but after that it
becomes relatively accessible so now
that we have kind of our hidden weights
let's take a look at what it looks like
to multiply things across so what I'm
gonna do is I just want to print out
what happens if I do like a hidden
weights multiplied by our XT so we know
that X dot T is actually now 2 by n
where n is the number of samples it's 2
by 4 in this case because 4 is how many
samples we have and now based on the
fact that our hidden weights is excuse
me 4 which is the number of neurons by
the input dimension this should match up
and we should get some reasonable number
now this isn't necessarily exactly what
we wanted yeah sorry
whoa that's a lot oh right I also put
this up here ok forgot about that we're
also going to actually just do NP s dot
to kind of clarify what exactly we're
using because we don't want an
element-wise multiplication so this is
the actual dot product of the two
multiplication when NP or when dumpee
uses it is actually just like an element
multiplication so we're that's not what
I wanted to do the dot product is
exactly what I wanted to do so I take my
hidden weights and I say okay now
multiply those by whatever our X our
input is and once we have that we can
say okay well if we were to do this for
just one X input we should end up with
something that's just one of those rows
and that's pretty reasonable from there
we're gonna just use the rest of them
just to David in the chat maddening
parents David yeah so hopefully I almost
slipped back into like TF mode where I
was like and any questions and I wait
for there's just nonsensical oh cool so
we can get this dot product it's not
necessarily super meaningful to us but
we do have one now from here what we're
trying to do is then add in our bias so
adding in bias in this case isn't really
gonna tell us a whole lot
it doesn't necessarily offset anything
we don't gain anything from that bias is
already still zero so we're like well
cool with that at zero to nothing so
there's nothing interesting there but we
do need to introduce something called
like a non-linearity to what we are
doing and essentially what that does is
it helps us kind of in the Lindt
non-linearity functions we're using it
helps us distinguish more clearly
between two classes but also in general
the problem we're trying to solve is
kind of by definition nonlinear so the
features that we're using we want to
ensure that we understand something
about their interaction rather than
assuming that their interaction is
linear I believe what I used in my
reference is undirected I know I use tan
okay cool
so what I did was I said and you can see
this here is I took the tan or the 10h
sorry the hyperbolic tan of this result
so what I'm gonna do is I'm gonna say
okay I'm tan H you'll notice that this
comes up all the time when we're talking
about machine learning stuff and all of
its like you know siblings and cousins
come up to so tan H is amongst a series
of functions that are uses were called
activations what we're gonna do is
implement those ourselves so we're gonna
build tan H it's not crazy none of these
functions are particularly wild but we
do have to define it and
so tan H is essentially just sign H over
cosine H and if you're familiar with the
approximation of those or ya
approximations to those functions you
can actually manipulate them using their
non real valued equivalents where like I
think e to the i-x is equivalent to like
cosine of X minus or plus I sine of X or
something along those lines and because
of that property we can actually get a
numerator for tan H that looks something
like e to the various IWI's NPI exp of
like two times X minus one and then
we'll get a denominator that looks like
something like this two times X plus one
and then we'll return the numerator
divided by the denominator now if we
want to see what this function looks
like I'm actually gonna just comment
this out for a second and I'm going to
graph what that function might look like
and I'm actually I'm gonna have to do
this several times so I'm gonna actually
just define a plot FX and this is given
some FX then what I want to do is our
kind of X is equivalent to numpy
linspace
first me said plot effect so they meant
like a thanks it's very very very close
and then I'll do like a plot stuck
figure plots dots on a now scatter of x
by f of X of X that was wild if that was
confusing don't worry I can use myself
as well and then I just need plots so
we're gonna import Matt plot live thank
you oh no that's so close there we go as
plot okay so now I can plot any function
more or less as long as I didn't mess
anything up so now I should be able to
plot FX of tan H let's see what it looks
like so if I run this nothing happened
and here's what our tan H function looks
like I've definitely seen this kind of a
curve oh yeah you'll notice this curve
comes up all the time and activations
it's actually really interesting because
your might be wondering why is this
curve so useful and it's because of this
property where it roughly approximates
what the step function might look
like so let's say for everything less
than zero I'm just at negative one and
everything greater than zero I'm at one
and there's no intersection in between
that would be kind of a perfect
classifier right you're either one or
the other
like I guess binary logic like a circuit
is exactly yeah it's literally where it
comes from it's like this square wave
logic the problem is that that's not
quite continuous I can't differentiate
that conveniently now you might say oh
well the differential of that is X but
it's a little bit difficult to
manipulate so we use this very nice
continuous function that I then
understand the mathematical properties
of now there are several of these
someone in the chat says it looks
sigmoid you're right in fact why don't
we plot the sigmoid function just for
funsies we'll need it later anyway so
let's define a sigmoid of X is I think
we can just return like one point
divided by I don't know it's like 1 + NP
exp of negative X correct me if I'm
wrong but I think that's about right and
if we plot that plot effects of sigmoid
then we can control C out of that I just
forget pi pi plot doesn't really like
control seeing so here's our tan H
function and you'll notice that it looks
very similar to our sigmoid function now
10h is a little bit stricter you'll
notice I can't really switch back and
forth between the two but no I'm sorry I
didn't plot them quite like that but
you'll notice that our tan H function
had a very slick kind of harsh almost
vertical line in the middle
whereas our sigmoid function actually is
a little bit softer it's a little bit
more generous if you will about how it
plots things so because of that it ends
up being a little bit less convenient
since we know we have exactly two
classes but you'll see sigmoid
activation all the time these kinds of
activation functions are useful because
they push things into one or the other
except when it's uncertain and this
uncertainty can kind of be quantified by
where you are on this curve so basically
if you're in an extreme case you're just
one or the other if you're not in an
extreme case then I can't
Sara Lee tell and I might give you some
probability using something called
softmax on where you are now if that
doesn't make any sense or if all of
these sorts of things look crazy then
don't worry you're welcome to go google
them they have very like similar or they
have the same names kind of all over the
place so don't worry about it
but these are kind of the two we're
gonna use tan H for now but we'll use
other ones for later on now oh yeah I
was gonna say thanks to grimwald Smith
saying good night people see you all
tomorrow what did you say thanks for
tuning in and some other people ask
gaming Babic asked if it was similar to
Taylor series they were you're talking
about earlier um so a big night you're
asking I'm not exactly sure which part
you're asking is similar to a Taylor
series for those that are familiar a
Taylor series is a way of a kind of
approximating functions by expanding out
their terms for example like the Taylor
expansion of like e to the X is like one
plus X plus x squared or something and
you can essentially just use the first
two terms and it always works not quite
in this case there is actually like a
complex relation with e of an NH
approximation yeah so this is Euler's
formula oh I got it right so it's e to
the i-x equals like cosine of X plus I
sine of X and essentially when you're
going into like tan H you can use I
believe something like that haven't
quite seen tan H in a long time or at
least the derivation of it how does your
Wikipedia stew how differently oh yeah
so I have a plugin for an extension
I love extensions I customize everything
um my extension for Chrome rehashes or
like reformats the like text over things
as fast native things you can do here
they kind of walk through how like
hyperbolic sine can be approximated like
this and cosine and so on and then you
divide one by the other
so I guess my oil is formula usage was
not quite here although you could
presumably also do that that way and I
think someone said in the chat that
Taylor series is for like polynomial
functions you can use it for polynomial
functions it's often used that way it
also works totally fine as
approximations for like sinusoidal
functions or like lower order polynomial
functions and it works for general
functions as well
it becomes less useful in a lot of cases
where the behavior changes away from
zero but it is generally you're
absolutely correct that it's generally
used for polynomial functions like
usually if I have what getting you to
the X or I have like some weird
polynomial and I just want to like
approximately its behavior you can do
that you also have like the Maclaurin
series which is like a specific I always
forget if it's specific case of Taylor
or the other way around but one of them
is centered only at zero I believe
Taylor centered at zero I may be
Maclaurin is centered at zero Taylor has
other centers and essentially because of
that you can like approximate the
behavior of a function at any input and
then you can generalize that to
multi-dimensional inputs and also to
higher-order functions so you can use it
for like trigonometric functions as well
it's not necessarily as like informative
at all times
but it can be and then as Al Gore said I
never thought I'd say that Mary said if
you use like a signal processing or
systems class they use like a Fourier
series which is to approximate periodic
functions and Fourier series and Fourier
transforms have all sorts of beautiful
properties such that like random noise
is really high frequency and if I do a
fast Fourier transform on an input
signal of amplitudes then I can kind of
ignore the noise because it's going to
be very high frequency and then the at
least usually noise is high frequency
compared to the actual signal that I'm
looking for and I can use that to clean
signals there's all sorts of cool things
you can do with those there's many
different kinds of series that can be
used Fourier series just happens to be
very very commonly used signal
processing is a very useful course to
take I actually had to take it last
semester for bioengineering
so I ended up absorbing lots and lots of
information I'm not a class that CS
undergrads typically take no yeah it's
not required for CS it's required for
bio either for all of our engineering
concentrators actually maybe not all
engineering I think just bio me so yeah
hopping from there we now have this kind
of like hidden we'll call it results if
you will and that's equivalent to just
whatever our hidden layer spits back at
us now from there what we want to do is
then say okay well what is our
put results well what do we do with this
these hidden layers pieces and I just
kind of want to like go clarify which
version I used okay I use a sigmoid for
that cool so then here when we want to
see what our output and I'm gonna
actually just call this a prediction
this is going to be I'll do it the same
way that I did the previous one we're
going to again do a dot product and
again of its weight our output weights
but this time it's going to be of our
hidden results and then we're going to
add in the output bias and here we can
print out what these predictions kind of
look like calling it prediction is a
little misleading because we currently
have not set up a full neural network
but you'll notice this is slightly
different from what we have before but
not terribly distinct and then what
we're going to do is apply the sigmoid
function to that and so again it's
nonlinear eyes activation and you'll
notice that it also eliminates that like
zero term here sorrow sorrow LCS says
that ice cream will go about two hours
two hours and 15 minutes going through
my streams in general pretty much nails
it's pretty much exactly what we're
gonna do and hopefully we'll get to the
end where were light and that was a ton
of work so next time we're gonna do the
same thing we did in like 50 lines code
in like one using tensor form but then
it's kind of the end in sight if you
will so yeah we've applied the sigmoid
activation function to this and you
might go okay that doesn't really look
like a great prediction but sure and
what you'll then want to have the
ability to do is say how do I like get a
probability of these and probabilities
should only range from 0 to 1 and so on
so what I'm going to do is called a
softmax function what this does is it
says okay let's take e to the X
oh that's just not that's how you math
write that e to the X is our numerator
and our denominator is actually just the
sum of all of those so I'm going to just
do the MP dot sum of MPI exp of X and
the reason that this is really useful is
that we can then say okay this has
specific properties so given that I was
given some X I raise it to the I raise e
to that power so I know no I
now know that I can't have a negative
probability so that's one property of
probabilities that I'm gonna keep and
then the other one is that because I'm
dividing by the sum of all of them now
each of my probabilities has to sum to
one so or sorry all of my probabilities
together now sums to 1 and that is
another property of probability it's the
kind of like law of total probability is
that we're going to have all of our
things accounted for I can't have like
well the probability of a coin landing
on 1 or let we'll say a like 2 2 is
strictly two-dimensional coin landing on
one side is points 4 and the chance of
it having on the other side is 0.4 0.2
percent of the time it just disappears
that would be non sound school well and
depending on kind of what kind of coin
maybe not but we're gonna pretend like
that would be nonsense cool so now if I
apply softmax to oh and the problem with
softmax in this case is that it actually
doesn't quite do what we think it does
because it's not built to apply over
several different predictions so we're
gonna just have it apply to the first
prediction and we'll print out both and
now what we see is we have roughly even
distribution of like all of our possible
predictions for this now our neural
network is set up a little weirdly in
that this should only be either 0 or 1
but you'll notice that we have four
spots we're gonna kind of just ignore
that for today we'll set it up a little
bit more formally next time when we're
using the actual implementation but this
allows us to kind of get this roughly
right I don't think softmax works on
prediction Rob and might not entirely
sure ok so you'll notice that if we run
it on all of the predictions we're now
dividing by the total sum and that means
that the probability of each of them is
now relative over several different
samples that's not quite what we wanted
to do so we're not gonna not gonna
necessarily do that also I think this
might actually be the wrong
dimensionality there we go that's a
little bit cleaner but again we just
want that first kind of pairing but
you'll notice for each of our inputs we
now actually should reshape prediction
in order to get this mostly right so
calculating softmax in this way is a
little strange but if we look at our
prediction
it should be a little bit more
meaningful to us as we have now reduced
it based on let me see if I can make
that a little bit clearer we've now
reduced it based on it I figure out what
I was saying well oh we reduced it to
only calculating on one input the
classic brain fart
yeah I was just like and the words are
out but yeah now we know we're trying to
say now we're all good and so that is
our actual prediction on that but we're
going to keep the sigmoid prediction
because that is what we're going to feed
into each epoch of this layer and we
call this kind of part of this the
forward pass and so essentially what
goes on here is this is where we're
going forward through our network so if
we imagine it as Colton kind of brought
up from left to right where our
beginning our input is on our left and
our outputs on our right then the
forward pass is just the forward part of
that now that would be great if we could
just run this over and over again and
that would delight like kind of just
make everything work sorry I put the
forward pass label in the wrong place
but the kind of problem is that a neural
network has to adjust itself and it's
modeled on human learning so if I do
something really silly like maybe stick
my hand in a fire or something then a
human who's going to say oh that hurt I
don't want to do that again right there
were some set of stimuli that I now have
matched to my head and I'm gonna say oh
if I see a literal flame I'm not gonna
stick my hand to that anymore
now you'll notice that this actually
analogous and the analogy is pretty
solid from like humans to machines in
this case where I don't necessarily have
a general understanding that I shouldn't
touch things that are extremely hot and
I can't always predict what things are
very hot but over time we learn some
features of heated things they tend to
change color for example and look white
or yellow or red now once I've started
to learn those features over time I
might say hey that thing looks like it's
really hot now we can't look at
something until it's temperature that's
crazy but we can kind of use proxies for
temperature in order to accurately
predict the underlying state maybe the
latent state if you will
and because of that kind of ability of
humans we wanted to mimic that in
machines and so essentially what's going
on here is we want to be able to say
okay how do I give the like out response
to a machine and then how do I get it to
adjust its fur like future behavior
based on that response so those are the
next two parts of the kind of neural
network ideology and the first one is
kind of that ouch like that was wrong
you messed up and the way that we do
that is we take our predictions and we
say how far off where our predictions
from our actual answers and then we're
gonna quantify this in that categorical
cross entropic way I'm gonna just kind
of copy that formula because I don't
want to recall it from memory I'll try
it from memory and then we'll just copy
it so it works
and essentially what that's going to
tell us is okay this is how else that
was how much it hurt it's often referred
to as cost can also be referred to as
like a log probability a log loss and
those all mean slightly different things
and in our scenario they happen to be
the same because we're defining our log
probability and like a slight
manipulation of that log probability as
our error so we're going to say let's
see log probability this is equivalent
to its MP dot multiply of NP dot dot dot
log of our hidden our prediction yeah
this is gonna be is gonna be good
why not t comma R naught comma that yeah
comma here of some thing yeah sorry we
made it so far and how we'll see how
close that first part was I'm gonna just
kind of copy that over and paste it and
I'll paste it below so we can see yeah
not terrible so it's not quite this it's
actually just here and then we can kind
of go from there plus MP dot multiply of
1 minus our Y dot T and MP dot log of 1
minus our prediction thanks to Drew
keaney's for
kindly putting in I think we what your
suggestion was last week
the pattern recognition machine learning
book oh yeah yeah Bush the bishop book
is quite good we use it in our class for
181 or for our ml class and that's great
it does a really good job I think it's
pretty awesome
yeah there are all sorts of like reading
and resources out there you'll notice
that a lot of like tutorials kind of
skip over the math or if they go into
the math they go like way into it we're
trying to go like half way and kind of
like talk a little bit about the math
but also skate over some parts of it if
you don't agree with that methodology
sorry our bad and I guess and
essentially what we're just trying to do
here is give you an intuition for how
this works and give you some of the math
and some of the like names of things you
know what's Google so essentially this
should print out for us the air nice
thanks wants to do a cell crowd says a
long time YouTube follower finally
register on Twitch love the vid skip
with a good work guys thank you that's
awesome yeah thanks so much for tuning
in on Twitch and so you'll notice I I
always say you'll notice and some people
do some people don't I probably don't
but essentially what happened here is I
now have printed out this quantified
error I'm gonna go into the math of this
a little bit so we understand somewhat
where it came from but I'm not gonna
like it won't be required that you
understand all of this math you're
welcome to go and kind of figure it out
on your own but essentially what's going
on here is I'm saying okay I want to do
this multiplication of the log of our
prediction by the actual prediction so
the reason that this kind of ends up
working is we're we're trying to
quantify how far we were from our
prediction and we don't necessarily want
to use a linear version of that so logs
and logarithms and their properties help
us kind of take care of this
you'll notice that multiplying like NP
dot log of prediction by Y of T is
essentially taking prediction to the
power of Y T if you're familiar with
your log properties then we are
essentially taking our prediction and
putting it to the power of
whatever Y dot T is and in our scenario
that's not intuitively useful however if
Y of T or Y dot T was actually indicator
variables or one hot encoding if you
will then this would actually be very
helpful because every time you raise it
to the wrong place it's not going to
it'll have zero won't count for our
error but if you raise it sorry every
time you get to the right place one
capper error but if you raise it to the
wrong place it will and that's that's
essentially what we're looking for here
true Keaney is asking do I understand
this this answer is not that well I'm
looking up some stuff on the side like
the softmax function is better yes I
guess just kind of see if I can pull
along but I know that if I keep asking
too many questions we'll be here all day
so yeah you're one ask away
we're getting we're good good and I it
helps having template code to copy over
oh we got a just coming over awkwardly
we changed our conference room set up so
it takes a couple person anybody is that
while watching in a zone from overseas
video this is the professor of states 50
right here yeah nice to see Morrison and
Manthey stuff today so yeah we're going
to smash some pretty decent math going
on we haven't even hit gradient descent
but we will shortly thank you and says I
think someone asks why vlogged why not
root mean squared error and so
essentially because RMS is going to say
and actually there are other ways of
manipulating this such that RMS would
work with what we're actually trying to
do is we're looking for across kind of a
general categorization how would our
error be quantified we use like a
variant of log probability to do that
root mean squared error would be based
on basically the kind of linearity of
the relationship or that I could use a
basis function to transform it into a
linear basis and from there get some
sort of linear relationship and so we're
kind of just assuming that we don't have
a linear relationship and there's no
convenient way for us to get one now
that sounds silly given that
or you can actually linearize that you
can get it to be linear but in the
general case where we're using a neural
net and let's say I have given oh but in
the context of what we have an image and
I want to classify this image by what
stream it came from then it's actually
really difficult for us to say what
features are important and how they
might be related it's really tricky for
me to say okay well here's a basis
function that transforms all of our
features such that every class is
linearly separated that's really
difficult to do and the more complex the
more features you have and the more ways
in which they're allowed to interact the
more difficult that becomes usually and
because of that we're kind of operating
on the assumption that we can't use
linear or strictly linear things in
order to get this to work great question
and that's essentially why it's the
intuition at least the map itself also
explains that if you were to use a
linear optimization function or error
function or cost function here that
linear function would allow us to
balloon in either direction we might we
might for one get stuck in like a local
minimum on our loss function that like
doesn't not only never converges to a
meaningful value but if we were to hop
out of that local minimum or it really a
local minimum then we might just like
shoot upwards or shoot downwards or
something poor poor behavior on our part
so what we're trying to do is have a
function that has some global minimum
that we can converge to by hopping to it
somehow now if you go like why what is
it was this minimum you're talking about
what I mean is if you were to plot this
error function as a function of all
possible predictions that could be made
or actually as a function of several of
its parameters then you should end up
with this in the two-dimensional space
you'd end up with like a curve it might
have several local minima and then
settle into a global one or have a
global one here and settle into local
minimum things like that and essentially
what we're trying to do is hop around
this curve and find that global minimum
because that means we have minimized our
error based on the model that we have
and the data that we have and that's
essential that's that's essentially
saying we understand our data it's it's
kind of what professors are looking for
when they test you presumably is have
you arrived at this kind of like global
minimum or are you stuck at some local
minimum where
we test you you'll fail that's it's a
rough kind of analogy but it's the right
idea so we want this this number or the
number that I've highlighted here to be
as small as possible the absolute number
is not necessarily meaningful um it I
think when I was showing Colton kind of
what we were doing before the stream it
was like a three this time it's point
seven we've got it under like point two
yeah it's not super helpful to know what
it is in absolutely terms point seven
doesn't seem that good it's not super
great and intuitively it should start
pretty high and then go down yeah
and if we change the way that we like
initialize certain layers so let's let's
switch this to like a normal layer it's
our number should change actually went
up so a normal distribution was much
closer we can kind of infer not really
not not rigorously and phurba kind of
intuitively infer from that a normal
distribution was closer to the actual
distribution of the data so it was
closer to the river sorry not the
distribution of the data the
distribution of the ideal weights and
the kind of presumption is and you can
mathematically show this that the ideal
weights will minimize this error
function and that's kind of a circular
argument because the way that we define
how we minimize our weights or the way
that we define how we get our weight is
by minimizing the error function so it's
two sides of the same coin to argue that
one is the causal together is a little
weird but it's it's really that we just
want to minimize this number and this
you'll I guess if you look at how this
works it's a little bit tricky but
you're essentially multiplying something
by or you're multiplying a log by
something so you're raising predictions
or the whitee power and then taking the
log of that and then over here you
inverted everything so we took this one
actually kind of in furs or assumes
something about our categories you can
generalize this relatively easily but
for now we're going to use just kind of
this one as if we only have classes
either 0 or 1 which we do and then we
have this 1 minus prediction and this
essentially says ok for the number of
times this is like the inversion so if
this was one hot encoded now every wrong
answer is now a 1 and every right answer
is a 0 so there's a single zero per each
of these
and then in this NP log anywhere that we
have a prediction we should have had as
close to one as possible in the right
place but now we have a zero presumably
and we're again doing the same thing and
so these are kind of just went ways of
measuring different like errors that
could have been made in this and then
what we do is just add them together
because they are linearly edible and a
staff ignite says is a really
sophisticated way to find a global
minimum generally we just do a first
derivative take second derivative and
you're done and that's right for
relatively simple functions so in this
case I don't know if there's that's
called a closed form solution
essentially then I can just do some math
and find a solution for it a lot of
machine learning problems that's not
possible the math behind it is too
complex
we actually maybe don't even understand
the math behind it and because of that
we actually can't find a global minimum
there is no closed-form or I'm
forgetting the term for this but it's a
like mathematically derive a bull
solution I can't remember the name of it
for the life of me but it's literally
only because I'm talking in front of
people now it's not anywhere else I can
tell you up top man and so essentially
what's we're trying to kind of assume
here is that in the general case where
we're actually using a neural network
you actually don't have a closed-form
solution of any kind we just kind of
know that this sort of thing worked in
kind of smaller cases like what we're
doing right now and it expands and
generalized as well to more complex ones
so it would be very very tricky and
actually possibly mathematically
intractable for us to try and find a
global minimum using only math which is
I think very wild it's one of the cool
things about ml and neural networks in
general it should strike you as very
strange especially if you're coming from
like a high school or collegiate you
know linear algebra class multivariable
calculus class essentially all of those
problems are tractable or they're not
intractable and you just say oh it
doesn't exist move on with your quizzes
or whatever the problem is that a lot of
those do not exist problems are really
useful in the real world and because of
that it's extremely important and it's
one of the motivations for ml to try and
create machines that can solve those
intractable problems that mathematically
we don't understand but they are so
emergently expecting them to figure
yeah more or less now which is why this
is some one of those things that just
blows my mind about this now I did
promise also that there is a backward
pass if you will we have to communicate
this new information that we have
obtained about our error back to the
hidden and output layers we need to
change their weights so essentially we
need to adjust our machines behavior
based on what we now know this is one of
those things that tends to be really
tricky for people and I'm gonna try and
break it down into something that's a
little bit more tractable but if you're
lost or confused don't worry it's it's
difficult it's something that you should
probably like sit down and kind of like
play through but it's called backwards
propagation or back prop and essentially
what it means is we need to teach our
like brain what we have learned and so
essentially what goes on I'm gonna just
kind of copy the code so that I can
spend more time talking you through it
is I'm gonna just copy all of this
actually this is from my template code I
promise I wrote most of it and
essentially what happens here is we have
this two-step process going on and what
we're gonna focus on first is the
gradient part so if you're familiar with
mathematically what a derivative is a
gradient it's kind of just an expansion
of that concept into higher dimensions
so a gradient says let's imagine that we
have some surface maybe like a bowl so
just a particularly convenient surface
and I have plotted this in some way or
even in higher dimensions they have ways
of converging to a point but in a bowl
it's pretty intuitively understandable
the center of the bowl is the
convergence point now if I take the
gradient and let's say I started
somewhere on the wall of the bowl I want
to dive down into the bowl so if I take
the gradient what that tells me is it
points that way kind of up the edge of
the bowl it tells me the point of
steepest ascent so it says hey go that
way and that will take you to the
maximum so if we had an inverted bowl it
would take us to the top center but if
we have a normal bowl we want to go down
and that's where the term gradient
descent comes from I have a gradient and
it points upward I wanted to send down
that gradient so if I take whatever my
weights are calculate their gradient
based on some sort of parameters or
inputs that I've given them then I now
know okay I don't necessarily know
anything about the general shape of my
surface or whatever I'm on it could be
like a valley or a tube or something but
I know that wherever I am if I go that
way I will get closer to some version of
the bottom now I say some version
because let's assume we have a big bowl
and then like a little bowl that's
connected if I'm in the little bowls
wall it might make more sense from
wherever I calculate this that my
gradient points me down to the middle of
the little bowl not the big one and
actually you'll notice that it has to by
continuity so there's no way for me to
just point at the dead center of the
other bowl even if I could I can't get
there by continuity I'd have to jump and
you might say oh well then that's
impossible I can't I literally cannot
get to the global minimum and that's
where stuck has to City and randomness
are super useful having a like
reasonable variance a reasonable kind of
will say randomness in your data allows
you to do what are called like little
hops you can hop over that Ridge between
the big bowl and the little bowl and
you'll hop into the big bowl and from
there you'll then descend to the global
minimum
now that is intuitively how that works a
lot of times they're not quite so easy
for those so to achieve those so there's
that and it's definitely a difficult
thing once you actually sit in the math
but I think that's a good intuition for
how it works that McKnight says yeah
what's with the upside-down Delta with a
dot product so that is actually a it's
called an habla it's an upside down
Delta looks like a triangle hashtag
Illuminati and essentially what it's
doing is that says what is our gradient
that is usually used as the symbol for
the gradient function and you can think
of that as just a multi-dimensional
derivative of your function so
essentially what we're doing here in our
code is we want to calculate what the
gradient is for our weights and for our
bias and we want to then descend that
gradient so we want our biases to be as
small as possible we also want our
weights to be as not necessarily minimum
as possible we want them
to converge to something that is
meaningfully useful this gradient is
calculated using the log probability as
it's kind of like understanding and then
what we end up actually doing is
multiplying by like the error of what we
have so essentially what this does is
this expands back out our error function
and then uses it and its gradient to
then calculate what our new descent
should be so these evaluate two numbers
since we fill in all parameters and then
we subtract those numbers multiplied by
some learning rate from the original
parameters that we had so that is called
a backward pass this all in total from
like here to here possibly counting your
error is really the backwards
propagation so we're propagating all of
the information we have now backwards
through the neural network this is why
we stuck to a single layer neural
network was it makes this very easy I
don't have to do any like weird
syntactical tricks to get this to work
and even in here there's a lot of math
going on I'm not I don't think it's
super important to understand the exact
math of why this works
I think the intuitive understanding is
more general and more easily applicable
there are plenty of Papers written on
why this works for this kind of neural
network something that we'll get into
pretty much right after this is like
switching to like a specific
convolutional neural network each type
of layer actually has its own way of
doing back propagation or at least a
different interpretation for what that
means because our gradients are
calculated differently so those sorts of
things we'll have to go and deal with
and you know we'll go from there so
essentially what we now have is the kind
of intuitive understanding of a single
layered neural network intuitively if we
were to take this and loop it it kind of
just over and over again we should be
able to minimize that error so for I in
range I don't know why I was starting up
at the screen I was like out of school
for iron range we'll say a hundred will
say ten to start we should see this
error go down now whether or not that's
true ya know well
find out of course it's not too fine
because I copied code from a previous
version of this you would think I would
name my variables the same thing every
time
probably not know but I just don't it's
not at all how I seem to operate I get
like the concept right I think it's the
problem with knowing too many synonyms
it's like you just don't that's why
people say don't use single literal
variable names exactly ran into the
issue like today's J do they use K every
single time yeah so I'm gonna just kind
of like debug my own code a little bit I
think that we're about right now nope
10:33 says he's 50 and three blue one
brown are getting me through university
right now much love you guys like other
people in the chat are saying three blue
and Browns pretty good I'd never seen it
it's really I'm not know people are
saying sounds pretty cool or an
Intendant was saying that it's got
really good math visualization oh that
is essential I'm a visual learner myself
so that's why I kind of explain these
things like these weird and hand
versions of of a visualization but oh my
gosh I'm losing my marbles hello Ashley
good to see you it's like I just never
was consistent between like plural and
singular for all of these so give me a
moment to kind of fix my like weird
little bugs this is incorrect of course
it is Oh cuz I fixed I'm gonna say I
have a weird bug when I did this
originally I was coming come and it'll
work and now I just need to debug kind
of our like matrix dimensionality oh
well oh that was what you would happen
before we start it every time I think
cool and then I just need hidden wait
this is wait hit our hidden weights
there we go
okay now everything I think I'll put
weights I think is everyone's sitting
there like really dude you couldn't name
your it's the same thing it's like yep
you got it cool I think that gets it oh
and then I just need an ADA so ADA is
kind of a weird little symbol we use it
to decide how quickly we're going to try
and learn and essentially if we try and
learn too quickly than we might descend
and skip over things that are meaningful
of course they're not broadcasted
and that would not be great for us in
the kind of version of like what was I
gonna say where we like reduce ADA such
that we can kind of like hop down a
gradient ooh really really slowly if you
have too small of an ADA then that's not
super useful for us so you kind of have
to be like very very careful with like
how you actually control you're learning
the learning rate and that is usually
what ADA is now that learning rate
simply refers to we now have our
gradient how much of that like we don't
just descend by like an arbitrarily
large amount every time we wants to
descend by like a little bit in that
direction and the gradient is really
just a direction at that point and then
I'm going to actually switch over to
running the template code that I
provided I promise it's not particularly
different now it's a little bit
different goal and then what we're gonna
do is go through this a hundred times
and print the error every ten and you'll
notice that our error doesn't go down a
whole lot maybe we'll increase that to
every like 100 or so and then we'll put
this out a thousand and see if we can
get this to show us something a little
bit more useful but essentially what's
going on is we eventually converge to
some actual error you'll notice that our
error should go down roughly starts at
like three point three four goes down a
little bit and then it kind of drops to
like this two point eight three or so
and you'll notice that it actually like
went even lower than that that's not the
global minimum of error but it is kind
of like our we like bounced around a
little bit and the reason for that is if
you imagine that we like got to the
bottom of our Bowl
but our learning rate was a little too
high then we might like ping across the
bowl right and never actually hit the
global minimum and in this case it's a
trivial data set so we we hit our
minimum pretty quickly and then kind of
just like bounced around noise more or
less and because of that we now have
this error rate and this is our neural
net
now you can this is not a very complex
or interesting neural net it's not super
wild there's all sorts of things about
it that are a little bit weird we don't
do any sort of batching or like
controlling for anything we don't
normalize we don't lose nothing
interesting in terms of like complexity
that we do here we've just kind of
implemented a very simple neural net to
demonstrate a lot of the concepts if
you're interesting and you're interested
in building more like complex ones from
scratch I would recommend using kind of
like crazier maybe like build a class so
you can maintain interfaces and have
better programming practices in general
I see a bunch of people were like why
aren't we using functional programming
to do all the math pythons fine it's
sufficient we we could switch to
functional programming and also do math
the math will not get any easier but you
can make it look syntactically a little
bit nicer what are they what do they
have you using the ml course here you're
allowed to use pretty much anything
you'd like they basically suggest Python
oh yeah okay and so in our ml of course
we actually just pretty much stick to
Python I would imagine that there are a
couple people who use C++ just for like
speed I don't think anyone uses a
functional language not to my knowledge
but you could the convenience of Python
is there are so many libraries that are
at a high level so if we're like really
rapidly debugging things and we're
trying to just get it to work then we're
not going to be implementing all of this
we will just use a numpy library or
sorry a like tensorflow library or a
scikit-learn library because the kind of
reason for understanding this is so that
you can improve your machine learning
but the reasoning behind like the actual
kind of or like the needs to actually
build everything specifically not so
much a lot of times those will work
normally crazy watermelon asks why it
says convolution and that's a great way
to switch into our new version of this
or not anakin version of this oh my
words are failing me entirely today too
many presentations around presentation
numbers like two out of three today
telling today my first one was in my cs
big data systems class i had to give a
small presentation there and like i'm
just wiped so there's a lot of talk to
you what's your next present and then my
next one is on cybersecurity so we
we'll be hacking some people that's
already been hacked they don't know it
isn't you in a prison some people with
their passwords pretty much it'll be
like a and it is this year card and
they'll be like that's my bank yeah no
mostly kidding it's all with their
consent okay so yeah we have implemented
kind of Li like dense parts of a neural
network now we're going to pivot into
the why are we calling this
convolutional there's no images in this
and I'm glad you asked
that's exactly what we'll be doing so
we're gonna kind of pivot entirely away
from the like tensor sorry neurons and
like dense like neural networks and
where things are connected and
backpropagation and we're gonna switch
to okay you have a neural network
architecture we have these dense layers
I want to use this to classify images
and every pixel position of an image
represents a feature so let's say we
have a really small image like the MNS
ist data set M missed now they're only
28 by 28 pixels and they're not
multi-dimensional like depth wise
they're just single channels so it's
literally just 28 by 28 use individual
pixels as features yeah and that's
essentially because every pixel could be
a feature now I guess sounds crazy it is
in order to train the network on just
raw features which is what we actually
did last time whether k-means stuff but
we weren't using a neural network is
wild 28 by 28 is what it's an absurd
number and we're not going to do that so
that's crazy so instead what we do is we
reduce the dimensionality of our of our
images while still retaining all their
key features so we're going to walk
through some of the functions that do
that now I'm going to delete all of the
functions to that from this file so that
we are aware of how to build them I'm
going to keep load data so that we kind
of like keep everything sorry I also
never got to using relu it's another
activation this just takes this
basically just says it's rectified
linear unit and this says everything's
linear unless you're less than zero then
just set u to zero and so with
saying is like given an image or a
flattened image then take anywhere that
it's less than or equal to zero and then
set it equal to zero I guess less than
zero would be sufficient but we'll use
less than or equal to for funsies so we
know we also have a way to load in data
and we're gonna use that method and
essentially what that does I don't know
why I have that there um what we're
gonna use with this load data is we're
gonna load in our images just the like I
think this is the luma filter part of it
I essentially converts it to grayscale
so what we're gonna do is I'm gonna load
in our images here so I'm gonna comment
out our entire neural network oh well I
was going to there we go our neural
network has now been commented out and
what we're gonna do is play around with
how do we look at these images so let's
load our data I'm gonna give it a data
directory
I'm presumably gonna give it a data
directory and essentially what we're
gonna do is take a look I just use the
wrong keyboard shortcut for that we're
gonna give it a data directory we're
gonna pull all these images they're
gonna be numpy arrays we're going to do
plot dot figure plot em show and plot
that show and I just want to see what
does like the first image look like now
if you were here last time oh of course
that doesn't why does that not work oh
okay there we go if you were here last
time you'll notice that this essentially
of course it's not defined all my back
engines are gone
I said it like that did you vote - Peru
hello dizzy votes thanks for joining us
all the way from Peru
well to Austin I'm probably gonna need
another couple of things so from he'll
import image I think I think that's all
of them we'll find out shortly if that's
not true yeah that works cool and that
is not what I wanted
what into it I have highlight that No
it like took out all the way and that's
yellow
yeah then like what the heck let's go
take a look at what this is doing so
load data says image dot open of F
convert L maybe L a does that work I
might throw an error dimensionality wise
yeah it's bad dimensions man that should
be right nope they look pretty great we
look weird no we're having great is a
way to do that Andres is like yeah
that's not that's not black and white I
I agree um why is that not black and
white that shouldn't be black and white
maybe we could convert it into gray and
just see what happens
l should be the luma parameter also I'm
wildin cuz I tested this early hmm let's
think about what I'm doing oh sorry
sorry I'm silly
I forgot to map into the gray color map
that's right thanks what color I'm gonna
see you des earlier cousin no use yeah I
see it covers a little wack I thought
Mosley's like a green orange kind of
deal okay pretty close I think is yellow
yeah that's not too bad and now it's
black and white there we go
so essentially the values that we got
are not like actually grayscale you'll
notice that they they can be mapped to
gray but the defaults is not in the
right like dimensionality or than sorry
not the right range for gray scales so
we got that abomination yeah so here's
our grayscale image and this is what the
original image looks like now what's
going to be really useful to us is to
say okay that's what the original looks
like let's let's try and pull some
interesting features out of this
original image and so now is it just a
matrix instead of RGB is it just a
matrix of the floats from 0 1 0 to 1 or
I think this one's probably like mm 0 to
255 the color mapping just kind of like
rescales it so that it can be mapped to
gray that's that's the essential piece
that I forgot so what we might want to
do is we might want to take our first
image I'm just gonna ileus that so yet
and now we can just do this and we want
to take that image and pull out
interesting
parts to that image interesting features
if you will now it's not necessarily
intuitive to us which features are
meaningfully interesting but generally
there are some features that are
interesting enough so let's say that I
designed some filter we'll call it like
our our line filter and this is going to
be a numpy array that is going to
consist of a couple of rows that are
like 0.0 dot and so on and I'm going to
just kind of copy and paste these so we
have now four of them
let's make eight and because our images
are pretty big so we have eight of these
zeros and then I'm going to put like a 1
and then I'm gonna put 8 more zeros and
this is this is kind of one row of our
filter and then I'm gonna just kind of
copy and paste this a couple of times to
get a roughly square filter but not not
entirely so this is a 17 length filter
currently I think that's 8 so here's
this and now it's a square filter I
believe cool so that was a lot of a lot
of manipulation for a very simple
looking array you'll notice that it I
believe is square you know we can double
check 1 2 3 4 5 6 7 8 9 10 11 12 13 14
15 16 crap and I think we just need one
more actually I can just paste cool all
right I think that that's right let's
print out our just line filtered shape I
don't think we Oh we do still show the
image sorry I'm a mess there we go and
this is now 18 by 70
oh I just miscounted well not a shocker
okay so now if we print this out is a 17
by 17 image or sorry I guess like pixel
set we can actually even like we can
plot it which is kind of cool I think
what's kind of cool and essentially what
we will then see is just a block of
just a single white line right and
essentially what this does is it says
okay we have a single just bar down the
middle it's a square filter but it's
just a single bar and this is looking
for white bars now if I wanted the
opposite of that I could just do this do
that and now we have the opposite a
single black bar how is it why is it
zero being white if the zero normally is
black so now I've just flipped them so
now anywhere there was a zeros now at
one and they where there's a one is now
zero so black is still zero and white is
still one so highest intensity is still
white lowest intensity is still black
we just we flipped what our filter
actually looked like by doing this one -
so now - plus ya know sorry so now our
highest intensity things are black or
sorry all right high intensity things
are still white but they're the majority
of the filter and then everything is
below Andre says typically you average
the values so typically is a very strong
way of phrasing that in this case
there's not necessarily a whole lot of
typical in this there are ways that are
typically taught in terms of like how to
get you familiar with this I just am
going to cover the like convolute the
pooling part of this after so in this
case we're using a filter to look for
particular features not necessarily to
combine things so we're not really
reducing the dimensionality of our data
but we are saying hey if something has
this characteristic then cause this
behavior otherwise caused this behavior
and hopefully that makes sense there are
actually two parts to what we're about
to talk through the first part is
convolution so how do we put a filter
and push it over the image entirely and
then the next part is how do we like
pool them oh I see so I'm just clarified
this is me talking to Babak sorry I
didn't read everything I just kind of
like read parts of it I got you but yeah
what I said is still true but we're
still going to go through and split them
apart into convolution and then pooling
by I see so others was actually talking
to how to grayscale a pixel you can
average all the values that is one way
to do it there are many ways of doing it
I just use an inbuilt method because I
didn't actually want to do that sorry
for misunderstood
uh-huh cool so we now have this kind of
like enormous glorious filter and what
we want to do is convulsion volution in
signals processing has a slightly
different meaning and usage that
generally means like and I'll just kind
of show you is it's easier to
demonstrate via picture did you do
a convolution of signals and with an
image this becomes fairly easy to see an
illustrative one is probably this and so
basically if we convolve two signals
that kind of I don't know what color it
is but the one that I've now traced with
my mouse that's the convolution the
thing being involved or sorry the thing
being convolve with is here it's this
box and then this triangle is the
convolution or the sorry the operator
the filter that it is being convulsed
and I was the weirdest phrasing it could
have used essentially this is a
depiction of like a signals processing
convolution and it's very similar in
concept but we're actually ending up
doing is creating more of like a cross
variance matrix and kind of moving from
there but I wouldn't worry too much
about that convolution is just kind of
convention it's the name of the term and
we're gonna create a function that
allows us to do that so let's go ahead
and build our define convolution our
convolution function and this is going
to take some image and some filter and I
think filter is a safe word or a
specific word in Python when I can use
okay Asli says I want to see like new
quote of the day on Nick's laptop yep my
quote of the day is all over the place
if I open up like a new terminal window
you get a new quote of the day well
sorry they're always like weird quotes
they're from cows a so I take no
responsibility for what comes up on this
okay all right so we essentially now
want to say okay we want to convulse
something and this filter is going to
require us to kind of understand like
given like where the filter currently is
we're gonna multiply it across
everything that exists in that space in
the original image we're gonna take the
sum of all of those and that is going to
be our result of the conv
Ellucian now if that sounds kind of
similar to multiplying across by all the
weights in something like in our neural
net then you're absolutely right that's
essentially the analogous thing that
we're doing here our weight is
essentially our filter and the image is
the data and so we're multiplying each
like representative parts the way that
that gets done is a little less simple
than it used to be in this like a plain
neural net and we do usually add a bias
term which is why you also have back
propagation for the convolutional layers
in your neural net we're going to not do
that here we're going to just kind of
motivate what a convolution can look
like and how different convolutions kind
of change our image but be aware that
the actual convolutional layers that are
used have a bias component which we're
not going to use or demonstrate and when
they're part of a neural net they are
also part of the back propagation step
that goes on but I don't think that it's
educationally useful to implement all of
that I also would not be able to do it
off the top of my head I would need to
kind of like see which ones are kind of
around and see just kind of like what
the math is behind it or I'd have to
write it down and kind of derive for a
while so we're going to just go through
simple convolutions so now what we're
gonna do is call this image height image
width
sorry this also assumes we're using a
single image which is again not
necessarily convention or wouldn't be
and we're also assuming single channels
so this wouldn't this extends fairly
easily to multi-channel images but we're
not gonna do that for now and then I
want the filters height and width which
I'm going to just call H and W and
that's filled on shape there's no error
handling here it's funny to me that
everyone freaks out in C code if you
don't error handle things but in Python
no one cares and that shouldn't be the
case you should care a lot about how
this is error handled there could be all
sorts of bad inputs to this but we're
gonna ignore them for the sake of
brevity and clarity of what we're
reading and what I want to do is I want
to know what like the output shape is so
I'm gonna do like output is actually
just NP zeroes I think it's sy shape
dang it and the shape of this isn't
terribly crazy to to kind of go through
but what we're gonna assume is that it's
really just like image height - height
and then plus 1 by I'm sorry by image
width minus width plus 1 and the reason
for that is like once we get to the end
there are other ways of handling this
but I'm gonna just kind of like ignore
it so this will reduce our
dimensionality kind of by accident by a
little bit but not not really so we're
gonna we're going to say that this isn't
a dimension reducing step because it's
not but it does kind of bring out some
of the features so in that sense it's
kind of like a feature reduction almost
like there are noise features that don't
really matter to us or just noise and
we're kind of trying to eliminate that
and so this is the shape of our output
and it's just a bunch of zeroes for now
and what we want to eventually do is
return that output and then what we're
gonna do from there is we're going to
iterate over our image now this is my no
means the most efficient way to do this
but it is illustrative of the point so
I'm going to use I J I as our row J as
our column for J Earth I looked at it
some thought to myself was like I and I
said J I meant I for I in a range of and
again this is going to be this kind of
like parameter here so we're iterating
through all of the rows of our output
and then we're going to start all yeah
all the rows of our output for J in the
number of columns in our output and
that's going to be this parameter and
all I want to do every time I get here
is I wanna say okay output at I command
J is going to be equivalent to and P dot
sum of MP dot multiply of and in this
case what happens is we say okay what is
our filter look like here and what does
our image look like here so this is our
image from everything from wherever I is
to I plus h which is our filters
dimensionality and then wherever j is to
our J plus W and now we have kind of
like a block of our image that we are
now trying to multiply our filter across
and then sum up and I think multiply
finishes here and we can just multiply
that by the filter now normally you
would also add this like bias term here
we're not going to do that in this case
again just to illustrate the point and
that should be our convolution unless I
made a mistake somewhere so let's go
ahead and instead of in showing our line
filter let's M show the convolution
of our image and line the filter and
let's hope that no errors occur we're
still printing line filter that's
shapeless I'm doing that and we can then
go over here now this is a much blurrier
form of the image that we had previously
hopefully you can tell that we don't
actually look like that if you can't
look at the bottom I think right of your
screen and you'll see what we actually
looked like in full color too
so this is a very blurry form of what's
going on you're like what happened what
things have been strengthened what
things have been weakened this is wild
but essentially what we're doing here is
we're saying okay we want to like kind
of look for things that have this linear
part this linear kind of like space in
between them and those things should be
emphasized
those are real features for us anything
else we're going to kind of just ignore
now if I modify this such that I'm not
doing the 1 minus part and it's just the
filter as it was we'll get rid of this
and we'll rerun it now this did you do
you know this is a very different image
it's blurry still but in a very
different way it's like like a vertical
blur kind of yeah in this case we
vertically blurred again but this time
we were looking at anywhere that we had
kind of like our individual whitespace
has now or like kind of a bar of white
has now been emphasized if you will now
usually you use a couple of conic well
maybe not usually but I have seen used a
couple of convolutional filters that are
looking for varying kinds of feature I
can look for curvature for example and
we might we can try and build like a
cell small version of that so I'm gonna
just kind of like go through and get rid
of all these ones one track system says
first I'm watching y'all will not be the
last much longer I understand awesome
thank you and we really appreciate it
thanks so much and what we're gonna kind
of do is look at this like and I'll do
like a length or size two curvature sort
the deal where we're looking to like
approximate what like a kind of curve
might look like
and the reasoning for this is
essentially like what might a curve kind
of a very tight pit hairpin curve look
like here sorry this is very sloppy the
most terrible way to draw court yeah
possibly
I suck at drawing curves in general and
then drawing them within matrix
yeah if you think this is very easy
please go ahead and try and if you
succeed please send us the code this is
a huge mess I thought it would be great
to do this live I was like oh it's a
less than sign yeah it's really kind of
a sharp hairpin kind of curve here it's
also not symmetric in case that wasn't
bothering you already it goes the wrong
way so yeah in case you are like really
OCD about that that's terrible and
you'll notice that what this did is
again you get this kind of weird blur
effect but it's essentially convolve as
it goes looking for wherever any sort of
curvature like this occurs if you have
that curvature then it gets emphasized
and if you don't it gets zeroed out now
you can make that emphasis effect
stronger in coloured images by
multiplying by like a like factor of 10
and you can do other things to get that
to happen in our case it shouldn't
change anything as far as I'm aware
because it's grayscale so it just gets
refactored out but you could do things
like that you could do we'll do a
slightly simpler filter so this one's
fairly large and hard for me to
manipulate in real time but let's say we
did like a zero and then we did like a1
a2 a3 a2 a1 and then a zero and
essentially what this is saying is
anytime you have something that goes
from like left to right and grows in
intensity and then diminishes quickly
those sorts of things will now get
emphasized in the right kind of context
and what this helps us kind of
understand is when we apply it to our
image assuming I didn't add an extra
thing there this doesn't necessarily do
anything wild to our image and that
should hopefully be a little bit
intuitive in that these values are very
close to each others so when they get
rescaled nothing really changes but if I
emphasize them a little bit more
using this kind of approximation then
what I can do is I can do this and I'm
keeping them roughly square so we don't
distort our image too much but it
wouldn't you can use any size of filters
is hopefully observable here and as we
go then in this case we see slight
differences I don't know if there is
visible on the big screen as they are on
mind you you see actually a little bit
more distortion it's hard to see if
there it's pretty easy to see my
computer yeah you can see that you're
like shoulders a little bit blurrier my
shoulders gonna get a little bit blurred
because anytime you go like this Delta
and intensity you're going these sorts
of changes as they go whereas like my
hair - they're there you can still see
the outlines pretty clear yeah and so
like you basically end up being able to
see you can account for different kinds
of change and so that's really useful to
a machine is there are certain filters
that are like canonically known to have
certain properties that tell us hey this
is super useful this is really not I
look for straight lines I look for
curves I look for hard transitions like
a delta and basically if you understand
your like matrix algebra you can
implement a lot of like any linear
function it as a matrix so you can
implement most forms of like derivative
as a matrix for example and that's a
really useful property by McKnight says
it's hard to see sorry it's not
necessarily super wide I could make the
filter bigger for a little bit easier to
see but that's you can implement it
yourself from catalog on your own
computer okay and so that's what we can
do with filtering so we now have a way
of convolving our image with itself and
doing all sorts of things and I'm gonna
kind of keep a filter I'm gonna keep a
bird a vertical bar filter around so
we're gonna use a five-piece vertical
bar we'll say actually we'll see you
like whoa that's not a number well then
I guess Sigma I think it was a cute yeah
yep not always the best at typing I'm
never the best at typing that's okay not
necessarily my only needed skill all
right so this is a 5x5 I think if I
counted right and this will be kind of
our filter to go to from there I'm also
going to still use the like one -
part of us not one equals area so the
second part of this that we promised was
pooling and pooling basically says hey I
have this set of pixels I now need to
understand you know what's the most
important attribute of these pixels of
this group of pixels and so a lot of
times the common way of doing this is
doing max pooling we're gonna implement
a general pooling function that can take
max min median whatever and what we're
gonna do is again an image we have some
pool size we're gonna just initialize
that's like a four and by four and then
we have the pool function oops
oh that's not right which we're gonna
just set to num P max so by default this
will implement max pooling which is
pretty common but we could swap out that
pool function for like min or median or
mean and you would do the same and so
what this is going to do is we're gonna
say okay kind of similar to the leg
setup up here so I'm actually gonna just
like copy-paste this and then instead of
Phil shape we have pool size no shape
and now we've kind of realized what
convolution is to pooling and again we
have our output so I'm going to copy
that back over because now all of our
parameters are the same except in this
case what I'm actually gonna do is a
little bit more math I'm gonna divide
that by the height and then I'm going to
oh sorry I should do this there's a lot
of parentheses that go into this okay
so that should be what our height is so
we're essentially just assuming that the
pool does not overlap is what we're now
implementing it's very very similar but
it's off by a little bit then I'm gonna
just replace these with w's got a load
of naming convention okay cool so now
what we're doing is our output is of
roughly the same size except we're
dividing out the factor of the size of
the pool and the reason that we do that
is because now every time we like try to
pool a segment we don't include anything
from that segment in the next pool now
this is not the most general
implementation of this the generalist
the most general way is to include
something called a stride and actually
even convolutions can also have this
stride parameter we just didn't include
it for
but for general like the most general
way to do this would be to include a
stride parameter and that stride says
how many over do I skip when I'm moving
the pool so maybe some parts of the
pools overlap not all of it
maybe the whole thing does or most of it
so the whole pool of relapse you're just
pooling the same thing over and over
again not useful but your stride should
to be at least one and then after that
it can kind of move over they should
intuitively be integers but I can then
if it's one then that's what our
convolution was this is implicitly a
stride of one I'm gonna always go over
by one pixel do my calculation again go
over my another pixel into it again and
then my reset I'm only going down by one
pixel so you can see how that might be
very computationally intensive
convolving tends to be your convolutions
tend to be pretty computationally
intense pooling tends to be a little bit
less because you're trying to reduce the
dimensionality of your data and that's
essentially that's exactly what this
does well so again we're gonna iterate
over actually it's not quite the same
iteration so we're gonna iterate over
for I in range except this time it's a
range from zero to whatever you're like
output dimension is or your output
number so we're gonna keep it kind of
like as this image h minus h plus one
but we're going to skip by h and
hopefully that makes sense it's
essentially just a reformat of the way
that we put this except this time we're
skipping by H's instead of dividing that
out because otherwise I have to do a
little bit more integer math to get this
to work and this just I think is a
little bit simply to understand but you
know if you don't like that go ahead and
do yourself and you can change it cool
and we do the same thing for going
across sorry and so now that we have
this done our output is actually going
to be written like this
our output is the integer result of what
I guess I could just do this I divided
by H and then sorry J divided by W and
now since our J's will be indices that
correspond to like the image that we're
pooling over we want to convert those
indices back into the output indices and
that's what this is and then all we have
to do is apply our pool effects and
you'll notice that pool effects should
return
some actual number if it doesn't and
this will just fall tall over the places
and nothing really technically faulted a
lot of errors and what we're then gonna
do is do this over wherever I is to I
plus h and then wherever J is 2j plus W
and hopefully that makes sense again you
would have a bias term and things like
that but we're gonna kind of just ignore
that for now and the reason I bring it
up is because when we were talking about
all that's like back prop in a neural
network that's where it would come into
play in this layer and then we're gonna
just return output cool and I think that
that is our pooling basically our pool
and so if we do pooling on our image and
we don't need a line filter anymore but
we can say well yeah let's give it no
other parameters and just see what
happens so we're doing a max pooling
nothing broke and we'll notice
everything got pixelated a little pings
AJ yeah yeah and you'll notice too that
things that are lighter are now more
important they're more significant so
essentially to make that a little bit
more dramatic let's do pool size and
let's make this a 32 by 32 pool and our
image isn't super large so that should
be pretty pronounced I think and you'll
notice that when we pull over this we've
basically just said okay there's a white
blob here there's a white blob here and
this black part is still there but there
was literally nothing white in it if
there had been like even a single white
pixel here it would have been the
maximum and it would have become
representative of its block so
essentially what this says this pixel
here says is there was literally a pure
white pixel and it was somewhere within
this entire block it has now been pooled
here so this image this pooled image I'm
going to kind of like pull out from that
function and I'm going to say our pooled
one equals that and we're gonna put
pooled one back in here so that we can
see and I'm gonna actually change the
function that it uses also that's wrong
there yeah we're also going to add that
we don't really want like like max isn't
necessarily representative of
what the image should look like right
like that doesn't really tell us a whole
lot so again people would use multiple
of these like sometimes use multiple of
these in order to gain many features now
what this says is okay take the minimum
one and our lowest intensity values or
black images they're the darkest color
and so what this tells you is like hey
this segment actually had a lot of
lightness in it such that even when we
were taking the minimum we still ended
up with something that wasn't pure black
same thing here these pixels were all
white the entire thing was white same
thing here same thing here except for
this little segment and then there's
like a little variation here so now it
might make sense for us to say well we
don't want it to be so dramatic like
that's kind of a lot I think just having
calculated the reverse of the enhance
yeah exactly we have just kind of you
know I'm like CIA NCIS they kind of like
oh like enhance and it gets clearer
we're doing the exact opposite or D
hansung it we're just like boom it's way
less clear and I'm gonna actually print
out to the images original shape so that
we can see how this reduces
dimensionality intuitively you should be
able to kind of think oh well we're only
going over so many parts of it it
reduces it by roughly the scale
so whatever the pool oh my gosh whatever
the pool size is what's kind of like if
we divide each corresponding term by the
pool size we end up with what the actual
scale is now I've reduced the pool size
and now we get a little bit more
understanding of like our individual
features and things which is cool and
I'm gonna swap this out one more time
for mean which is kind of the
mathematical average now our
dimensionality reduction is the same we
still reduce it by the exact same amount
but we now have a little bit clearer of
an approximation of people which is
maybe meaningful so it might be
meaningful for our neural net to learn
oh if I see something that has this kind
of dark here and this white here then
that's like a person and this is also a
person but that is a board or something
that's a very weird way of phrasing that
but it's the right kind of concept so
now what we might do from there is we
have the single pool and I'm gonna start
putting titles on these because we're
gonna plot a bunch of them
yeah I'm not going to dynamically do
that so now let's say that we want to
like we pull like the mean first and
we're gonna use a pretty small pool at
first we just want to like get it to
like pool a little bit and then from
there we're actually going to convolve
it a little bit it's not where the
convolve 1 equals convolution is mean
the most useful function that we're
looking to use to not necessarily a lot
of people actually use Mac's as their
most useful function but also you can
use multiple of these so you can have
multiple layers that pull out different
features as we go I like to use mean in
this case because I'm just trying to
like reduce dimensionality and then I'm
gonna convolve it and then I'm gonna
reduce dimensionality again but it'll be
a little bit different meaning so in
this case we're just kind of assuming
that I got like an image that was just
way too big I can't process these fast
enough I'm gonna use mean to kind of
retain like in general what was there
but I want to reduce the dimensionality
by about a factor of 16 so 4 by 4 and so
once I do that I can say ok let's do a
convolution where our image is now
pooled 1 and our filter is our line
filter up here so we do like this line
filter there's no other parameters we
need and then here I'm going to just
plot that now no oh my gosh what was the
complete catastrophe and this will be
convolve 1 oops
that participate no and this will now
also be convolve I could spell there we
go and now when we run this I'll stop
displaying those shapes people
understand I think and so here's our
pooled one so we only pulled by a little
bit so we just kind of like pixelated a
little it's not not huge it's not crazy
we just kind of reduced her
dimensionality a little bit but it was
meaningful to us we scaled it down by 16
so it's clear it's like a human we
didn't lose a whole lot of information
but we made it a lot faster for us right
now from here then we would do this
convolution and you'll notice that this
convolution is a lot more dramatic than
the one we had before yeah and the
reasoning for that it looks like now I'm
doing it a feeling like thank you vision
exactly and it's kind of like if we were
to pass like a line over it and then say
hey like what things do you see on
either side of this line right and it's
it's literally what happened here is we
just kind of like blocked out part of
like the middle
and then just kind of like scanned
across and so you get this like weird
stuttering pattern yeah like the weight
like where the left side of my head
meets the blackboard looks a bit weird
yeah right around here I think your ear
got da yam is essentially what happened
and the kind of reasoning for that is
that we have this vertical bar and
that's what we were checking
so we've now done that but I've done
this convolution I now know what kind of
features are really important from that
I want to kind of repeat this I want to
pool again but this time I'm going to
pool using just the max of the features
maybe and I'm going to call this pooling
too and I want our filter to be a lot
larger maybe like a factor of that would
be like four times larger and so yeah
and I'm gonna use NP max here and I know
that's the default but I'm gonna use it
anyway just to be explicit so again
here's our pooled one here's our
convolve one now when we do pool - oh
sorry I did that on the images so I was
like well and we're back to roughly
confusing yeah so here's like involved
one little bit the same thing yeah it
was literally the identical thing that's
like oh that's wack uh-huh so here yeah
this is better pulled one can involved
one and now our mmm that's not what I
wanted not at all what I wanted oh yeah
this is convolve because you literally -
sorry I'm literally showing you pulled
one that's like that can't be right
let me just double check myself before I
embarrass myself for a fourth time yeah
this is okay that should be right cool
nice it's like what is going on looks
pretty close - yeah and it looks very
similar to as if we had just pulled the
first one like this as it was using max
right so if we wanted to like double
check that we can replace this with Max
and we would see that it's roughly
that's roughly true let's increase the
size of our filter here though and from
that we should be able to kind of like
convey what's going on but we're again
going to use mean and we're going to
kind of try out what this does and so
you'll notice that we're even more
distorted this time so yeah things got
carried over and
when week involved it things got even
more distorted so that was a lot more
blurry but you'll notice that we
retained this kind of like gradient
around each of our like descents across
things
yeah and that's usually pretty useful is
like where is there a gradient where is
there not you can also do the opposite
where you want to really sharply define
kind of like increase your contrast if
you will and when we do this now this
looks meaningless this isn't right it's
just not because if you know what what
it is you can still tell you do yeah you
get kind of like still into it what's
going on but it's roughly meaningless
now it's meaningless to us ish like we
kind of know this like Wayne up here
there was a person here there was a
person here and there's a blackboard
behind them but let's imagine that this
was in color still then having like a
blue blob here for us to in human terms
that represents me wearing a blue shirt
that day and a kind of like darker color
pattern here whereas other ones might be
a little bit lighter represents you
wearing your glasses and so this might
actually be a really useful feature map
for a machine who's trying to classify
which stream we're in because this
feature map will look very different
from this stream as opposed to the next
stream and in order to kind of
illustrate that point I'm gonna grab the
image from the opposite end of this
stream so we should be somewhat
different looking okay and we'll compare
the two maybe probably not side-by-side
but we'll compare them and so what we're
gonna do is just copy this entire
segment so this is terrible code design
and you're looking for what not to do in
a code review this is it and we're going
to okay well I should have put more
spaces there because now I can't tell
where it starts I think down a little
bit everything yeah perfect and we're
gonna just redefine image as images
negative one this is from the complete
opposite stream so we're gonna go
through these again this is the original
one that we're looking at and we kind of
see the progression as it gets pooled
here's pool two now if we go to the next
image completely different right it's a
very different image a you know
different science - we've also switch
sides and that's really important you're
also not wearing your glasses in this
one yeah and so now as this gets pulled
again the convolution is just kind of
like it looks like it just kind of
blurred everything but here this pool -
if you can kind of remember what the
previous pool game looks like it's
complete
and that is cristobalite corner in the
top right though that white right that
way box is still there and so this to a
computer is a lot easier for it to
process because we've reduced its size
by a substantial amounts by non-trivial
amounts and we can now flatten this and
so that's kind of our last layer to
throw in here and describe how it works
and flattens fairly easy flatten is just
saying hey given an image let's get the
height and width and that's gonna be
image dot shape and then we're gonna
just return NP that reshape of image to
be of the size height times width comma
and essentially what this just does is
it says okay we can flatten this image
out to be just an array of numbers but
you'll notice that array of numbers is
what we can throw through a neural map
right and now we have the ability to
kind of combine this all together to be
a convolutional neural network this
these three layers kind of this
convolutions pooling a lot of times
they're used in pairs so like i'll
convolve the image I'll pull it i'll
convolve it again and then I'll pull it
and once I kind of like broken those
apart and like gotten my important
features out of it then I'm going to
flatten the result pass that into the
dense layers of my neural network and at
the end you'll spit out a class and so
the next time we're on this stream
you'll see us build these without
building the actual independent like
pieces on our own we'll just use an API
but you now should have a pretty solid
understanding of where these things came
from they have extra features and kind
of knobs that we can adjust but this is
the essence of what's going on in each
of those layers and essentially what
we're trying to motivate is this sucks
to implement on your own and it there's
a lot of places for reusability and
optimization and that's where a library
like tensorflow is extremely useful is
we can then just access each of these
things the reason that I think it's in -
it's helpful to build an intuition on
these things is let's say that you have
two pooling layers and two convolutional
layers and they're intertwined and your
classifier is just not getting it it's
just not quite it has one class of
images that it keeps mistaking for
another class you're not sure why it's
doing that you're really annoyed you're
like what the heck is going on
and you remember oh well you know
there's all sorts of kind of problems
with if I'm convulsion of filters
maybe my filter set only includes two
filters and they look for vertical bars
and horizontal bars but the
distinguishing characteristic between
class 3 and class 4 is not vertical
versus horizontal it's convex versus
concave curvature
well vertical and horizontal convex and
concave have roughly the same amount I
mean if they're like parts of a circle
convex versus concave have like the same
amount of vertical and horizontal so you
can't distinguish them but if you were
to add a concave and a convex filter to
your filter set you are now capable of
explaining one why your classifier now
works and to how everything actually
interacts and you can kind of
intuitively understand where that came
from that would be part of the
convolutional step if we wanted to say
hey my machine learning network is
taking super long to train but I'm
pretty sure that it's just using a bunch
of redundant or irrelevant features or
maybe it has way too much variance and
it's miss predicting all over the place
when it tries to generalize then what
you might say is oh maybe I can just
reduce the dimensionality try to boil it
down to these key features and from
there I can then reduce training time
for one because my dimensionalities
reduce my feature dimensionality is
reduced but also I now have only the
meaningful features contained so I'm not
going to generalize poorly so when I
then get a new piece of data I can
generalize to let that a little bit more
intuitively the noise isn't going to
freak me out that's what that kind of
like kind of boiling it down to its mean
helps reduce like high variance noise
and then reducing a future
dimensionality from there is also usable
sorry LCS says so basically these
transformations are to reduce
dimensionality even put images for
classification not purely just that kind
of for all the reasons we've just
enumerated they do a lot of other things
they they're very powerful
transformations and convolutions allow
you to pick out which features are
important so it doesn't the convolutions
of themselves don't actually reduce the
dimensionality at all or at least not
not meaningfully they essentially just
say hey these features have been
emphasized and here's what an image with
those features emphasize looks like and
so you might then create a composite
image that has several of these kind of
like convolution volved layers in it
and that composite says oh okay well in
this image these features are really
important in this image these features
are really important and it's really
useful for your machine to then be able
to distinguish between those but then we
want to be able to pool those features
back together to allow us to figure out
what features are actually important and
that's where your dimensionality
reduction actually comes in I think what
you Andrea says is it essentially a
polarizing operation I think kind of by
intuition
yeah that's roughly I'll so polarizing
kind of means like we're pushing them
separate in two directions I think the
general form of that is about right it's
kind of like how do we distinguish
between different images and it's kind
of like how do we pull out or emphasize
certain features so that we can then
compare those features across images and
I think that that's what you meant which
is as bergen we right so yeah these
these ideas are they definitely take a
couple of kind of passes in order to
forward and background if you will to
understand and neural networks are by no
means simple they're not even fully
mathematically understood but it's
definitely I think here you have an
intuition for why they work what they do
at least at the very if this is even
still a fairly high level if you wanted
to really dig into it there's a bunch of
white papers is a bunch of Khan Academy
tutorials you can go and look through
the math I would recommend a decent like
probability or statistical understanding
and a decent understanding of like
linear algebra a lot of these things
base themselves in like probability
theory like a log likelihood theory and
linear algebra theory so how many
matrices do you know how to manipulate
and things like that but I don't think
that should scare anyone away
essentially yeah I think it was pretty
cool I mean a lot of definitely flew
over my head yeah where is cool yeah
it's quote the very end I sort of see
what you were doing and why it was
important yeah I think I'm going to
spend some time doing a little bit of
all the stuff but no thanks so much for
doing this part too and I guess the next
week part three will go at what up what
would you say would be the outline for
next week so I think next week we can
look forward to building our we'll set
the goal of like a style transfer at the
least and kind of just building that and
then
our eventual goal is to build these
sharp adversarial images
I think style transfer gives us kind of
the motivate or kind of the
understanding of how you can actually
use a convolutional neural network
beyond just a classifier because they
are still kind of used that way
convolutional neural networks are really
well-suited for classification tasks but
I'm trying I guess we're trying to push
towards the direction of how do we use
them to generate new image and what does
this how would you define style transfer
what does that mean
so style transfer is going to say hey
given a let's say a piece of art and a
image that I have taken like a
photograph can I get the photograph to
look in the style of the art that we
have so make us a van Gogh painting
exactly and I think that's kind of our
next step okay that would be super cool
well thanks everybody for tuning in so
we'll do that again a little bit next
week I'll be here an attic sounds good
tune in tomorrow for just me we're gonna
do a part two of our chat codes
everything in the stream which turned
out quite well he's pretty good yeah we
did a chaos run and then we did a goal
where yeah we made a square like an
ASCII square oh nice
so that was we had a lot of we had a lot
of trolls in the chat yeah all in good
humor of course it was fun
course it was fun seeing how people like
got around people like obviously
trolling the script right like some
people would be like okay amend this
line to actually do this instead like
okay oh yeah no it was great and so
thanks everybody who's gonna tune in who
tune in today who might tune in tomorrow
if you're watching this on youtube you
can tune in to twitch.tv slash cs50.tv
to watch cs50 live or if you're on
facebook same thing
no and tomorrow we'll we're gonna be
able to do libraries I think I'm gonna
add the ability for you'll do like a
very nice song like request a pip
install suite um I might have to vet
just to make sure they're not like
victim librarian library that just likes
this is completely malicious so probably
like well-known libraries but if people
want to get kind of fancy and do stuff
like that we'll do a couple of hours of
that tomorrow and then on Friday if you
haven't already definitely submit at CCO
LOI slash code review David is going to
do a code review on Friday and also if
you're in the diff you let me see if I
get a link to our discord actually where
we're trying to potentially take live
suggestions via our discord so let me
just see if I get the cs50 discord link
here so if you are watching this and you
go and you can also just kind of get
access to this by doing a Google search
for cs50 discord let me make sure I'm in
the right thing here it's this one you
can go to the cs50 discord that's an
invite it's a public invite so you can
join the discord channel definitely do
that there's a bunch of sub threads post
in the hash live code review channel on
the discord when we're actually going
live or in advance and I'll curate it
and we'll potentially show it live on
stream so it's sweet time that's all
next week we'll do we'll do a whole
bunch of more fun stuff so thanks again
so much thank this is cool overwhelming
but I at least pretty good but I
actually am inspired by it to uh to dig
in I think a little bit more in my math
and a little bit more comfort yeah
that's the whole goal yeah some reason
the thing is freaking out I can't do it
I can't do oh you keep my shortcut sir
yeah what is going on okay yeah it is
completely freaking out but anyways
thanks everybody so much this was cs50
live neural nets part two next week
we'll do neural nets part three with
Nick nip in the meantime stay safe and I
will catch all of you tomorrow bye bye