What's going on everybody welcome to the Neural Networks
from Scratch series where we will be creating
creating Neural Networks end-to-end so from...
the Neurons, the Layers,
Activation functions, Optimization,...
Back Propagation all this stuff were going to be coding
that from scratch in python. Now...
everything we do we are going to show first in truly raw python
no libraries or no 3rd party libraries
and then we are going to use NumPy for multiple reasons
NumPy just makes a lot of sense here. It's a
extremely useful library and
it'll cut our lines of a full application down-a-ton
it'll make it much faster
and NumPy is a great thing to learn, so...
we will show everything from scratch in python first and then we are going to use NumPy. Now...
why would anybody wanna do this to themselves?
Well, first of all, it's very interesting umm...
the idea is not-even though we effectively we are going
to be programming our own Neural Networks
framework, it's not really the
purpose here. The purpose is to actually learn
how Neural Networks work at a very deep level so that
when we go back to whatever Framework
we actually use, be it PyTorch or Tensorflow's Keras
or maybe some library that doesn't even exist yet.
We actually understand what we are doing. So
from myself when I at least when I learned Deep Learning, it was all-
-yes it was hard but at the same time
everything was kinda solved for me you know, how
many layers, how many multiple layers, what Activation functions to use
but all that stuff was just fed to me
I just kind of-okay
and then I just sort of memorized like here's the activation function
used for this sort of task
But I didn't understand why and
this became a problem when I tried to solve problems that
had not yet been solved for me.
So classifying images of handwritten
digits? Pretty darn simple. Taking that
a step further and classifying images of cats and dogs?
Pretty simple.
But then taking that just a tiny step further
and classifying images that were instead frames from a video game
and trying to map that to actions that I want to take
in the video game. Suddenly, I'm lost.
and that's.....
...that's no good and there's no way to really know
where to go next and you can see that there is a problem
but actually trying to solve
more custom problems is going to require to have a deeper
understanding of how things actually work
Unfortunately though deeper understanding Deep Learning
can be very very complicated.
So you can just-just a simple forward pass of a Neural Network
Neural Network looks
extremely daunting. So looking at the calculations
for a Neural Network, it looks pretty confusing fairly quick
You've got your input data
and every unique input and every unique neuron.
The information coming through has a unique weight associated-
with it. Those get summed together per neuron
neuron plus a bias. Run through an
activation function and then we do that for
every single layer, giving us the output
information. From there we wanna calculate a loss
which is a calculation of how wrong the neural Network is,
so that hopefully we can fix it.
And at the end of that, even though that was just a forward pass
that looks already extremely-extremely daunting
Now let's take a look at the exact same formula just in code version.
You have all your inputs times your weights (Input x Weights)
And you don't have to follow along perfectly here just look at each element and see if any of these elements
look like they are over your head. They really shouldn't be.
So inputs times weights, you can make this even simpler by doing just a dot product so 'np.dot'
y1 is just the max of zero and the output
We do this for all the layers. Then we have
our softmax (activation function) at the very end.
And then the loss turns out to be
a negative log (logarithmic) loss
due to the nature of neural networks.
So this is the entire forward pass and calculation of loss formula
I urge you to look at all of the functions and things that we are doing here
to determine if any of these is really over your head because it shouldn't be right?
We've got very simple functions going on here
We are calculating a log, sum-if you don't know what a 'log' means btw we're going to explain it.
'Sum', you should know what that means.
'Exponential', again if you don't know what that means we are going to explain but it's very simple.
'Dot Product', again if you don't know what that means
no worries we will explain it, very simple.
'Maximum', is just the max whatever two values you pass here. Again very simple.
Some more dot products,
'Transpose', again very simple.
If you don't know it we will explain it anyway
That's it! None of this is over your head I promise you
As far as prerequisites go the only expectation I have from the viewers is that you understand
programming and Object-Oriented Programming (OOP)
otherwise, you're going to feel kinda lost.
If you're coming from a different programming language you will probably find
python simple
You can probably use the programming language that you are familiar with
Everything is so low level here that you should be able
to follow along in any other language that you want
So feel free to do that, if you wanna do that
otherwise, if you wanna follow along in python make sure you know the basics and OOP
I'll put links in the description for both of those
If you need to brush up
Next, is the version of python
If you follow along with python we are gonna do things like 'fstring'
I'm gonna be on Python 3.7
The NumPy version, I can't think of any reason of why it would matter.
I'll put that information in the description
just in case any function does change.
Like I said, everything's so low level this series should be good for like 10 years.
Let's hope for that :)
So those are the prerequisites
it's really not much.
I don't expect anybody to have any background knowledge of Deep Learning
So if you do know things about deep learning, yes we are going to cover hopefully quickly
the fundamentals, just so people understand
like what exactly are we aiming for here?
And then the bulk of your understanding of how Neural Networks work
is gonna come from us just building these Neural Networks
If things feel a little fuzzy to you
it's probably normal, to be honest.
I think once you build a Neural Network from scratch
that is all the understanding you are gonna need
So you're not expected to know math or anything like that.
For me in college, the only math class I took was 'Math Fundamentals'
And I don't think we did any math calculation at all in that class.
It was definitely a joke.
So if I can do this, I know you can do this too.
So If you wanna brush up on Math
There is Khan academy for Linear Algebra and the calculus stuff
But I wouldn't even suggest you go through the full series on either of those topics
You can use those to kinda like a spot check issues that you still find confusing
With that in mind, this series is also provided in conjunction with the 'Neural Networks from Scratch' book
We are going to be covering the same material for the most part.
The book might be a little more verbose.
The series is obviously free
but the book ha various prices depending on what you want.
The E-book, Softcover, Hardcover. We ship everywhere in the world
Access to the book gives you access to the E-book.
So whichever version you buy you always have access to the E-book.
That gives you the opportunity to access the google docs
draft, currently it's in a draft form
at some point, it won't be a draft anymore
but you can highlight, post comments, ask questions inline with the text
Also if you're impatient you can access even though its a draft right now
it's complete from basically end-to-end
of training the Neural Network
and we're doing testing right now.
So, we are already obviously quite a bit ahead of where the videos are
so if you're impatient you can also access information earlier with the book
I would use the book as either reviews or the videos as reviews
So I would maybe read the book before watching the video and then use the video as a review of what I learned in the book or vice versa
This is a topic that you're not gonna blow through this in a weekend.
It's gonna require multiple sittings multiple environments
and ideally multiple mediums so,
If you're interested in the book you can get that at 'nnfs.io'.
So we call these Neural Networks because they look visually like a network.
You've got your Neurons which in this case are the blue circles.
They're connected via those orange lines
and in this case, we have basically the input layer, two hidden layers of 4 neurons each
and then your output layer.
Now data is gonna get past forward through this,
so it starts at your input layer, so in this case, we only have two
pieces of data they are gonna come in.
That gets past forward to that first hidden layer
Then that hidden layer passes data to the second layer and then finally to the output layer
where we hope that it will output something that we want, for example:
based on some sensor data maybe we wanna predict
Failure or not failure. You could either have one neuron or in this case we have two neurons.
So the top neuron might be a failure neuron, the bottom neuron is a not failure neuron.
And depending on which one has the higher value
that's the supposed prediction.
Now the end goal of neural networks like most machine learning is to take some input data
and produce output data that is desired.
In this case, we've got images of cats and dogs.
We hope that we can pass it through in pixel form to our neural network
and if its a dog then that final output neuron on top
is going to be the strongest, if it's a cat
then that final output neuron on the bottom will be the strongest.
And we can do this by tuning the weights and biases
So all those unique weights and biases, we do that by tuning those
and that is the actual training process.
It's tuning those in such a way so that
hopefully, we can take data that this neural network has never seen,
give it pictures of cats and dogs that it has never seen
and have it accurately predict those
So how and why do neural networks work?
Well, if you just look at them and really consider what's going on
every neuron is connected to the subsequent layer of neurons in folds.
So each of those orange lines, that connection
that's a unique weight
and then every neuron is a unique bias.
So what this ends up giving us
is a huge number of uniquely tunable parameters
that go into this GIGANTIC function. So for example, with
64 x 3 hidden layers here we have
9164 tunable parameters
in this gigantic function. And each of those parameters
impacts the output of the next neurons
and so on. And so what we end up having
are these complex relationships that can in theory
be mapped. So to me the
impressive thing of neural networks is not
necessarily all of those connections its not
really complex to understand.
What the hard part of neural networks and deep learning is figuring out
how to tune such a thing.
Alright! since I think it would be lame to not post
any code at all in this first video we are going to begin to code neuron.
But first I wanna go over the version numbers real quick
because quite frankly I am gonna forget to put it in the description
So I'm using Python 3.7.7
NumPy 1.18.2
Matplotlib 3.2.1
Again all of this should work
very far into the future
But just in case not, there are
the exact versions if anybody needs to have the exact version to follow along
Now, let's go ahead and begin to code
just to say I am using Sublime Text
You can use whatever editor you want.
Everyone's got a really strong opinion on editors.
I'm gonna be using Sublime text. There might be a time where we use Jupyter Notebook.
Who knows, I'm gonna use whatever makes the most sense to me at the time
which may not make sense to you or me later
Anyway, continuing on, so every neuron basically-
Let's pretend we are coding a neuron that's somewhere in this-
densely connected feed-forward multilayer
perceptron model. Using all the big words.
Don't worry about knowing what those mean, by the end of this
you will know all of those words.
But part the problem with learning deep learning is people use the same 3-4 words
for the exact same thing and it can be very daunting. So for now,
we are gonna called this a neuron, it's somewhere in our neural network.
Now in this fully connected neural network
every neuron has a unique connection
to every single previous neuron.
So, let's say there are 3 neurons
that are feeding into this neuron that we are gonna build.
So, we don't know much about those neurons but we know that they are outputting some value
So first, their output becomes the neurons that we are coding's inputs
We are gonna just make up some numbers, we're just gonna say 1.2, 5.1, 2.1
Those are the unique inputs
So these are outputs from the three neurons in the previous layer
Every unique input is also going to have a unique weight associated with it.
So we are going to say weights and you should know how many weights we are going to have
Well since we have three inputs we know we are going to have three weights (3.1, 2.1, 8.7)
I'm just making up these numbers.
It's just for, beginning to code how neurons are gonna work
So you've got your inputs, your weights
and then every unique neuron has a unique bias
So bias equals to 3
So now, the first step to for a neuron
is to add up all the inputs times weights plus the bias
So this is relatively simple.
In very raw python no loops required at this stage.
We're just going to say basically
output so far of this neuron is going to be
*look at the screen now*
*You need to have some basic knowledge of 'list' here*
That is basically so far...
There will be other things that will happen soon enough, but so far
that is the output to our neural network
So we'll just print the output, run it. 35.7
If you're new to Sublime Text for some reason
'ctrl + b' to run. But you'll have to set up the build system to run with python.
But I would expect how
that people know how to work with a programming language
But if not feel free to
either comment below, join us in 'discord.gg/sentdex
And we'll be happy to help.
Anyway that's it for now
if you've got questions, comments, concerns, whatever
moving forward feel free to ask them below
otherwise, we're just gonna keep slowly shipping away
at Neural Networks from Scratch
As you can see pretty darn simple so far
and for the most part, it's just adding a bunch of-
as we've broken it down so far
We are gonna break it down to the point where every little additional step
is gonna look an awful lot like that
There's a couple of points that may be a little more challenging
but the goal is to break it down
so much that it is painfully simple
So I will see you guys in the next video :)