Hey are you okay?
Siraj, show them the math behind deep learning.
Totally.
hello world it's SIraj and
let's learn about the math needed to do
deep learning. Math is in everything not
just every field of engineering and
science. It's between every note in a
piece of music and hidden in the
textures of a painting. Deep learning is no
different. Math helps us define rules
for our neural network so we can learn from our data.
if you wanted to, you could use deep learning without
ever knowing anything about math.
There are a bunch of readily
available APIs for tasks like computer
vision and language translation, but if
you want to use a library like TensorFlow
to make a custom model to solve a problem
knowing what math terms mean when
you see them pop up is helpful
and if you want to advance the
field through research,
don't even trip! You definitely need to
know the math. Deep learning mainly pulls from
three branches of math: linear algebra,
statistics and calculus. if you don't
know any of the topics, i recommend
a cheat sheet of the important concepts
and I've linked to one for each in the description
so let's go over the four-step process
of building a deep learning pipeline
and talk about how math is used at each step
once we've got a dataset that we want to use
we want you process it we can clean the
data of any empty values, remove features
that are not necessary but these steps
don't require math. A step that does, though,
i s called normalization.
This is an optional step that can help our model
reach convergence, which is that point
when our prediction gives us the lowest
error possible, faster, since all the
values operate on the same scale.
This idea comes from statistics.
You have a 17.4 percent chance of making a straight.
there are several strategies to normalize data
although a popular one is called min max scaling.
If we have some given data we can use the
following equation to normalize it.
We take each value in the list and subtract
the minimum value from it, then divide
that result by the maximum value minus
the min value. we then have a new list of data
within the range of 0 to 1 and we do
this for every feature we have so
they're all in the same scale
after normalizing our data we have to ensure
that it's in a format that our neural
network will accept. This is where linear
algebra comes in. There four terms in
linear algebra that show up consistently.
Scalars, vectors, matrices and tensors.
A scalar is just a single number.
A vector is a one-dimensional array of numbers.
A matrix is a two-dimensional array of numbers.
And a tensor is an N dimensional array of numbers.
So a matrix, scalar, vector and spectre, wait not spectre,
can all be represented as a tensor.
Want to convert data, whatever form it's in, be that images,
words, videos, into tensors, where n is
the number of features are data has
and defines the dimensionality of our tensor.
Let's use a three-layer feed-forward neural network
capable of predicting a binary output
given an input as our base example
to illustrate some more math concepts going forward
When do we use math and deep learning? When we normalize during processing.
Learn a model parameters by searching. And random weights be initializing.
Tensors flow... From input to out Then measure the error to measure the doubt
It gives us what's real and what's expected. Back propogate to get cost corrected.
We'll import our only dependency, Numpy,
then initialize our input data and output data as matrices.
Once our data is in the right format, we want to build our deep neural network.
Deep nets have what are called hyperparameters. These are the
high level tuning knobs of the network that
we define and they help decide things
like how fast our model runs, how many
neurons per layer, how many hidden layers.
Basically the more complex your neural
network gets, the more hyperparameters you'll have.
You can tune these manually using knowledge you have about
the problem you're solving to guess
probable values and observe the result.
Based on the result, you can tweak them
accordingly and repeat that process iteratively.
But another strategy you could use is random search. You can
identify ranges for each, then you can
create a search algorithm that picks
values from those ranges at random
from a uniform distribution of possibilities
which means all possible values have the
same probability of being chosen.
This process repeats until it finds the optimal hyperparameters.
Yay for statistics!
We only have number of epochs as our hyperparameter,
since we have a very simple neural network
We'll use probability to decide our weight values, too.
One common method is randomly initializing samples
of each weight from a normal distribution
with a low deviation, meaning values are
pretty close together. We'll use it to
create a weight matrix with a dimension
of three by four, since that's the size
of our input. So every node in the input
layer is connected to every node in the next layer.
The weight values will be in the range from -1 to 1.
Since we have three layers, we'll initialize two weight matrices.
The next set of weights has a dimension four by one
which is the size of our output.
As data propagates forward in a neural network
each layer applies its own respective operation to it.
transforming it in some way, until it eventually outputs a prediction
This is all linear algebra.
It's all tensor math.
We'll initialize a for loop to train our network 60,000 iterations
Then we'll want to initialize our layers.
The first layer, our input, gets input data.
The next layer computes the dot product of
the first layer and the first weight matrix.
When we multiply two matrices together,
like in the case of applying weight values to input data,
we call that the dot product.
Then it applies a non-linearity to the result which we
decided it's going to be a sigmoid.
It takes a real value number and squashes
it into a range between 0 and 1.
So that's the operation that occurs in layer 1,
and the same occurs in the next layer.
We'll take that value from layer 1
and propagate it forward to layer 2,
computing the dot product of it and the next weight matrix,
then squashing it into output probabilities with our non-linearity.
Since we only have three layers, this output value is our prediction.
The way we improve this prediction,
the way our network learns, is by optimizing our network over time.
So how do we optimize it?
Enter calculus.
The first prediction our model makes will be inaccurate.
To improve it, we first need to quantify exactly how wrong our prediction is.
We'll do this by measuring the error, or cost.
The error specifies how far off the
predicted output is from the expected output.
Once we have the error value we
want to minimize it because the smaller
the error the better our prediction.
Training a neural network means
minimizing the error over time.
We don't want to change our input data but we can
change our weights to help minimize this error.
If we just brute forced all the
possible weights to see what gave us the
most accurate prediction, it would take a
very long time to compute.
Instead, we want some sense of direction for how we
can update our weights such that in the
next round of training our output is more accurate.
To get this direction we'll
want to calculate the gradient of our
error with respect to our weight values.
We can calculate this by using what's
called the derivative in calculus.
When we set deriv to true for our nonlin function,
it'll calculate the derivative of a sigmoid.
That means the slope of a sigmoid at a given point,
which is the prediction values we give it from l2.
We want to minimize our error
as much as possible, and we can intuitively
think of this process as dropping a ball
into a bowl where the smallest error
value is at the bottom of the bowl.
Once we drop the ball in, we'll calculate the
gradient at each of those positions,
and if the gradient is negative, we'll move the ball to the right.
If it's positive, we'll move the ball to the left.
And we're using the gradient to update our weights accordingly each time.
We'll keep repeating the process until eventually the gradient is zero,
which will give us the smallest error value.
This process is called gradient descent,
because we are descending our gradient to approach zero
and using it to update our weight values iteratively.
I understand everything now.
Still understand everything.
So to do this programmatically, we'll multiply the
derivative we calculated for our prediction by the error.
This gives us our error waited derivative which we'll call l2_delta
This is a matrix of values, one for each predicted output,
and gives us a direction.
We'll later use this direction to update this layer's associated weight values.
This process of calculating the error at a given layer
and using it to help calculate the error weighted gradient
so that we can update our weights in the right direction
will be done recursively for every layer
starting from the last back to the first.
We are propagating our error backwards
after we've computed our prediction by propagating forward.
This is called back propagation.
So we'll multiply the l2_delta values by the transpose of its associated weight matrix
to get the previous layer's error,
then use that error to do the same operation as before,
to get direction values to update the associated layers' weights.
so error is minimized.
Lastly, we'll update the weight matrices for each associated layer
by multiplying them by their respective deltas.
When we run our code we can see that
the error values decreased over time,
and our prediction eventually became very accurate.
So, to break it down.
Deep learning borrows from three branches of math,
linear algebra, statistics and calculus.
A neural net performs a series of
operations on an input tensor to compute a prediction
and we can optimize a prediction by using gradient descent
to back propagate our errors recursively,
updating our weight values for every layer during training.
The coding challenge winner from the last video is Jovian Lin.
Jovian tried out a bunch of different models to predict sentiment
from a dataset of video game reviews. Wizard of the week!
And the runner-up is Vishal Batchu.
He tested out several different recurrent nets
and eloquently recorded his experiment in his ReadMe.
The coding challenge for this video is to train a deep neural net
to predict the magnitude of an earthquake
and use a strategy to learn the optimal hyperparameters.
Details are in the ReadMe.
Post your GitHub link in the comments, and I'll announce the winner next video.
Please subscribe if you wanna see more videos like this.
Check out this related video, and for now I got to get
my math turned up to a million.
So thanks for watching!