So now you’re probably thinking – wow, deep nets are really great!
But why did it take so long for them to become popular?
Well as it turns out, when you try to train them with a method called backpropagation,
you run into a fundamental problem called the vanishing gradient, or sometimes the exploding
gradient.
When that happens, training takes too long and the accuracy really suffers.
Let’s take a closer look.
When you’re training a neural net, you’re constantly calculating a cost value.
The cost is typically the difference between the net’s predicted output and the actual
output from a set of labelled training data.
The cost is then lowered by making slight adjustments to the weights and biases over
and over throughout the training process, until the lowest possible value is obtained.
Here is that forward prop again; and here are the example weights and biases.
The training process utilizes something called a gradient, which measures the rate at which
the cost will change with respect to a change in a weight or a bias.
Deep architectures are your best and sometimes your only choice for complex machine learning
problems such as facial recognition.
But up until 2006, there was no way to accurately train deep nets due to a fundamental problem
with the training process: the vanishing gradient.
Let’s think of a gradient like a slope, and the training process like a rock rolling
down that slope.
A rock will roll quickly down a steep slope but will barely move at all on a flat surface.
The same is true with the gradient of a deep net.
When the gradient is large, the net will train quickly.
When the gradient is small, the net will train slowly.
Here's that deep net again.
And here is how the gradient could potentially vanish or decay back through the net.
As you can see, the gradients are much smaller in the earlier layers.
As a result, the early layers of the network are the slowest to train.
But this is a fundamental problem!
The early layers are responsible for detecting the simple patterns and the building blocks
– when it came to facial recognition, the early layers detected the edges which were
combined to form facial features later in the network.
And if the early layers get it wrong, the result built up by the net will be wrong as
well.
It could mean that instead of a face like this, your net looks for this.
The process used for training a neural net is called back-propagation or back-prop.
We saw before that forward prop starts with the inputs and works forward; back-prop does
the reverse, calculating the gradient from right to left.
For example, here are 5 gradients, 4 weight and 1 bias.
It starts with the left and works back through the layers, like so.
Each time it calculates a gradient, it uses all the previous gradients up to that point.
So, lets start with that node.
That edge uses the gradient at that node.
And the next.
So far things are simple.
As you keep going back, things get a bit more complex - that one for example uses a lot
of gradients, even though this is a relatively simple net.
If your net gets larger and deeper, like this one, it gets even worse.
But why is that?
Well, a gradient at any point is the product of the previous gradients up to that point.
And the product of two numbers between 0 and 1 gives you a smaller number.
Say this rectangle is a one.
Also, say there are two gradients - a fourth - like that - and a third.
If you multiply them, you get a fourth of a third which is a twelfth.
A fourth of a twelfth is a forty-eighth.
You can see that numbers keep getting smaller the more you multiply.
Have you ever had this issue while training a neural network with backpropagation?
If so, please comment and let me know your thoughts.
As a result of all this, backprop ends up taking a lot of time to train the net, and
the accuracy is often very low.
Up until 2006, deep nets were still underperforming shallow nets and other machine learning algorithms.
But everything changed after three breakthrough papers published by Hinton, Lecun, and Bengio
in 2006 and 2007.
In the next video, we’ll begin taking a closer look at these breakthroughs, starting
with the Restricted Boltzmann Machine.