Hello, I'm Josh Starmer and welcome to StatQuest. Today we're going to talk about
recurrent neural networks, and they're going to be clearly explained!
Lightning and Grid are totally cool. Check them out when you've got some time.
NOTE: This StatQuest assumes that you are already familiar with the main ideas behind
neural networks, backpropagation and the ReLU activation function. If not, check
out the 'Quest. ALSO NOTE: Although basic, or vanilla recurrent neural networks are
awesome, they are usually thought of as a stepping stone to understanding fancier
things like Long Short-Term Memory Networks and Transformers, which we will talk
about in future StatQuests. In other words, every Quest worth taking take steps,
and this is the first step. So with that said, let's say Hi to StatSquatch. Hi! And StatSquatch says,
Hello! The other day I bought stock in a company called Get Rich Quick, but the next
day their stock price went down and I lost money. Bummer. So, I was thinking, maybe
we could create a neural network to predict stock prices. Wouldn't that be cool?
That sure would be cool 'Squatch, unfortunately the actual stock market is crazy complicated
and we'd probably both get in a lot of trouble if we offered advice on how to make
money with it, but if we go to that mystical place called StatLand things are much
simpler and there are far fewer lawyers. So let's build a neural network that predicts
stock prices in StatLand. However, first let's just talk about stock market data
in general. When we look at stock prices, they tend to change over time. For example,
the price of this stock went up for four days before going down. Also, the longer
a company has been traded on the stock market, the more data we'll have for it. For
example, we have more time points for the company represented by the blue line then
we have for the company represented by the red line. What that means is, if we want
to use a neural network to predict stock prices, then we need a neural network that
works with different amounts of sequential data. In other words, if we want to predict
the stock price for the Blue Line company on day 10, then we might want to use the
data from all nine of the preceding days. In contrast, if we wanted to predict the
stock price for the Red Line company on day 10, then we would only have data for
the preceding five days. So we need the neural network to be flexible in terms of
how much sequential data we use to make a prediction.
This is a big difference compared to the other neural networks we've looked at in
this series. For example, in Neural Networks Clearly Explained, we examined a neural
network that made predictions using one input value, no more and no less. And if
you saw the StatQuest on neural networks with multiple inputs and outputs, you saw
this neural network that made predictions using two input values, no more and no less.
And in the StatQuest on Deep Learning Image Classification, you saw a neural network
that made a prediction using an image that was six pixels by six pixels, no bigger
and no smaller. However, now we need a neural network that can make a prediction
using the nine values we have for the blue company and make a prediction using the
five values we have for the red company. The good news is that one way to deal with
the problem of having different amounts of input values is to use a Recurrent Neural
Network. Just like the other neural networks that we've seen before, recurrent neural
networks have weights, biases, layers and activation functions. The big difference
is that recurrent neural networks also have feedback loops. And, although this neural
network may look like it only takes a single input value, the feedback loop makes
it possible to use sequential input values, like stock market prices collected over time, to make predictions.
To understand how, exactly, this recurrent neural network can make predictions with
sequential input values, let's run some of StatLand's stock market data through it.
In StatLand, if the price of a stock is low for two days in a row, then, more often
than not, the price remains low on the next day. In other words, if yesterday and
today's stock price is low, then tomorrow's price should also be low. In contrast,
if yesterday's price was low and today's price is medium, then tomorrow's price should
be even higher. And when the price decreases from high to medium, then tomorrow's
price will be even lower. Lastly, if the price stays high for two days in a row,
then the price will be high tomorrow. Now that we see the general trends in stock
prices in StatLand, we can talk about how to run yesterday and today's data through
a recurrent neural network to predict tomorrow's price. The first thing we'll do
is scale the prices so that low equals 0, medium equals 0.5, and high equals 1.
Now let's run the values for yesterday and today through this recurrent neural network
and see if it can correctly predict tomorrow's value. Now, because the recurrent
neural network has a feedback loop, we can enter yesterday and today's values into
the input sequentially. We'll start by plugging yesterday's value into the input.
Now we can do the math just like we would for any other neural network. Beep. Boop.
Beep. Boop. Boop. At this point, the output from the activation function, the y axis
coordinate that we will call Y sub 1, can go two places. First Y sub 1 can go towards
the output. And if we go that way and do the math beep, boop, boop then the output
is the predicted value for today. However, we're not interested in the predicted
value for today because we already have the actual value for today. Instead, we want
to use both yesterday and today's value to predict tomorrow's value. So, for now,
we'll ignore this output, and instead, focus on what happens with this feedback loop.
The key to understanding how the feedback loop works is this summation. The summation
allows us to add Y sub 1 times W sub 2, which is based on yesterday's value, to the
value from today times W sub 1. In other words, the feedback loop allows both yesterday
and today's values to influence the prediction.
Hey, this feedback loop has got me all turned around. Is there an easier way to see how this works?
Yes! There's an easier way to see what's going on. Instead of having to remember which
value is in the loop, and which value is in the input, we can unroll the feedback
loop by making a copy of the neural network for each input value. Now, instead of
pointing the feedback loop to the sum in the first copy, we can point it to the sum
in the second copy. By unrolling the recurrent neural network, we end up with a new
network that has two inputs and two outputs. The first input is for yesterday's value,
and if we do the math straight through to the first output like we did earlier, we
get the predicted value for today. However, as we saw earlier, we can ignore this
output. This second input is for today's value and the connection between the first
activation function and the second summation allows both yesterday and today's values
to influence the final output, which gives us the predicted value for tomorrow. Now,
when we put yesterday's value into the first input and we do the math just like before
beep boop beep boop then we follow the connection from the first activation function
to the summation in the second copy of the neural network. Now we put today's value
into the second input and keep doing the math beep boop beep boop beep boop beep
and that gives us the predicted value for tomorrow, zero,
which is consistent with the original observation. In other words, the recurrent neural
network correctly predicted tomorrow's value.
Likewise,
when we run yesterday and today's values for the other scenarios through the recurrent
neural network we predict a correct values for tomorrow.
This recurrent neural network performs great with two days worth of data, but what if we have three days of data?
When we want to use three days of data to make a prediction about tomorrow's price,
like this, then we just keep unrolling the recurrent neural network until we have
an input for each day of data. Then we plug the values into the inputs, always from
the oldest to the newest. In this case, that means we start by plugging in the value
for the day before yesterday, then we plug in yesterday's value, and then we plug
in today's value. And when we do the math, the last output gives us the prediction
for tomorrow. NOTE: Regardless of how many times we unroll a recurrent neural network,
the weights and biases are shared across every input. In other words, even though
this unrolled network has three inputs the weight, W sub 1, is the same for all three
inputs. And the bias, B sub 1, is also the same for all three inputs. Likewise, all
of the other weights and biases are shared. So, no matter how many times we unroll
a recurrent neural network, we never increase the number of weights and biases that we have to train.
Okay, now that we've talked about what makes basic recurrent neural networks so cool,
let's briefly talk about why they are not used very often. One big problem is that
the more we unroll a recurrent neural network, the harder it is to train. This problem
is called The Vanishing / Exploding Gradient Problem. Which is also known as the "hey wait, where the gradient go?"
problem. In our example, The Vanishing / Exploding Gradient Problem has to do
with the weight along the squiggle that we copy each time we unroll the network. NOTE:
To make it easier to understand the Vanishing / Exploding Gradient Problem, we're
going to ignore the other weights and biases in this network and just focus on W
sub 2. Also, just to remind you when we optimize neural networks with backpropagation,
we first find the derivatives, or gradients, for each parameter. We then plug those
gradients into the gradient descent algorithm to find the parameter values that minimize
a loss function, like the sum of the squared residuals. Bam. Now, even though the
Vanishing / Exploding Gradient Problem starts with Vanishing, we're going to start by showing how a gradient can explode.
In our example, the gradient will explode when we set W sub 2 to any value larger than one.
So let's set W sub 2 equal to 2. Now, the first input value, input sub 1, will be
multiplied by 2 on the first squiggle and then multiplied by 2 on the next squiggle
and again on the next squiggle and again on the last squiggle. In other words, since
we unrolled the recurrent neural network four times,
we multiply the input value by W sub 2, which is 2, raised to the number of times
we unrolled, which is 4. And that means the first input value is amplified 16 times
before it gets to the final copy of the network. Now, if we had 50 sequential days
of stock market data, which to be honest, really isn't that much data, then we would
unroll the network 50 times, and 2 raised to the 50 power is a huge number. And this
huge number is why they call this an Exploding Gradient Problem.
If we tried to train this recurrent neural network with backpropagation, this huge
number would find its way into some of the gradients,
and that would make it hard to take small steps to find the optimal weights and biases.
In other words, in order to find the parameter values that give us the lowest value
for the loss function, we usually want to take relatively small steps. Bam. However,
when the gradient contains a huge number, then we'll end up taking relatively large
steps and instead of finding the optimal parameter we will just bounce around a lot.
Bummer. One way to prevent the Exploding Gradient Problem would be to limit W Sub
2 to values less than 1. However, this results in the Vanishing Gradient Problem. "Hey, wait, where the gradient go?"
To illustrate the Vanishing Gradient Problem, let's set W sub 2 to 0.5. Now, just
like before, we multiply the first input by W sub 2 raised to the number of times
we unroll the network. So if we have 50 sequential input values, that means multiplying
input sub 1 by 0.5 raised to the 50th power and 0.5 raised to the 50th power is a number super close to zero.
Because this number is super close to zero. This is called The Vanishing Gradient Problem.
Now when optimizing a parameter, instead of taking steps that are too large, we end
up taking steps that are too small. And as a result, we end up hitting the maximum
number of steps we are allowed to take before we find the optimal value.
Hey, Josh, these Vanishing / Exploding Gradients are a total bummer. Is there anything we can do about them?
Yes, and we'll talk about a popular solution called Long Short-Term Memory Networks in the next StatQuest.
Now, it's time for some Shameless Self Promotion.
If you want to review statistics and machine learning offline, check out my book.
The StatQuest Illustrated Guide to Machine Learning at statquest.org. It's over 300 pages of total awesomeness.
Hooray! We've made it to the the end of another exciting StatQuest. If you liked this
StatQuest and want to see more, please subscribe. And if you want to support StatQuest,
consider contributing to my patreon campaign, becoming a channel member buying one
or two of my original songs or a t-shirt or a hoodie, or just donate. The links are
in the description below. Alright, until next time Quest on!