Hello, and welcome.
In this video, we will provide an overview of the Long Short-Term Memory Model.
The recurrent neural network is a great tool for modeling sequential data, but there are
a few issues that need to be addressed in order to use the model at a large scale.
Recurrent nets need to keep track of states, which is computationally expensive.
There are also issues with training like the vanishing gradient, and the exploding gradient.
A popular method to solve these problems is the Long Short-Term memory unit, or LSTM for short.
LSTM is an abstraction of computer memory that works in tandem with a recurrent net.
It's used to maintain, update, and regulate the states of the network model, outside of
its normal execution flow.
The LSTM units are composed of four main elements.
The "memory cell", or "information cell", is responsible for holding data, and the three
logistic "gates", define the flow of data inside the LSTM.
The Write Gate, or the Input Gate, is responsible for writing data into the memory cell.
The Output, or Read, Gate, reads data from the information cell and sends that data back
to the recurrent network.
And the Keep Gate, or Forget Gate, maintains or deletes data from the information cell.
These gates have a similar structure to the neurons in a traditional neural network, since
they are multiplicative analog sigmoid-activated nodes.
By manipulating these gates, a Recurrent Network is able to remember what it needs, and forget
what is no longer useful.
This is what a recurrent network would look like, if we unfolded it at each time step.
The LSTM cells exist between the time steps in the network's flow.
So the network's output can be input into the LSTM, and the LSTM can send its output
to the net's input in the next time step.
Let's take a closer look at an LSTM unit's structure.
The data input and the network's state are directly connected to all the gates.
So each gate receives the same information as the recurrent net's processing cell.
Additionally, the network's output is connected to the Write gate, and the Read gate is connected
to the network's processing cell.
The Write gate sends its output to the information cell, and the information cell can send data
into the Read gate.
These gates are all analogue logistic units, which means their behavior is defined by a
logistic function.
For example, a value of 0 in the Keep gate means "forget everything", a value of 1 means
"retain everything", and a value of 0.7 means "keep exactly 70 percent of the current data".
This applies to all three of the gates.
For now, let's just focus on the individual Gates themselves.
The Keep Gate is responsible for maintaining the data in the information cell.
It receives the same input data and state as the network, and then it calculates how
much of the current data should be remembered.
The Write Gate is responsible for inputting new data into the memory cell.
Like the Keep gate, it receives the same input data and state as the recurrent network.
However, it also receives the recurrent net's output data from the most recent time step.
The Write Gate uses the inputs to determine how much of the output data should be written
into the memory cell.
The Read Gate is responsible for sending data from the LSTM back into the Recurrent network.
Like the other gates, it receives the inputs and state from the network.
The Read Gate reads a value from the information cell, and this value is interpreted as a signal
between -1 and 1.
The input data and state are then used to determine how much of this signal should be
sent to the Recurrent network.
Working in tandem, all three gates allow the model to freely regulate and manipulate the
data inside the information cell.
But you might be wondering why the LSTM uses logistic gates.
The reason is that logistic-based functions have very nice derivatives.
As a result, we can backpropagate through the gates, which enables the recurrent net
to understand how to utilize the LSTM structure.
But the details of this process are beyond the scope of this video.
The main point is that by manipulating values through the gates, we eliminate the vanishing
gradient and exploding gradient problems from before.
And since the gates allow the network to forget the states that are no longer needed, the
computational load of the recurrent model decreases drastically.
By now, you should understand the structure and purpose of the Long Short-Term memory model.
Thank you for watching this video.