Seen this movie called Memento or there
is a
bollywood movie called Ghajni? our basic
RNNs
are like the heroes of that movie they
suffer from a short-term memory problem
LSTM is a special version of RNN
which solves the short-term memory
problem and in this video I will explain
LSTM
in a very simple manner using real-life
examples
let's say you have NLP task for centers
completion
here in both the sentences based on what
word
appeared in the beginning your
autocomplete sentence might be different
for example for the first one I would
say I need to take a loan
whereas for the second one I would say I
had to take a loan
and this decision between need and head
was
made based on what appeared at the very
beginning
now looking at our traditional RNN and
architecture
if you have seen my previous videos you
would know
this is how RNN architecture looks like
so if you're not seeing those videos I
highly recommend you
watch them because they are kind of a
prerequisite
here when you feed the sentence word by
word
okay so first you will feed today it
will learn some weight
there is an activation which is fed back
now you uh work on the second word which
is due
then the third word which is two so this
is how
basic RNN works if you unroll
this thing in time then this is how the
architecture will look like. Many people
get confused they think that
this is a neural network with so many
layers. Actually there is only one layer
look at this time axis t1 t2 t3
so it's the same layer represented in
different time
and when you unroll it this is how it
looks.
now to predict this word need
it needs to know about the word today
which appeared at the very beginning of
the sentence
and because of vanishing gradient
problem
the traditional RNNs have short-term
memory
so they don't remember what appeared in
the beginning of the sentence
they have a very short-term memory of
just few
words which are like nearby, hence
to autocomplete this kind of sentence
RNN won't be able to do
a good job. similarly this is the second
sentence you know where
head was derived based on the earlier
word which is last year
now let's look at the network layer
a little more in detail so I'm going to
just expand
this particular network layer which
looks like this. so there are set of
neurons
in that layer and this hidden state is
nothing but a short-term memory.
okay so i'm just going to remove this
neurons just to kind of make it simple
and this square box is called a memory
cell
because this hidden state is actually
containing
the short-term memory. now if you want to
remember
long-term memory we need to introduce
another state called
long-term memory. so that state is called
C so there are two states now hidden
state which is short-term memory
and the there is a self-state which is a
long-term memory. and we will look into
this in detail how exactly this will
work
but going back to our short-term memory
cell
in traditional RNN if
it looks something like this so I have
drawn the vertical neurons here but
you can draw on draw them this
in a horizontal fashion as well so it's
just a layer of neurons and your x(t) and
h(t) are vectors
so when you have a word for example you
will first convert into a vector vector
is nothing but a
list of numbers and your hidden state
will be
also a vector and using both these
vectors you will do
you know like sigma operation like
weighted multiplication and then you
apply
activation function which is which is
tan h in the case of RNN
and then you get a new hidden state so
here is a simplistic example
right so here you have 10 h you have
weighted sum going on
and this is how the short term memory
cell looks in traditional RNN
in LSTM we are going to introduce a new
uh cell state for a long term memory so
let's say there is this cell state okay
now let's see how exactly this works by
looking at one more example
I love eating samosa by the way so
i have one more sentence for you to auto
complete
can you tell me what would you put here
at dot dor dot
well obviously Indian. Samosa is an
Indian cuisine so you will say
his favorite cuisine is Indian now
take a pause and think about this when
is a human
when you make this case when you are
processing this sentence
which words uh told you that this will
be an Indian cuisine
well it is this word samosa if i didn't
have samosa here
if i had just had eat or every day it
so based on these words you can't guess
it is an Indian cuisine, right?
there is some key words if you are doing
a movie review for example
you're looking for key words like okay
excellent
or horrible you know terrible movie
or amazing the hero performed very well
so you're just looking
for specific words and the remaining
words you can actually ignore
now let's see how our traditional RNN
would behave for this sentence
so a traditional RNN which is like Amir
Khan of ghajni
having short-term memory when you feed
all these words
it can remember only let's say last two
words in reality RNNs can remember more
words but i'm just giving a simple
example you know just to explain this
concept
so let's say they have short-term memory
just remembers only two words
so when you are read this sentence
almost for example
it remembers almost and the samosa like
the two words last two words
when you are at here cuisine is it will
remember
is and cuisine so at this point it
doesn't have a knowledge of samosa
so then for a traditional RNN it is hard
to make a guess that the cuisine is
Indian
what if we build this long-term memory
along with short-term memory in such a
way that
we store the meaningful words or the key
words
into this long-term memory so when I
feed the world or
eats it will not store it this is a
blank string
it will not store it in a long-term
memory
but when you find things like samosa it
will store it in a long-term memory see
samosa
when you get almost almost is also not
important so i just store samosa here
and when i go all the way here now when
i have to make a prediction on cuisine
I have that memory that this we are
talking about samosa and hence it has to
be Indian.
Let's look at little more complicated
example
here while I love indian cuisine my
brother bhavin lost
which cuisine, do you have any guess? If
you read this sentence
carefully you will figure it is an
italian cuisine
and you made that decision based on the
two key words which were
pastas and cheese so again you are
reading the sentence
and you are keeping some keywords in
memory and throwing everything out like
and that means these are not important
for you the the important keywords are
pastas and cheese
based on that you make a decision that
it is an Italian cuisine
so going back to our RNN with
long-term memory when you encounter
let's say samosa, you will store samosa
in your long-term memory
but you will keep on storing samosa
until
you encounter pasta so now the moment
you
encounter pasta you need to forget
the previous long-term memory which is
samosa so here I
threw that thing out and I have new
memory
which is pasta and then you keep on
preserving this
until you hit cheese so when you hit
cheese
you need to add that so now you can't
ignore pasta you need to add cheese
on top of it and then in the end
when you are about to be asked
you know what is your answer for auto
complete you will say Italian because
you have the memory of pasta and cheese
now you would ask me how do you
make this decision how do you let's say
when cheese comes
you don't discard pasta but when pasta
pasta comes you discard samosa
well all of this happens during the
training process so when you're training
your
RNN you are not giving only this
particular statement this is this is a
statement for prediction
when the training is happening you are
giving thousands and thousands of such
statements
which will build that understanding in
RNN uh
on what to discard and what to store
so here we learned a very important
concept okay
so when you're talking about LSTM or a
long
short term memory cell so each of these
cells are LSTM cells
the first most important thing is the
forget get.
So the role of forget get is when you
come
at this word pasta it knows that it has
to discard
samosa okay so this is how forget get
looks like so I have just expanded that
cell
xt is a one word you know you process
sentence one word by one word
and t is the timestamp so that's why x(t)
so when you feed pasta into this
forget get so the forget get is simple
you have previous hidden state you take
the current input which is your current
word
and you apply sigmoid function now you
know that sigmoid function
restricts your number between 0 and 1 so
it will
if it has to discard the previous memory
it will
output a vector which will have all
zeros or
all the values which are close to zero
and when you multiply that with
previous memory which is previous cell
state
you know you have a vector of let's say
all zeros here and you have another
vector which is a memory of
previous cell state. the
multiplication will
of course be zero because you have
discarded
the previous memory. you know when this
new word appeared
so this is what a forget get looks like
so here you forgot about samosa now
there is another
thing which is an input gate so when
pasta came
not only you forgot about samosa you
need to add a memory of pasta
so the way it will work is
you will use sigmoid
and tan h both on these two vectors okay
so when you're doing by the way these
vectors will have weights here
so there will be some weight here some
weight here so in this
function what you are doing is h t minus
1 into that weight
plus x t into that weight
plus bias and then you are applying 10 h
on top of it
and it's the same equation here the only
difference is instead of tan h you are
using sigmoid function
and you multiply both of this output and
then add
that as a memory for this word
the third one is output gate so in the
output gate again you are doing weighted
sum
of hidden state nxt
and applying sigmoid function whatever
is the output
uh you take that and then you take long
term memory apply 10 h and you multiply
that
and you will get your hidden state okay
so this will be your hidden state there
are cases like
let's say the sentence auto complete
case that we're looking at
at there is no yt actually
the state is carried using the
short-term memory but
you know if there is a task like a named
entity recognization
there you need y t which is an output and
that output is same as
h t i mean you might apply sigmoid uh not
sigmoid but
let's say softmax type of function here
uh but other than that
it is kind of similar to h d so now if
you think about the long-term memory
so long-term memory look at this line
okay look at this highway for example
it has two things forget get an input
brigade so forget get is like it will
help you
forget things like samosa when pasta
comes in
and the input gate will add
new things into memory like meaningful
things you want to add in a memory
if it is a movie review you want to add
like horrible or amazing
beautiful those kind of words you know
is the
there are so many words you don't care
about so that's what this
lSTM will do and i'm going to refer you
to a nice
article i personally found this article
to be very useful I'm going to link in
the video description below
I did not mention any mathematical
equation because
you can use this article for any math c
here
they explain how ct is calculated in
terms of math formulas
all math formulas again I suggest you
read this article properly because I
myself have learned a lot from this
and I hope you found this video
useful if you did please give it a
thumbs up
share it with your friends I think
this
this can provide you a simple
explanation
of LSTM and in the
coming videos we'll be doing coding on
an
LSTM we will also be going over
GRU in the future videos all right so
till then
thank you very much and bye