to encode you unroll to decode you
unroll stat Quest
hello I'm Josh Darman welcome to
statquest today we're going to talk
about seek to seek and encoder decoder
neural networks and they're going to be
clearly explained
it's the easiest way to scale your work
up in the cloud lightning
this stat Quest is also brought to you
by the letters a b and c a always b b c
curious always be curious
note this stat Quest assumes that you
are already familiar with long
short-term memory neural networks and
word embedding if not check out the
quests
also I want to give a special triple bam
thanks to Ben Trevitt his awesome
tutorials on GitHub the links are in the
description below made it possible for
me to create this Quest
Hey look it's stat Squatch and the
normal saurus hey Josh I want to tell my
friend in Spain let's go but I don't
know Spanish can you help sure thanks
Squatch
I've got a sequence of amino acids that
I want translated into 3D structures
like Alpha helices what I've got a
sequence of amino acid that I want
translated don't worry Norm I can help
both of you have sequences of one type
of thing that need to be translated into
sequences of another type of thing
both of these problems and many others
are called sequence to sequence or seek
to seek problems one way to solve seek
to seek problems is with something
called an encoder decoder model which
we'll talk about in this stack Quest
damn
note the basic seek to seek encoder
decoder models that we're going to talk
about today are totally awesome but they
are also a stepping stone to learning
about Transformers which we'll talk
about in future stat quests in other
words today we're taking another step in
our quest to understand Transformers
which form the basis of big fancy large
language models like Chachi BT anyway to
illustrate how to solve a sequence to
sequence problem using a seek to seek
encoder decoder model let's create one
that translates English sentences into
Spanish
specifically Squatch wants to tell his
Spanish friend let's go so we'll create
an encoder decoder model that can do
this first however let's talk a little
bit more about the problem we want to
solve
the first thing which is pretty obvious
is that not all sentences in the English
language are the same length
for example someone might say
let's go
or they might say
my name is statsquatch
so we need something that can take
different length sentences as input
likewise not all Spanish sentences are
the same length so we need something
that can generate different length
sentences as output
lastly this Spanish translation of an
English sentence can have a different
length than the original
for example the two-word English
sentence let's go translates to the one
word Spanish sentence vamos
so we need our seek to seek encoder
decoder model to be able to handle
variable input and variable output
lengths
the good news is that we already know
how to use long short-term memory units
to deal with variable length inputs and
outputs
for example if the input sentence is
let's go
then we put lets into the input for the
lstm and then unroll the lstm
and then plug go into the second input
however we're getting ahead of ourselves
if you remember from the word embedding
stack Quest we can't just Jam words into
a neural network
instead we use an embedding layer to
convert the words into numbers
note to keep the example relatively
simple the English vocabulary for our
encoder decoder only has three words
let's to and go and this symbol EOS
which stands for end of sentence
oh no it's the dreaded terminology alert
because the vocabulary contains a mix of
words and symbols we refer to the
individual elements in the vocabulary as
tokens
also note in this example we're just
creating two embedding values per token
instead of hundreds or thousands
okay now that we have an embedding layer
for our input vocabulary
we can put it in front of the input for
the lstm
now when we have the input sentence
let's go
we put a 1 in the input for lets and a
zero for everything else
and then we unroll the lstm and the
embedding layer
and put a 1 in the input for go and a 0
for everything else
note to be clear when we unroll the lstm
and the embedding layer we reuse the
exact same weights and biases no matter
how many times we unroll them
in other words the weights and biases in
the lstm cell and embedding layer that
we use for the word let's are the exact
same weights and biases that we use for
the word go
now in theory this is all we need to do
to encode the input sentence let's go
bam
however in practice in order to have
more weights and biases to fit the model
to our data people often add additional
lstm cells to the input
to keep things simple we'll just add one
additional lstm cell to this stage this
means that the two embedding values for
the word let's are used as the input
values for two different lstm cells
and these two lstm cells have their own
separate sets of weights and biases
and when we unroll them for the word go
the original lstm cell reuses its set of
weights and biases and the new lstm cell
reuses its separate set of weights and
biases
now to add even more weights and biases
to fit the model to our data people
often add additional layers of lstms
to illustrate how this works we'll add
one more lstm layer to the encoder
what that means is that the output
values the short-term memories or the
hidden States from the unrolled lstm
units in the first layer are used as the
inputs to the unrolled lstm units in the
second layer
note just like how both embedding values
are used as inputs to both lstm cells in
the first layer
both outputs the short-term memories or
hidden States from each cell in the
first layer are used as inputs to both
lstm cells in the second layer
lastly the only thing left to do is to
initialize the long and short-term
memories the cell and hidden States
and now we're done creating the encoder
part of the encoder decoder model in
this example we have two layers of lstms
with two lstm cells per layer
in essence the encoder encodes the input
sentence let's go into a collection of
long and short-term memories also known
as cell and hidden States bam
oh no it's the dreaded terminology alert
again
the last long and short-term memories
the cell and hidden States from both
layers of the lstm cells in the encoder
are called the context vector
thus the encoder encodes the input
sentence let's go into the context
vector now we need to decode the context
vector so the first thing we do is
connect the long and short term Memories
the cell in Hidden states that form the
context Vector to a new set of lstms
that just like the encoder have two
layers and each layer has two cells
note to be clear the lstms in the
decoder are different from the ones in
the encoder and have their own separate
weights and biases anyway the context
Vector is used to initialize the long
and short-term memories the cell in
Hidden states in the lstms in the
decoder and the ultimate goal of the
decoder is to decode the context factor
into the output sentence
so just like in the encoder the input to
the lstm cells in the first layer comes
from an embedding layer
however now the embedding layer creates
embedding values for the Spanish words
ear vamos and E and the EOS end of
sentence symbol
in other words this is the embedding
layer that we use in the encoder
and this is the embedding layer that we
use in the decoder
as you can see they have different input
words and symbols or tokens
and different weights which result in
different embedding values for each
token now because we just finished
encoding the English sentence let's go
the decoder starts with the embedding
values for the EOS end of sentence token
in this case we're using the EOS token
to start the decoding because that is
what they used in the original
manuscript
however sometimes you'll see people use
SOS for start of sentence
anyway the decoder does the math with
the two layers of lstms each with two
lstm cells
and the output values from the top layer
of lstm cells
are transformed by additional weights
and biases in what is called a fully
connected layer
a fully connected layer is just another
name for a basic vanilla neural network
this fully connected layer has two
inputs for the two values that come from
the lstm cells in the top layer
and four outputs one for each token in
the Spanish vocabulary
and in between we have connections
between each input and output with
weights and biases
then we run the output of the fully
connected layer through a soft Max
function to pick out the output word
now going back to the full encoder
decoder model
we see that the output from the softmax
function is vamos the Spanish
translation for Let's go double bam not
yet
so far the translation is correct but
the decoder doesn't stop until it
outputs an EOS token
so we plug the word vamos into the
decoder's unrolled embedding layer
and unroll the two lstm cells in each
layer
and then run the output values into the
same fully connected layer
and the next predicted token is eos
and that means we translated the English
sentence let's go into the correct
Spanish sentence vamos double bam
to summarize the decoder stage
the context Vector created by both
layers of the encoder's unrolled lstm
cells are used to initialize the lstms
in the decoder
and the input to the lstms comes from
the output word embedding layer that
starts with EOS
but subsequently uses whatever word was
predicted by the output layer in
practice the decoder will keep
predicting words until it predicts the
EOS token or it hits some Maximum output
length note by decoupling the encoder
from the decoder the input text and the
translated output text can be different
lengths in this case we translated the
two word English sentence let's go to a
one word Spanish sentence vamos
bam
note just like for all neural networks
all of these weights and biases are
trained using back propagation and if
you're not already familiar with back
propagation check out the quests
that said encoder decoder models have
two special things that happen during
training
in this example we use the predicted
token vamos as the input to the unrolled
lstms
in contrast when training an encoder
decoder instead of using the predicted
token as input to the decoder lstms we
use the known correct token
in other words if the first predicted
token was the Spanish word e which
translates to and in English and thus is
the wrong word
then during training we'll still use
vamos the correct Spanish word as input
to the unrolled lstms
also during training instead of just
predicting tokens until the decoder
predicts the EOS token
each output phrase ends where the known
phrase ends
in other words even if the second
predicted token was the Spanish word ear
instead of the correct token EOS then
during training will still stop
predicting additional tokens
plugging in the known words and stopping
at the known phrase length rather than
using the predicted tokens for
everything is called teacher forcing
which sounds vaguely like what happened
when I took introduction to statistics
in school
lastly let's talk about the differences
between this super simple encoder
decoder model
and the model used in the original
sequence to sequence manuscript
first instead of just using three words
and one symbol or a total of four tokens
in each input and output vocabulary
the original manuscript had an input
vocabulary with 160 000 tokens
and an output vocabulary with eighty
thousand tokens
also instead of just two embedding
values per token the original manuscript
created one thousand
and instead of two layers of lstms with
two lstm cells in each layer
the original manuscript used four layers
with one thousand lstm cells per layer
also the output layer had one thousand
inputs from the 1000 lstm cells in the
fourth layer
and eighty thousand outputs to match the
size of the output vocabulary
lastly my simple model which I coded in
pi torch lightning only has 220 weights
and biases to train but the model in the
original manuscript had 384 million
weights and biases to train
and that gives you a sense of the scale
that these models can have
triple bam
now it's time for some Shameless
self-promotion if you want to review
statistics and machine learning offline
check out the statquest PDF study guides
and my book The stackquest Illustrated
guide to machine learning at
stackquest.org there's something for
everyone
hooray we've made it to the end of
another exciting stat Quest if you like
this stat Quest and want to see more
please subscribe and if you want to
support statquest consider contributing
to my patreon campaign becoming a
channel member buying one or two of my
original songs or a t-shirt or a hoodie
or just donate the links are in the
description below alright until next
time Quest on