CHRISTOPHER POTTS: Welcome, everyone.
This is part two in our series on contextual word
representations.
We're going to be talking about the transformer architecture,
which is the central piece for all the models we'll
be exploring in this unit.
Let's dive into the model structure,
we'll work through this using a simple example.
At the bottom here I've got the input sequence, "the Rock
rules" and I've indicated in red that we're
going to be keeping track of the positions
of each one of those tokens in the sequence.
But the first step is a familiar one.
We're going to look at both the words and the positions
in separate embedding spaces.
Those are fixed embedding spaces that we'll
learn as part of learning all the parameters in this model.
I've given the word embeddings in light
gray and the positional embeddings in dark gray.
To form what we think of as the actual embedding
for this model, we do an elementwise addition
of the word embedding with the positional embedding.
And that gives us the representations
that are in green here, and you can see that
on the right side of the slide, I'm
going to be keeping track of all of the calculations with regard
to this C column here and they're completely
parallel for columns A and B.
So to form C input, we do elementwise addition of X34,
the embedding for the word "rules"
and P3 which is the embedding for position 3 in this model.
The next layer is really the hallmark of this architecture
and what gives the paper its title,
Attention is All You Need.
We're going to form a bunch of dense dot product
connections between all of these representations.
So you can think of those as forming
these connections that look like there's
a dense thicket of them.
On the right here, I've given the core calculation
and it should be familiar from part one in this unit.
It's exactly the calculation I've presented there
with just two small changes, but fundamentally if our target
vector is the C input here, we're
attending to inputs A and B, and we do that
by forming the dot products here.
And the one twist from before is that instead of just taking
those dot products, we'll normalize them
by the square root of the dimensionality of the model dk.
dk is an important value here because of the way
we combine representations in the transformer.
All of the outputs and all the layers we look at
have to have the same dimensionality as given by dk.
And so what we're doing here is essentially
scaling these dot products to kind of keep them
within a sensible range.
That gives us a score vector off the tilde,
we softmax normalize that.
And then the other twist is that instead
of using mean here as we did before, we use summation.
But the actual vector is the one we calculated before, we're
going to take weighted versions of A input and B input
according to this vector of weights that we created here.
That gives us the representation C attention
as given in orange here, and we do
that of course for all the other positions in the model.
The next step is kind of interesting,
we're creating what's called a residual connection.
So to get CA layer here in yellow,
we add up C input and this attention representation
that we just created and apply dropout
as a regularization step there.
And that gives us CA layer, the interesting thing
there of course is this residual connection.
Instead of simply feeding forward C attention,
we feed forward actually a version of it
that's combined with our initial positionally encoded embedding.
If we follow that with a step of layer normalization, which
should help with optimization, it's
going to kind of scale the weights
in these representations.
The next step is more meaningful.
This is a series of two dense layers,
so we'll take Ca norm here and feed it
through this dense layer with a non-linearity followed
by another linear layer to give us Cfx
that's given in dark blue here.
And that's followed by another one
of these interesting residual connections,
so we'll apply dropout to Cff.
And then add that in with Ca norm,
as given down here at the bottom,
and that gives us the second yellow representation.
And we follow that by one more step of layer normalization
and that gives us the output for this block of transformer
representations.
And you can imagine of course as you'll
see that we can stack up these transformer blocks,
and the way that we do that is essentially
by taking these dark green representations at the top
here and using them as inputs and all the calculations
are the same.
So you might imagine that we could continue here
by just doing a dense series of attention
connections across these and then
continuing on with the calculations I just presented.
And in that way, we could stack up transformer blocks.
And I'll return to that later on.
There are a few other things that
are worth pointing out that are kind
of noteworthy about this model.
It looks like a complicated series of calculations,
but I would say that fundamentally what's
happening here is we're doing positional encoding to get
embeddings, so that they are position-sensitive
representations of words.
We follow that within an attention layer
which creates that dense thicket of connections
between all of the words as positionally encoded.
Then we have these optimization things woven in,
but fundamentally we're following that attention step
with two series of feed-forward layer steps
here, followed by the same process of dropout and layer
normalization.
So if you kind of elided the yellow and the purple,
you would see that what we're really doing
is attention followed by feed-forward.
And then as we stack these things that
would be attention feed-forward, attention
feed-forward as we climbed up.
And interwoven into there are some things
that I would say help with optimization.
Another noteworthy thing about this model
is that the only sense in which we
are keeping track of the linear order of the sequence
is in those positional embeddings,
if not for them the column order would be completely irrelevant.
Because of course, we've created all
of these symmetric connections at the attention layer.
And there are no other connections
across these columns.
So the only sense in which column order, that
is word order matters here, is via those positional
embeddings.
Here's a more detailed look at the attention calculations
themselves.
I just want to bring up how this actually
works at a mechanical level.
So this is the calculation as I presented it
on the previous slide and in part one of this unit.
In the paper, and now commonly it's
presented in this matrix format.
And if you're like me, it's not obvious
right away that these are equivalent calculations.
So what I've done for these next two slides
is to show you via worked out examples
how those calculations work and how they arrive
at exactly the same values.
I'm not gonna spend too much time on this here,
this is really just here for you if you
would like to work through the calculations in detail,
which I strongly encourage because this is really
the fundamental step in this model.
And here's all the details that you
would need to get hands on with these ideas.
Now so far, I've presented attention
in a kind of simplified way.
A hallmark of attention is in the transformer
is that it is typically multi-headed attention.
So let me unpack that idea a little bit concretely.
We'll start with our input sequence from before
and we'll be looking at these green representations here.
And the idea behind the multi-headed attention
mechanisms that we're gonna inject
a bunch of learned parameters into this process to encourage
diversity as part of the learning process,
and are really diverse and interesting representations.
So here's how that works.
We're gonna form three representations here using
that same dot product mechanism as before.
And fundamentally it's the same calculation except now
we're gonna have a bunch of learned weight parameters
that's given in orange here.
And those will help us with two things.
First injecting diversity into this process,
and also smushing the dimensionality
of the representations down to one third of the size
that we're targeting SDK for our model dimensionality.
And you'll see why that happens in a second.
But fundamentally what we're doing
is exactly the calculation we did before but now
with these learned parameters injected into it.
So if you squint, you can see that this is really
the dot product of c input with a input as before.
But now, it's transformed by these learned parameters
that are given in orange.
And that repeats for these other calculations.
So we're going to do that for position a.
And we do it also for position b.
And it's the same calculation, but now it's
new parameters for the second position.
And then for the third head, exactly the same calculation
but new learned parameters up at the top here.
So this is three-headed attention.
And then we actually form the representations that
proceed with the rest of the calculation of the transformer
architectures presented before is by concatenating the three
representations we created for each one of these units.
So the A column is the first representation
in each one of these heads.
The B column is the second representation in each head
and similarly for the C column it's
the third representation in each one of these heads.
And that's why each one of these needs
to add one-third the dimensionality
of our full model so that we can concatenate them and then
feed those into the subsequent calculations.
The idea here, of course, is that injecting all of these
learned parameters into all of these different heads we're
providing the model a chance to learn lots of diverse ways
of relating the words in the sequence.
The final point is one I've already mentioned before,
which is that typically we don't have just one transformer
block but rather a whole stack of them,
we can repeat them N times.
For models you're working with you might have 12 or 24 or even
more blocks in the transformer architecture.
And the way we do that as I said is simply
by taking the dark green representations of the output
layer here and using them as inputs to subsequent blocks
so they get attended to, and we proceed
with the subsequent regularization
and feed-forward steps just as before.
And when you work with these models in Hugging Face,
if you ask for all of the hidden states, what you're getting
is a grid of representations corresponding to these output
blocks in green here.
And of course, just as a reminder
I'm not indicating it here but there is actually
multi-headed attention of each one of these blocks
through each one of the layers.
So there are a lot of learned parameters in this model,
especially if you have 12 or 24 attention heads.
At this point, I'm hoping that you can now fruitfully
return to the original Vaswani, et al paper
and look at their model diagram and get more out of it.
For me, it's kind of hyper compressed but now
that we've done a deep dive into all the pieces,
I think this serves as a kind of useful shorthand for how
all the pieces fit together.
So let's just do that quickly.
As before we have positional encodings and input word
embeddings, and those get added up
to give us the intuitive notion of an embedding in this model.
That's followed by the attention layer as we discussed,
and it has a residual connection here into that layer
normalization part.
That's fed into the feed-forward blocks
and that's followed by that same process of kind of dropout
attention layer normalization.
And this is essentially saying each one
of these is repeated for every step in the encoder
process, every one of the columns that we looked at.
And if because then in the paper,
they're working with an encoder/decoder model.
Each decoder state self attends with all of the fellow decoder
states and with all of the encoder states, so right?
Imagine this double sequence, we have a dense area
to potentially connect some connections across both parts
of the representation.
On the right side when we're doing decoding,
again, we repeat this block for every decoder state.
And if every state there has an output as for machine
translation or some kind of generation process,
then we'll have something like this output
stack at every one of those output states.
If by contrast we're doing something
like NLI or sentiment classification problem,
maybe just one of those states will have
one of these outputs on them.
And then the attention gets a little bit complicated
if you are doing decoding for a model like natural language
generation or machine translation, in the decoder,
you can't attend into the future as you're doing generation.
So there's a masking process that
limits self-attention to the preceding words in the sequence
that you're creating.