CHRISTOPHER POTTS: Welcome, everyone.
This is part 5 in our series on natural language inference.
We're going to be talking about attention mechanisms.
Attention was an important source of innovation in the NLI
literature, and, of course, it's only
grown in prominence since then.
Let's begin with some guiding ideas.
In the context of the NLI problem,
we might have an intuition that we just need more connections
for a lot of our architectures between the premise
and hypothesis, right?
Possibly, in processing the hypothesis,
we just need the model to have some reminders about what
the premise actually contained.
And whatever summary representation we have
of that premise might just not be
enough from the point of view of processing the hypothesis
and feeding the representation into the classifier layer.
Relatedly, there's a persistent intuition in the NLI literature
that it's useful to softly align the premise and the hypothesis
to find corresponding words and phrases between those two
texts.
It can be difficult to do that at a mechanical level,
but attention mechanisms might allow
us, via our data-driven learning process,
to find soft connections in the weights for these attention
layers between the premise and hypothesis
and achieve some of the effects that we would get
from a real alignment process.
So let's begin with global attention.
This is the simplest attention mechanism
that you see in the NLI literature,
but it's already quite powerful.
And as you'll see, it has deep connections with the attention
mechanisms in the transformer.
So to make this concrete, let's start with a simple example.
We have "every dog danced" as our premise,
"some poodle danced" as our hypothesis,
and they're fit together into this chained RNN model for NLI.
Now, standardly, what we would do
is take this final representation, hC,
as the summary representation for the entire sequence
and feed that directly into the classifier.
What we're going to do when we add attention mechanisms is
instead offer some connections back from this state
into the premise states.
The way that process gets started
is via a series of dot products.
So we're going to take our target vector,
hC, and take its dot product with each one
of the hidden representations corresponding to tokens
in the premise.
And that gives us this vector of unnormalized scores,
just the dot products.
And it's common, then, to softmax
normalize those scores into our attention weights, alpha.
What we do with alpha is then create our context vector,
and the way that happens is that we're
going to get a weighted view of all those premise states.
Each one-- h1, h2, and h3--
is weighted by its corresponding attention weight, which
is capturing its kind of unnormalized notion
of similarity with our target vector hC.
And then to get a fixed dimensional version of that,
we take the mean-- or it could be the sum--
of all of those weighted views of the premise.
Next, we get our attention combination layer,
and there are various ways to do this.
One simple one would be to simply concatenate our context
vector with our original context--
target vector hC and feed those through a kind of dense layer
of learned parameters.
Another perspective, kind of similar,
is to give the context vector and our target vector
hC, each one their own weights, and have
an additive combination of those two,
and, again, feed it through some kind of non-linearity.
And you could think of various other designs for this.
And that gives us this attention combination, h tilde.
And then, finally, the classifier layer
is a simple dense layer, just as before, except instead
of using just hC, we now use this h tilde representation,
which incorporates both hC and that kind of weighted mixture
of premise states.
It might be useful to go through this
with some specific numerical values here.
So what I've done is, just imagine
that we have two-dimensional representations for all
of these vectors.
And you can see, what I've done here is kind of ensure that,
proportionally, "every" is a lot like this final representation
here.
And then that kind of similarity drops off
as we move through the premise states,
and you'll see what happens when we take the dot products here.
So the first step gives us the unnormalized scores,
and you can see that the highest unnormalized similarity is
with the first token, followed by the second, and then
the third.
The softmax normalization step kind of just flattens out those
dot products a little bit, but we
get the same proportional ranking with respect to hC.
Here's that context vector, and you
can see it's just a mean of the weighted values of all
of these vectors.
That gives us k.
And that k is then fed into this attention combination layer.
And you can see, in orange here, this
is the context vector, two dimensions.
Down here, we have hC, just faithfully repeated.
And then this matrix of weights Wk
is going to give us, in the end, after this non-linearity, h
tilde.
And then the classifier is as before.
So that's a simple worked example
of how these attention mechanisms work.
And the idea is that we are kind of fundamentally weighting
this target representation hC by its similarity
with the previous premise states.
But all of them are mixed in, and the influence
is kind of proportional to that unnormalized similarity.
There are other scoring functions that you could use,
of course.
We've just done a simple dot product up here,
but you can also imagine having learned parameters in there
or doing concatenation of the learned parameters.
This does a kind of bilinear form,
and this is just a concatenation of those two states fed
through these learned weights.
And once you see this kind of design space,
you can imagine there are a lot of other ways
in which you could mix in parameters
and have different views of this global attention mechanism.
We could go one step further here.
That was global attention.
In word-by-word attention, we're going
to have a lot more learned parameters and a lot
more connections between the hypothesis back
into the premise.
So to make this kind of tractable,
I've picked one pretty simple view of how this could work.
And the way we should track these computations
is focus on this vector B here, because we're
going to move through time.
But let's imagine that we've already processed the A state,
and we will subsequently process the C state.
So we're focused on B.
And the way we establish these connections
is by taking the previous context
vector that we've created.
That's kA here.
We're going to multiply that by repeated copies of the B state,
and that's simply so that we get the same dimensionality
as we have in the premise over here,
where I've simply copied over into a matrix all three
of those states.
And we have a matrix of learned parameters here
and an additive combination of the two,
followed by a non-linearity.
That's going to give us this M here, which kind of corresponds
to the attention weights in the previous global attention
mechanisms.
We're going to softmax normalize those,
and that literally gives us the weights.
And you can see that there are some additional parameters
in here to create the right dimensionalities.
And then, finally, we have the context at B.
So that's going to be a repeated view of all these premises,
weighted by our context vector, as before,
and then fed through some additional parameters Wa here.
And that gives us, as you can see here, the context
representation for the state B.
When we move to state C, of course,
that will be used in the place of A here.
And C will go in for all these purple values,
and the computation will proceed as before.
And in that way, because we have all of these additional learned
parameters, we can meaningfully move
through the entire sequence, updating our parameters
and learning connections from each hypothesis token back
into the premise.
So it's much more powerful than the previous view, where
we had relatively few learned parameters in our attention
mechanisms, and, therefore, we could only really meaningfully
connect that from the state that we're going
to feed into the classifier.
So this is much more expressive, right?
And then once we have done the entire sequence processing,
finally, we get the representation for C
here, as fed through these mechanisms,
and that becomes the input to the classifier
that we ultimately use.
The connection with the transformer should be apparent.
This is going to return us back to the global attention
mechanism.
Recall that for the transformer, we
have these sequences of tokens with
their positional encodings.
That gives us an embedding here.
And at that point, we establish a lot
of dot product connections.
And I showed you in the lecture on the transformer
that the mechanisms here are identical to the mechanisms
that we used for dot product attention.
It's just that in the context of the transformer,
we do it from every state to every other state.
And then, of course, the computations
proceed through subsequent steps in the transformer layer
and on through multiple transformer layers,
potentially.
And there are some other variants, right?
This is just the beginning of a very large design
space for attention mechanisms.
Let me just mention a few.
We could have local attention.
This was actually an early contribution
in the context of machine translation.
And this would build connections between selected points
and the premise and hypothesis, based
on some possibly a priori notion we have
of which things are likely to be important for our problem.
Word-by-word attention, as I've said,
can be set up in many ways, with many more learned parameters.
And the classic paper is the one that I'm
recommending for reading for this unit, Rocktaschel
et al., where they do a really pioneering view of this
in using even more complex attention mechanisms than I
presented under word-by-word attention
but following a lot of the same intuitions, I would say.
The attention representation at a time t
could be appended to the hidden representation
at time t plus 1.
This would give us another way of moving sequentially
through the sequence, having meaningful attention
at each one of those points, as opposed
to the global attention, which would just
be for that final state.
And then there are other connections even
further afield.
For example, memory networks can be
used to address similar issues, and they
have similar intuitions behind them as attention mechanisms
as applied to the NLI problem.
And that's kind of more explicitly drawing on this idea
that we might, in late states in processing,
need a bit of a reminder about what
was in the previous context that we processed.