CHRISTOPHER POTTS: Hello, everyone,
welcome to our final screencast in our unit
on distributed word representations.
Our topic is going to be deriving static representations
from contextual models.
That might sound awfully specific, but as you'll see,
I think this could be really empowering for you
as you work on your original system for your assignment
and the associated bake-off.
So let's dive in.
A question on your minds might be,
how can I use BERT or related models like RoBERTa or XLNet
or ELECTRA in the context of deriving
good static representations of words?
You probably have heard about these models
and heard that they lift all boats and the question
is, how can you take advantage of those benefits?
But there's a tension here.
We've been developing static representations,
but these models like BERT are designed
to deliver contextual representations of words.
And I'll return to what that means in a second,
but that is the central tension between static and contextual.
So the question is, are there good methods
for deriving static representations
from the contextual ones that these models offer?
And the answer from Bommasani et al is yes.
They are effective methods for doing this
and it's those methods that will be
the focus of this screencast.
I really want to do two things though for this lecture.
I would like to get hands on a little bit
with a high-level overview of models like BERT.
We're going to look later in the quarter in much more detail
at how these models work.
So for now, we're just going to treat them
as kind of black box.
It's just like you might look up a GloVe representation
of a word and just get back that representation and use it.
So too here, we can think of these models as devices
for feeding in sequences and getting back
lots and lots of representations that we might use.
And later in the quarter, we'll come
to a deeper understanding of precisely
where those representations come from.
And in addition, of course, I want
to give you an overview of these exciting methods from Bommasani
et al in the hopes that they are useful to you
in developing your original system.
So let's start with the structure of BERT.
BERT processes sequences, here I've
got a sequence, the class token, the day broke, SEP, class
and separate designated tokens, the class token typically
starts the sequence and then SEP ends the sequence.
It can be also used internally in sequences
to mark boundaries within the sequence
that you're processing.
But the fundamental thing is that we
have the short sentence, "the day broke."
BERT processes those into an embedding layer and then a lot
of additional layers.
And here I depicted 4, but it could be 12 or even 24 layers.
What we're seeing here, the rectangles represent vectors.
They are the outputs of each layer in the network.
A lot of computation goes into computing those output vector
representations at each layer.
We're going to set that computation aside for now
so that we can just think of this as a grid of vector
representations.
Here is the crucial thing that makes BERT contextual.
For different sequences that we process,
we will get very different representations.
In fact, individual tokens occurring
in different sequences will get very different representations.
I've tried to signal that with the colors
here so like the two sequences both contain the word "the"
and the word "broke."
But in virtue of the fact that they have different surrounding
material and different positions in the sequence,
almost all of the representations
will be different.
The class and SEP tokens might have the same embedding,
but through all of these layers because of the way
all these tokens are going to interact with each other
when we derive the representations,
everything will be different.
We do not get a static representation out
of these models.
And I've specified that even in the embedding
layer, if the positions of the words vary,
one and the same token will get different representations.
The reason for that is that this embedding layer is actually
hiding two components.
We do at the very center of this model
have a fixed static embedding, where we can look up
individual word sequences.
But for this thing that I've called the embedding layer,
that static representation is combined
with a separate positional encoding
from a separate embedding space, and that
delivers what I've called the embedding layer here.
And that means that even at this first layer
because, for example, "the" occurs
in different points in the sequence,
it will get different representations even
in the embedding space.
And from there, of course, as we travel through these layers,
we expect even more things to change
about the representations.
A second important preliminary is
to give some attention to how BERT and models like it
tokenize sequences.
And here I'm giving you a bit of code in the hopes
that you can get hands-on and get a feel for how
these tokenizers behave.
I'm taking advantage of the Hugging Face library.
I have loaded a BERT tokenizer and I load that
from a pre-trained model.
In cell 3, you can see that I've called the tokenize function
on the sentence.
This isn't too surprising and the result is a pretty normal
looking sequence of tokens.
You see some punctuation has been separated off,
but you also see a lot of words.
When you get down to cell 4, though,
for the sequence "encode me" this is a bit surprising.
The word encode in the input that's
been broken apart into two subword tokens, "en"
and then "code" with these boundary markers on it.
BERT has broken that apart into two subword sequences.
And if I feed in a sequence that has a really unfamiliar set
of tokens in it, it will do a lot of breaking apart
of that sequence, as you can see in cell 5 for the input
"snuffleupagus" where a lot of these pieces have come out.
This is the essential piece for why
BERT is able to have such a small vocabulary, only about
30,000 words, compare that with the 400,000 words
that are in the GloVe space.
The reason it can get away with that
is that it does a lot of breaking apart
of words into subword tokens.
And of course, because the model is contextual,
we have an expectation that, for example, when it encounters
code here in the context of "en" at some conceptual level,
the model will recognize that it has processed the word "encode"
even though there was two tokens underlining it.
Let's flesh this out a bit by looking at the full interface
for dealing with these models and, again,
taking advantage of Hugging Face.
I'm going to load a BERT model and a BERT tokenizer.
It's important that they use the same pre-trained weights, which
Hugging Face will download for you from the web.
And so those are tied in and I set up
the tokenizer and the model.
If I call tokenizer.encode on a sequence,
it will give me back a list of indices.
And those indices will be used as a lookup
to start the process of computing this entire sequence.
In cell 6, I actually use the model
to derive that grid of representations.
Hugging Face is giving us an object
that has a lot of attributes.
If I call output_hidden_states equals true when I use
the model here, but I can call .hidden_states and get that
full grid of representations that I showed you before.
This is a sequence with 13 layers.
That's 1 embedding layer plus 12 of the additional layers.
And if I key into one of the first layer,
that will be the embedding.
You can see that its shape is 1 by 5 by 768.
This is the batch of one example.
It has five tokens.
The three that we can see here plus the class and SEP tokens.
And each one of those tokens in the embedding layer
is represented by a vector of dimension 768.
And that remains consistent through all the layers
in the model.
So I went to the final output states,
I again just index into .hidden_states here.
The shape is the same and that will be
consistent for all the layers.
Those are the preliminaries.
And let's think about how we could derive
some static representations.
The first approach that Bommasani et al considered
is what they call the decontextualized approach
and this is like the simplest thing possible.
We are just going to process individual words as though they
were sequences and see if we can make any sense of them.
So we would start by feeding in a word like "kitten"
and we would allow the model to break it apart
into its subword pieces.
And then we simply process that with the model,
we get a full grid of representations.
Now because we potentially have subword tokens here,
we need some pooling function.
So what we can do is just pool using something
like mean to get a fixed static representation of dimension 768
for this individual word.
And of course, we don't have to use the final layer,
we can use lower down layers.
And we don't have to use mean as the pooling function.
You could consider something like max or min
or even last, which would just disregard
all of the representations except for the one
corresponding to the final subword token.
This is really simple.
It's potentially unnatural, though.
BERT is a contextual model.
It was trained on full sequences.
And especially if we leave off the class and SEP tokens,
we might be feeding in sequences that BERT has really
never seen before.
And so it might be unknown how it's
going to behave with these unusual inputs.
Nonetheless, though, we could repeat this process
for all the words in our vocabulary
and derive a static embedding space
and maybe it has some promise.
However, to address this potential unnaturalness
and potentially take more advantage of the virtues
that BERT and related models have,
Bommasani et al consider also the aggregated approach.
So in this approach, you process lots of corpus examples
that contain your target word.
You've got that sort of glimpse of a corpus.
Our target word is kitten, of course,
we allow it to be broken apart into subword tokens.
The full sequences in these examples
would also be broken apart into subword tokens.
But the important thing is that our target word
might have subword tokens.
We pool those as we did before for the decontextualized
approach and we're also going to pool
across all of the different context examples
that we processed.
And the result of that should be a bunch of natural inputs
to the model.
But in the end, we derive a static representation
that is some kind of average across all of the examples
that we processed.
This seems very natural.
It's taking advantage of what BERT is best at.
I will warn you, though, that this is very computationally
demanding.
We're going to want to process lots of examples,
and BERT requires lots of resources
because it develops really large representations as we've seen,
but it might be worth it.
Now, Bommasani et al offer lots of results
that help us understand these approaches
and how they perform.
Let me give you a glimpse of them as a kind of summary.
So what we've got here is results for the SimVerb 3,500
dataset, a word similarity data set
that's very similar to the ones that you'll
be working with on the homework and bake-off.
Our metric is Spearman correlation and higher
is better.
That's along the y-axis.
And along the x-axis, I have the layer in the model
that we're keying into.
And then, of course, what we should watch
is that we have two pooling functions f and g.
f is subword pooling and g is context pooling
for models that have it, and it's
decont for the decontextualized approach.
Now we have a very clear result across these results,
and I think across all the results in the paper.
Lower layers are better.
Lower layers are giving us good high fidelity representations
of individual words.
As we travel higher in the model we
seem to lose a lot of that word level discrimination.
In addition, your best choice is to do
mean pooling for the context and subword pooling
seems to matter less, right?
All of these lines here are all for the context
pooling model with mean as your context pooling function.
The very best choice, though, I think consistently
is mean for both of these pooling functions here.
You can see that in this result, and I
think that's consistent across all the results in the paper.
But the overall takeaway here is that, as expected,
the aggregated approach is better
than the decontextualized approach.
However, if you don't have the computational budget for that,
then mean pooling and the decontextualized approach
looks really competitive.
That's not so evident in this spot,
but if you look across all the results in the paper,
I think that's a pretty clear finding.
So that would be a good choice.
And one thing is clear, that simple approach
is better than some kinds of context
pooling where you choose the wrong context pooling function
like min or max.
Despite all of the effort that went into this set of results
and also these, they're all kind of
down here entangled with the decontextualized approach.
But mean as the pooling function there is really
an outstanding choice, as you can see from these results.