CHRIS POTTS: Welcome, everyone, to our first screencast
on contextual word representations.
My goal here is to give you an overview for this unit
and also give you a sense for the conceptual landscape.
Let's start with the associated materials.
You might think that the name of the game for this unit
is to get you to the point where you can work productively
with this notebook called finetuning,
which shows you how to fine-tune contextual word representations
for classification problems.
I think that could be a very powerful mode for you
as you work on the current assignment and bakeoff.
For background and intuitions, I highly recommend this paper
by Noah Smith.
The beating heart of this unit is really
the Transformer architecture, which
was introduced by Vaswani, et al., 2017 in a paper
called "Attention is All You Need."
It's a highly readable paper.
But I recommend that if you want to read it,
you instead read Sasha Rush's outstanding contribution,
"The Annotated Transformer."
What this does is literally reproduce the text of Vaswani,
et al., 2017 with PyTorch code woven in,
culminating in a complete implementation
of the Transformer as applied to problems
in machine translation.
This is a wonderful contribution in the sense
that to the extent that there are points of unclarity
or uncertainty in the original text,
they are fully resolved by Sasha's code.
And of course, this can give you a really good example
of how to do efficient and effective implementation
of model architectures like this using PyTorch.
In practical terms, we're going to make
extensive use of the Hugging Face Transformers
library, which has really opened up access
to a wide range of pretrained transformer models.
It's very exciting.
And it's enabled lots of new things.
For us, the central architecture will be BERT.
We'll have a separate screencast on that.
And we're also going to have a screencast on RoBERTa, which
is Robustly Optimized BERT.
I think it's an interesting perspective in the sense
that they explored more deeply some of the open questions
from the original BERT paper.
And they also released very powerful pretrained parameters
that you could, again, use in the context
of your own fine-tuning.
And then for a slightly different perspective
on these transformers, we're going
to look at the ELECTRA architecture, which
came from Kevin Clark and colleagues
at Stanford and Google.
I really like this as a new perspective.
There are, of course, many different modes
of using the transformer at this point.
I'm going to mention a few at the end of the screencast.
And just for the sake of time, I've
decided to focus on ELECTRA.
And you can explore the others in your own research.
Let's begin with some intuitions.
And I'd like to begin with a linguistic intuition, which
has to do with word representations
and how they should be shaped by the context.
Let's focus on the English verb break.
We have a simple example, the vase broke,
which means it shattered to pieces.
Here's a superficially similar sentence, dawn broke.
Now the sense of break is something more like begin.
The news broke.
Again, a simple intransitive sentence.
But now the verb break means something
more like publish, or up here, or become known.
Sandy broke the world record.
This is a transitive use of the verb break.
And it means, surpass the previous level.
Sandy broke the law is another transitive use.
But now it means, Sandy transgressed.
The burglar broke into the house is a physical act
of transgression on its face.
The newscaster broke into the movie broadcast
is a sense that it's more like interrupt.
And we have idioms like break even, which
means we neither gained nor lost money.
And this is just a few of the many ways
that the verb break can be used in English.
How many senses are at work here?
It's very hard to say.
It could be one.
It could be two.
It could be ten.
It's very hard to delimit word senses.
But it is very clear from this data
that our sense for the verb break
is being shaped by the immediate linguistic context.
Here are a few additional examples,
things like flat tire, flat beer, flat note, flat surface.
It's clear that there is a conceptual core running
through all of these uses.
But it's also true that a flat tire
is a very different sense for flat
than we get from flat note or flat surface.
I have something similar for throw a party, throw a fight,
throw a ball, throw a fit.
We have a mixture of what you might call
literal and metaphorical here.
But again, a kind of common core that we're drawing on.
But the bottom line is that the sense for throw, in this case,
is very different depending on what kind of linguistic context
it's in.
And we can extend this to things that seem to turn more
on world knowledge.
So if you have something like, a crane caught a fish,
we have a sense that the crane here is a bird.
Whereas, if we have a crane picked up the steel beam,
we have a sense that it's a piece of equipment.
This seems like something that's guided
by our understanding of birds, and equipment, and fish,
and beams.
And when we have relatively unbiased sentences like,
I saw a crane, we're kind of left
guessing about which object is involved,
the bird or the machine.
And we can extend this past world knowledge
into things that are more like discourse understanding.
So if you have a sentence like, "are there typos?
I didn't see any."
The sense of "any" here, we have a feeling
that something is elided but probably localized on "any."
And "any" here means any typos as a result of the preceding
linguistic context.
"Are there any bookstores downtown?
I didn't see any."
Same second sentence.
But now the sense of "any" is probably
going to be something more like bookstores,
as a result of the discourse context that it appears in.
So all of this is just showing how much
individual linguistic units can be shaped by context.
Linguists know this deeply.
This is a primary thing that linguists try to get a grip on.
And I think it's a wonderful point of connection
between what linguists do and the way we're
representing examples in NLP using contextual models.
This is a very exciting development
for me as a linguist, as well as an NLPer.
Here's another set of intuitions that's related more to things
like model architecture and what you might call inductive biases
for different model designs.
Let's start off here in the left.
This is a high-bias model in the sense
that it makes a lot of a priori decisions about how
we will represent our examples.
The idea is that we have three tokens, which we look up
in a fixed embedding space.
And then we have decided to summarize those embeddings
by simply summing them together to get a representation
for the entire example.
Very few of these components are learned as part of our problem.
We've made most of the decisions ahead of time.
As we've seen, as we move to a recurrent neural network,
we relax some of those assumptions.
We're still going to look upwards in a fixed embedding
space.
But now instead of deciding that we know the proper way
to combine them is with summation,
we're going to learn from our data
a very complicated function for combining them.
And that will presumably allow us
to be more responsive, that is, less biased about what the data
are likely to look like.
The tree structured architecture down here
is an interesting mixture of these ideas.
It's like the recurrent neural network.
Instead of assuming that I can process the data left-to-right,
the data get processed into constituents,
like "the Rock" is a constituent, as excluded
from "rules" over here.
Now this is probably going to be very powerful
if we're correct that the language data are structured
according to these constituents.
Because it will give us a boost in terms of learning.
It could be counterproductive, though, to the extent
that constituent structure is wrong.
And I think that's showing that biases
that we impose at the level of our architectures
can be helpful, as well as a hindrance, depending
on how they align with the data-driven problem
that we're trying to solve.
In the bottom right here, I have the least biased model
in all these sentences, of all the ones depicted here.
I've got a recurrent neural network like this one,
except now I'm assuming that information
can flow bidirectionally, so no longer
a presumption of left-to-right.
And in addition, I've added these attention mechanisms,
which we'll talk a lot about in this unit.
But essentially, think of them as ways
of creating special connections between all of the hidden
units.
And the idea here is that we would let the data tell us
how to weight all of these various connections
and, in turn, represent our examples.
We're making very few decisions ahead of time
about what connections could be made,
and instead of just listening to the data and the learning
process.
And we are, at this point, on the road
toward the Transformer, which is kind
of an extreme case of connecting everything with everything else
and then allowing the data to tell us how to weight all
of those various connections.
And that does bring us to this notion of attention, which
we've not discussed before.
But I think I can introduce the concepts and a bit of the math.
And then we'll see them again throughout this unit.
So let's start with the simple sentiment example
and imagine we're dealing with a recurrent neural network
classifier.
Our example is really not so good.
And we're going to fit the classifier, traditionally,
on top of this final hidden state over here.
But we might worry about doing that.
That by the time we've gotten to this final state,
the contribution of these earlier words,
which are clearly important linguistically,
might be sort of forgotten or overly diffuse.
So attention mechanisms would be a way for us
to bring that information back in and infuse
this representation, hc, with some
of those previous important connections.
So here's how we do that.
We're going to first have some attention scores, which
are simply dot products of our target vector with all
the preceding hidden states.
So that gives us a vector of scores which are traditionally
softmax normalized.
And then what we do is create a context
vector by weighting each of the previous hidden states
by its attention weight, and then taking
the average of those to give us the context vector K.
And then we're going to have this special layer here, which
concatenates K with our previous final hidden state,
and feeds that through this layer of learned parameters
and a nonlinearity to give us this new hidden representation,
h tilde.
And it is h tilde that is finally
the input to our softmax classifier.
Whereas before, we would have simply directly input hc
up here, we now input this more refined version
that is drawing on all of these attention connections
that we created with these mechanisms.
And again, as you'll see, the transformer
does this all over the place with all of its representations
at various points in its computations.
Here's another guiding idea that really
shapes how these models work.
I've called this word pieces.
And we've seen this before.
These models typically do not tokenize data in the way
that we might expect.
I've loaded in a BERT tokenizer.
And you can see that for a sentence,
like "this isn't too surprising" the result is some pretty
familiar tokens, by and large.
But when I feed in something like "encode me"
the intuitive word "encode" is split apart
into two-word pieces.
And clearly, we're implicitly assuming
that the model, because it's contextual,
can figure out that these pieces are in some conceptual sense
one word.
And you might extend that up to idioms like "out of this world"
where we treat them as a bunch of distinct tokens.
But we might hope the model can learn
that there's an idiomatic unity to that phrase.
And this also has the side advantage
that for unknown tokens like "Snuffleupagus"
it can break them apart into familiar pieces.
And we at least have a hope of getting
a sensible representation for that out-of-vocabulary item.
The result of all this is that these models can get away
with having very small vocabularies,
precisely because we are relying on them implicitly
to be truly contextual.
Here's another inspiring idea that we've not
encountered before.
This is called positional encoding.
And it's another way in which we can capture sensitivity
of words to their contexts.
So as you'll see, when you go all the way down
inside the transformer architecture,
you do have a traditional static embedding
of the sort we discussed in the first unit for this course.
Those are in light gray here, fixed representations
for the words.
However, in the context of a model like BERT,
what we traditionally think of as its embedding representation
is actually a combination of that fixed embedding
and a separate embedding space called
the positional embedding, where we have learned representations
for each position in a sequence.
This has the intriguing property that one and the same word,
like the, will have a different embedding
in the sense of the green representation here,
depending on where it appears in your sequence.
So right from the get-go, we have a notion
of context sensitivity even before we've
started to connect things in all sorts of interesting ways.
Now let's move to some current issues and efforts,
some high-level things that you might think about
as you work through this unit.
This is a really nice graph from the ELECTRA
paper from Clark, et al.
Along the x-axis here, we have floating point operations,
which you could think of as a kind of basic measure
of compute resources needed to create these representations.
And along the y-axis, we have GLUE score.
So that's like a standard NLU benchmark.
And the point of this plot here is
that we're reaching kind of diminishing returns.
So we had rapid increases from GloVE, GPT,
and up through to BERT, where we're really
doing much better on these GLUE scores.
We're increasing the floating point operations.
But it seems to be commensurate with how
we're doing on the benchmark.
But now with these larger models like XLNet and RoBERTa,
it's arguably the case that we're
reaching diminishing returns.
RoBERTa involves more than 3,000 times the floating point
operations of GloVe.
But it's not that much better along this y-axis
than some of its simpler variants like BERT-Base.
And so this is something we should think about
when we think about the costs in terms of money,
and in the environments, and energy,
and so forth when we think about developing these large models.
And here's a really extreme case.
Who knows how long we can train these things
or how much benefit we'll get when we do so.
But at a certain point, we're likely to incur
costs that are larger than any gains
that we can justify on the problems we're trying to solve.
And that goes and leads us to this lovely paper,
which talks about the environmental footprint
of training these really big models.
And it just shows that by training a big transformer
from scratch, it really incurs a large environmental cost.
That's certainly something we should
have in mind as we think about using these models.
For me, it's a complicated question though.
Because it's offset by the fact that, by and large, all of us
aren't training these from scratch, but rather, benefiting
from publicly available pretrained representations.
So while the pretraining for that one version
had a large environmental cost, it
feels like it's kind of offset by the fact that a lot of us
are benefiting from it.
And it might be that in aggregate, this
is less environmentally costly than the old days
when all of us always trained all of our models
literally from scratch.
I just don't know how to do the calculations here.
But I do know that increased access has been empowering
and is likely offsetting some of the costs.
And a lot of that is due to the contributions of the Hugging
Face library.
There are a lot of efforts along these same lines
to make BERT smaller by compressing it
literally fewer dimensions than other kinds of simplifications
of the training process, and BERT distillation,
and so forth.
Here are two outstanding contributions,
kind of compendiums of lots of different ideas in this space.
And I also highly recommend this lovely paper
called "A Primer in BERTology," which
explores a lot of different aspects of what
we know about BERT and how it works, various variations
people have tried, various things
that people have done to probe these models,
and understand their learning dynamics,
and so forth and so on.
It's a very rich contribution, certainly can
be a resource for you as you think about these models.
And just because we don't have time to cover them all,
there are a bunch of interesting Transformer variants
that we will not be able to discuss in detail.
I thought I'd mention them here.
SBERT is an attempt to develop sentence-level representations
from BERT that are particularly good at finding which sentences
are similar to which other sentences according
to cosine similarity.
And I think that could be a powerful mode
of thinking about these representations
and also of practical utility if you
need to find which sentences are similar to which others.
You have probably heard of GPT, the Generative Pre-trained
Transformer, in various of its forms.
You can get GPT-2 from Hugging Space--
and Hugging Face.
And you have unfettered access to it.
And of course, there's more restrictive access
at this point to GPT-3.
These are conditional language models,
so quite different from BERT.
And they might be better than BERT
for things like truly conditional language
generation.
XLNet is an attempt to bring in much more context
into these models.
It stands for Xtra Long Transformer.
So if you need to process long sequences,
this might be a good choice.
And this is also an attempt to bring
in some of the benefits of conditional language models
into a mode that is more bidirectional, the way BERT is.
T5 is another conditional language mode, as is BART.
These models might be better choices for you
if you need to actually generate language.
What I think the standard wisdom is
that models like BERT and RoBERTa
are better if you simply need good representations
for fine-tuning on a classification problem,
for example.
More models will appear every day.
And I think it's worth trying to stay up
to speed on the various developments in this space.
Because this is probably just the tip of the iceberg here.