CHRISTOPHER POTTS: Welcome back, everyone.
This is part four in our series on Contextual Word
Representations.
We are going to be talking about a robustly optimized BERT
approach a.k.a.
RoBERTa.
So recall that I finished the BERT screencast by listing out
some known limitations of the BERT model, most of which
were identified by the original BERT authors themselves.
And top of the list was simply that,
although the original BERT paper does
a good job of exploring ablations of their system
and different optimization choices.
There's a very large landscape of ideas here, and most of it
was left unexplored in the original paper.
Essentially, what the RoBERTa team did
is explore more widely in this space.
That is the robustly optimized part of RoBERTa.
So what I've done for this slide here
is list out what I take to be the central differences
between BERT and RoBERTa.
And I'll follow this up with some evidence from the RoBERTa
paper in a second.
But first let's go through the central differences beginning
with this question of static versus dynamic masking.
So for the original BERT paper what they did
is create four copies of their dataset,
each with different masking.
And then those four copies were used repeatedly
through epochs of training.
The RoBERTa team had the intuition
that it would be useful to inject some diversity
into this training process.
So they went to the other extreme, dynamic masking.
Every single example when it's presented to the model
is masked a potentially different way,
via some random function.
There are also differences in how examples
are presented to the models.
So BERT presented two concatenated document segments.
This was crucial to its next sentence prediction task.
Whereas for RoBERTa we're just going
to have sentence sequences, that is
pairs, that may even span document boundaries.
Relatedly whereas BERT had, as one of its central pieces,
this next sentence prediction task.
RoBERTa simply drops that as part of the objective here.
That simplifies the presentation of examples
and also simplifies the modeling objective.
Now, RoBERTa is simply using a masked language modeling
objective.
There are also changes to the size of the training batches.
So for BERT that batch size was 256 examples.
RoBERTa cranked out all the way up to 2000.
There are differences when it comes to tokenization.
So as we've seen BERT used this very interesting word piece
tokenization approach which mixes some sub-word pieces
with some whole words.
RoBERTa simplified that down to just character-level byte-pair
encoding which I think leads to many more word pieces
intuitively.
There are also differences in how the model was trained.
So BERT trained on a substantial corpus, the BooksCorpus
plus English Wikipedia, is a lot of data indeed.
RoBERTa again cranked that up even further,
they trained on the BooksCorpus, the CC-News corpus,
the OpenWebText corpus and the Stories
corpus, a substantial increase in the amount of training data.
There are also differences in the number of training steps,
and there's a subtlety here.
So for the BERT model, it was originally
trained on 1 million steps.
The RoBERTa model was trained on 500,000 steps.
Which sounds like fewer steps, but overall this
is substantially more training, in virtue of the fact
that the training batch sizes are so much larger for RoBERTa
than they are for BERT.
And finally, the original BERT authors
had an intuition that would be useful in getting
the optimization process going, to train just
on short sequences first.
The RoBERTa team dropped that idea,
and they train on full length sequences
throughout the life cycle of optimization.
There are some additional differences
related to the optimizer and the data presentation.
I'm going to set those aside, if you
want the details I refer to section 3.1 of the RoBERTa
paper.
So let's look at a little bit of evidence
for these various choices, starting
with that question of dynamic versus static masking.
So this is the primary evidence, they're using three benchmarks.
SQuAD, MNLI and SST-2, and you can
see that more or less across the board dynamic masking
is better.
Not by a lot, but dynamic masking also
has going for this intuition that BERT is kind of data
inefficient, we can only mask out a small number of tokens.
And it seems like it ought to be useful to inject
a lot of diversity into that, so that a lot of different tokens
get masked, as we go through the training process.
But the choice is of course, supported numerically here,
I think pretty substantially.
This slide table here summarizes the choice
about how to present the examples to the model.
And this is also a little bit subtle, so numerically
the DOC-SENTENCES approach was best.
And this was an approach where they just took contiguous
sentences from within documents and treated a document boundary
as a kind of hard boundary.
That's numerically better, according
to the benchmark results.
But they actually decided to go with the FULL-SENTENCES
approach.
And the reason for that is, in not respecting document
boundaries, it is easier to create lots of batches
of exactly the same size.
Which leads to all sorts of gains
when you think about optimizing a large model like this.
So basically, they decided that those gains offset
the slightly lower performance of FULL-SENTENCES as
compared to DOC-SENTENCES, and that's
why this became their central approach.
Here's the summary of evidence for choosing 2K
as the batch size.
You can see that they chose 256, which was the BERT original.
2K and 8K and 2K looks like the sweet spot
according to MLNI, SST-2.
And this kind of pseudo perplexity
value that you get out of bidirectional models like BERT
and RoBERTa.
So that's a clear argument.
And then finally, when we come to just the amount of training
that we do.
The lesson here apparently is more is better.
On the top of this table here we have some comparisons
within the RoBERTa model, pointing to 500K as the best.
And I would just remind you that, that
is overall substantially more training.
Than was done in 1 million steps with BERT,
in virtue of the fact that our batch sizes for RoBERTa
are so much larger.
In closing, I just want to say that RoBERTa too only explored
a small part of the potential design choices
that we could make in this large landscape.
If you would like to hear even more about what we know
and what we think we know about models like BERT and RoBERTa,
I highly recommend this paper called The Primer in BERTology,
which has lots of additional wisdom,
and insights, and ideas about these models.