welcome everyone this is part three in
our series on contextual word
representations we're going to be
talking about the bert model which is an
innovative and powerful application of
the transformer architecture which we
covered in the previous screencast
let's dive into the core model structure
of burt we'll begin with the inputs as
usual we'll work with a simple example
the rock rules that has three tokens but
you'll notice that for bert we begin
every sequence with the designated class
token and sequences end with a
designated set token and the sep token
can also be used as a boundary marker
between subparts of an input sequence
we'll learn positional embeddings for
each one of those tokens and in addition
as you can see in maroon here we're
going to learn a second notion of
position
it's sent a for all of these tokens so
it won't contribute in this particular
case but if we had a problem like
natural language inference where
examples consist of a premise sentence
and a hypothesis sentence we could use
learn separate embeddings for the
premise and the hypothesis and thereby
hope that we can capture that kind of
second notion of position within our
input
we're going to have learned embeddings
for each one of these notions for the
word and the two notions of position
and as in the transformer the actual
embeddings given here in my green will
be additive combinations of those three
learned embedding representations
from there we just do a lot of work with
the transformer have we have repeated
transformer blocks it could be 12 it
could be 24 it could be even more
and the output of all of those
transformer blocks in the end is a
sequence of output representations these
are vectors not given them in dark green
that is the core model structure for
burnt
and that brings us to how this model is
trained the mask language modeling
objective is the fundamental training
objective for this model
fundamentally what the model is trying
to do is work as an auto encoder and
reproduce entire input sequences
to make that problem non-trivial though
we're going to employ this mass language
modeling idea and what that means is
that we're going to go through these
input sequences and randomly replace
some small percentage 10 to 15
of the input tokens with this designated
mask token
and then the job of the model is to
learn for those masked inputs to re
reconstruct what was the actual input
right so in this case we masked out
rules and the job of the model is to use
this bi-directional context that's
flowing in from all those attention
mechanisms to figure out that rules was
actually the token that belonged to that
initial position
ver team uh did a variant of this as
well which is uh masking by a random
word so in this case we might replace
rules with a word like every picked
randomly from our vocabulary but there
again the fundamental job of the model
is to learn to figure out that rules was
the actual token in that position
so it's going to make some prediction in
this case if it's different from rules
and the error signal will flow back down
through all the parameters of this model
affecting we hope all the
representations because of that dense
ticket of attention connections that
exists across these time steps and in
that way the model will learn to update
itself effectively learning how to
reconstruct the missing pieces from
these inputs that we created during
training
so let's dive into that mask language
modeling objective a little more deeply
for transformer parameters h theta
and some sequence of tokens x with its
corresponding masked version x hat this
is the objective here well let's zoom in
on some time step t
uh the fundamental scoring thing is that
we're going to look up the vector
representation in the embedding for that
time step t
and we'll take the dot product of that
with the output representation at time t
from the entire
transformer model
that much there that scoring procedure
looks a lot like what you get from
conditional language models you just
have to remember that because of all
those attention mechanisms connecting
every token to every other token this is
not just the preceding context before
time step t but rather the entire
surrounding context for this position
and then as usual we normalize that by
considering all the alternative tokens x
prime that could be in this position
now you'll notice over here there's an
indicator variable mt mt is 1 if token t
was massed l0 so that's like saying
we're going to turn on this loss
only for the tokens that we have masked
out
and then the final thing is kind of not
a definitional choice about this model
but something worth noting you'll see
that we're using the embedding for this
token
effectively as the softmax parameters
there could be separate parameters here
that we learned for the classifier part
that's learning to be a language model i
think but i think people have found over
time that by tying these parameters by
using the transpose of these parameters
to create the output space we get some
statistical strength and more efficient
learning
there is a second objective in the verb
model and it is the binary next sentence
prediction task
uh and i think this was an attempt to
find some coherence uh beyond just the
simple sentence or sequence level
uh so this is pretty straightforward for
positive instances for this class we're
going to take actual sequences of
sentences in the corpus that we're using
for training so here you can see these
actually occurred together and they are
labeled as next and for negative
examples we just randomly choose a
second sentence and label that as not
next and i think the aspiration here was
that this would help the model learn
some notion of discourse coherence
behind beyond the local coherence of
individual sequences
now what you probably want to do with
the burp model is not train it from
scratch but rather fine tune it on a
particular task that you have there are
many modes that you can think about for
doing this a kind of default choice a
standard and simple choice would be to
use the class token
more specifically it's output in the
final layer of the burp model as the
basis for setting some task specific
parameters and then using that um
whatever labels you have for supervision
up here
and that could be effective because the
class token appears in this position in
every single one of these sequences and
so you might think of it as a good
summary representation of the entire
sequence and then when you do the fine
tuning you'll of course be updating
these task parameters and then you could
if you wanted to also update some or all
of the actual parameters from the
pre-trained model that would be a true
notion of fine-tuning
now you might worry that the class token
is an insufficient summary of the entire
sequence and so you could of course
think about pooling all the output
states in the sequence via some function
like sum or mean or max and using those
as the input to whatever test specific
parameters you have up here at the top
i just want to remind us that
tokenization invert is a little unusual
we've covered this a few times before
but just remember that we're getting
effectively not full words but word
pieces so for cases like encode me you
can see that the word encode has been
split apart into two word pieces and
we're hoping implicitly that the model
can learn that that is in some deep
sense still a word even though it has
been split apart and that should draw on
the truly contextual nature of these
models
the bur team did two initial model
releases bird base consists of 12
transformer layers and has
representations of dimension 768 with 12
attention heads for a total of 110
million parameters that's of course a
very large model but this is manageable
for you to do local work especially if
you just want to do some simple fine
tuning or use this model for
inference the bird large release is much
larger it has 24 layers
twice the dimensionality for its
representations and 16 attention heads
for a total of 340 million parameters
this is large enough that it might be
difficult to do local work with but of
course you might get much more
representational power from using it
for both of these models we have a
limitation to 512 tokens and that is
because that is the size of the
positional embedding space that they
learned there are many new releases of
course you can find those at the project
site and hugging face has made it very
easy to access these models and that's
been very empowering
to close this let me just mention a few
known limitations of burke that we're
going to return to as we go through some
subsequent models for this unit so first
in the original paper there is a large
but still partial number of ablation
studies and optimization studies there's
a huge landscape that's in play here and
only small parts of it are explored in
the original paper so we might worry
that there are better choices we could
be making within this space
the original paper also points out that
there's some unnaturalness about this
mass token they say the first downside
of the mlm objective is that we are
creating a mismatch between pre-training
and fine-tuning because the mass token
is never seen during fine tuning so
that's something we might want to
address
they also point out that there's a
downside to using the mlm objective
which is just that it's kind of data
inefficient we can only mask out a small
percentage of the tokens because we need
the surrounding context in order to
sensibly reproduce those tokens
and that means that it's kind of data
inefficient
and finally this is from the excel net
paper i think this is quite perceptive
burt assumes that the predicted tokens
are independent of each other given the
unmasked tokens which is oversimplified
as high order long-range dependency is
prevalent in natural language what they
have in mind here is essentially if you
have a an idiom like out of this world
and it happens that both the first and
the last words in that idiom are masked
out then byrd is going to try to
reproduce them as though they were
independent of each other when in fact
we know that there is a statistical
dependency between them coming from the
fact that they are participating in this
idiom
so there's some notion of
representational
coherence that bert is simply not
capturing with its mlm objective