CHRISTOPHER POTTS: Welcome, everyone.
This is part 4 in our series on analysis methods in NLP.
We're going to be talking about probing.
This is the first of the two structural evaluation methods
that we're going to consider.
It's time to get really introspective about what
our models are doing.
Here's an overview of the probing idea.
The core thing is that we're going to use supervised models,
those are the probe models, to determine
what's latently encoded in the hidden representations
of our target models.
This is often applied in the context of BERTology, which
should be like, I have BERT as a pre-trained artifact
and I would like to understand the nature
of its hidden representations.
What do they latently encode?
And for that you might use probe models.
Probing, as you will see, can be a source
of really valuable and interesting insights,
but we do need to proceed with caution on two major issues
here.
First, a very powerful probe model
since it is a supervised model might
lead you to see things that aren't really
in your target model, but rather just things
that your probe model has learned.
And you might, therefore, overdiagnose latent information
in your target model when, in fact, it's
all being stored in the probe.
And I'm going to offer you a technique for navigating
around that issue.
And the second one is that probes cannot tell us about
whether the information that we identify has any causal
relationship with the target model's behavior.
It will be very tempting for you to say, Oh,
I have discovered that this representation layer includes
part of speech information.
And you might therefore conclude that part of speech information
is important for whatever task you have set,
but we can't actually make that inference.
It could be that the part of speech information
is simply latently encoded, but not actually
especially relevant to your model's input/output behavior.
Final section of the slide show, I'm
going to just talk briefly about unsupervised probes, which
seek to address this first problem here that the probe
model might actually be the thing that's
encoding all of this information that we
claimed to have discovered.
And then when we talk about future attribution methods,
we'll get closer to being able to address
some of these causal questions.
Let's begin with the core method for probing.
And just because this is a typical framing of these ideas,
I've got depicted here what you might think of as a,
kind of, generic transformer-based model, where
we have three layers with all of these blocks.
These maybe are the output representations
from each of the transformer blocks.
And you can see that I've got an input sequence coming in here.
And the idea would be that we could
pick some hidden representation in this model,
like this middle one, h here.
And decide that we're going to fit a small linear model
presumably on that hidden representation,
and see whether we can figure out
whether that representation encodes
some information about some task that we care about.
So for example, if you wanted to figure out
whether sentiment or lexical entailment
was encoded at that point, you'd need a labeled data set
for sentiment or entailment.
And then you would fit the probe model
on this representation and use that
to determine the extent to which that information is encoded
there.
This depiction is a little bit poetical.
So it's worth just walking through mechanically
what you'd actually be doing.
You would use this BERT model and process different examples,
like for the sequence here.
And get an output representation,
which should be paired with some task label.
And you would repeatedly do that for different inputs.
You're essentially using this BERT model
as an engine for creating representations
that will become your feature representation matrix
x paired with your labels y.
And it is this model that will be
the basis for your linear probe model, this small linear model
as I've identified in here.
So you're kind of using BERT as an engine
to create a data set that is then
the input to a supervised learning problem.
Another perspective would be that you're
kind of using frozen BERT parameters in this case,
and fitting a model on top of them.
It's just that instead of picking an output point,
you've picked possibly one of the internal representations.
And this is very general.
And in fact, most often, when you
read without quotes in the literature,
they're actually sequence problems
like part of speech tagging or named entity recognition.
And therefore, you might use an entire layer or even
a set of layers as the basis for your probe model.
Now you can hear in my description there,
that there is a interesting judgment call that you're
making about whether you are probing or simply learning
a new model, right?
Probes, in the sense that I just presented them,
are supervised models whose inputs
are frozen parameters of the models that we're probing,
our target models, right?
This is hard to distinguish from simply
fitting a supervised model as usual
with some particular choice of featurization.
As a result of this, it is essentially
a foregone conclusion that at least some
of the information that we identify with our probe
is actually stored in the probe model parameters
and it's just that we provided useful input features that
allow this probe to be successful.
And that's the sense in which the inputs are latently
encoding this information.
But with the probe, we have not determined
that it is truly latently there, but rather
that it's a stepping-- a stepping stone toward a model
that could be successful at this as
conceived of as a supervised learning task.
So those are important distinctions to keep in mind.
As a result of this, more powerful probes,
like deep neural networks, might find more information
than simple linear models.
But that's not because they're able to tease out
more information from the representations themselves,
but rather because the probe model now
has so much more capacity for storing information
about the task that you're probing for.
So there are a bunch of different judgment calls here.
And that's difficult. But a very productive entry
into this space is this really lovely paper
from Hewitt and Liang 2019, where
they introduce the notion of a control task
and the corresponding metric of probe selectivity.
So here's the idea.
A control task will be some random task
with the same input/output structure as the target task
that we want to use for our probing.
Like for example, for word-sense classification,
you might have words assigned random fixed words senses
independent of their context.
Or for part of speech tagging, instead
of using the actual part of speech tags,
you might randomly assigned words
to fixed tags from the same tag space.
Or for parsing, it gets a little bit more nuanced,
but you might have some edge assignment strategies
that you use semirandomly to link different pairs of words
into a kind of pseudo-parse.
And that would serve as a control task
for trying to surface latent actual parsing information.
So those are control tasks.
And then selectivity is simply the difference
between your probe performance on the task
and your identical probe model structure on these control
tasks.
And Hewitt and Liang used this to tease out what, I think,
is a pretty clear intuition, which
is that as you get more powerful probes,
they simply become less selective.
So along the x-axis here, we have MLP hidden units.
So we have model complexity from left to right,
where we have very complicated, powerful models
at the right-hand side.
And here we have accuracy.
And we're measuring our control task in red
and our actual probe task in this light blue here.
And selectivity is the difference between those two.
So you can see, for example, that the very weak models,
the ones with two hidden units have very high selectivity.
Whereas by the time I have this very powerful NLP
with lots of hidden units, selectivity
has gone almost to 0.
And it's very hard to say that you've uncovered any latent
information.
Because even the control task is fully solvable with a model
that has this much capacity.
So I think what this is pushing us toward
is always having control tasks as part of the picture
and always reporting selectivity so
that we can control for the complexity of the probe model
itself.
That's an important and easy, practical step
that will give you a clearer picture of what you've actually
surfaced with your probe.
That's the first issue.
The second issue is just something that you should keep
in mind as a theoretical fact about probing,
which is that it is fundamentally limited
in the sense that it cannot tell you that the information you
discover has any causal impact on the model's input/output
behavior.
To illustrate that, I'm just going
to show you a simple example that kind of proves this.
So imagine over here on the left,
I have a simple model that's going to take in three integers
and sum them.
So the output here will be the sum of three integers.
Like if I put in 1, 2, 3, it will output 6.
And it does that by representing each one of those integers
as the single-dimensional vector that just is that integer.
And then we have a whole bunch of transformer
like model parameters, dense connections here
that will lead us finally to the output layer.
So you can easily imagine that you probe this position L1 here
and you find that it computes x plus y, which might be starting
to reveal for you that there's some kind of tree
structure to this model even though it was densely
connected.
It has learned a structured solution to the problem.
And you might probe L2 and find that it computes z.
And that would really lead you to think
that you've got a, kind of, interesting tree
structure with constituents for this addition problem.
And that's certainly suggestive.
However, here is an example of a model that
shows that neither L1 nor L2 has anything to do with the model's
output predictions.
It is entirely that middle state that
tells the complete story about the output.
I'll leave you to work through the details if you choose to,
but a shortcut way to see that is
that the final parameters that take us from these output
representations to the predictions
have zeroed out the first and third positions,
leaving only the second one as having
any kind of causal efficacy.
Even though in this model, if you probe,
you do indeed find that it looks like those representations
perfectly encode these two pieces of information.
That's a dramatic and clear, simple illustration
of how a probe could get divorced
from the actual causal behavior of the model.
Again, something that's worth keeping in mind.
And finally, to close this out, of course,
for that first problem about distinguishing between probe
capacity and actually latently encoded
information, one response to that that's
developing in the literature now is
to develop unsupervised probes.
These would be models like these that
seek to find in actual facts about the model
with no additional supervision, the latent information
that we hope to find.
And this would come from simply doing
linear transformations of the parameters
and measuring distance between parameters
as a way of getting a sense for what's actually there
without the complications that come
from having this additional supervised probe model.
And finally, for much more information about probes,
and what we think we've learned from them,
and what they can tell us, I encourage
you to check out this paper by Rogers
et al, a primer on BERTology.
It has a large and interesting subsection
entirely devoted to what probes have told us.
Certainly worth a look, and a great overview of the space.