CHRISTOPHER POTTS: Welcome, everyone,
to part 2 in our series on natural language inference.
We're going to focus on the three data sets
that we'll be concentrating on this unit, which are SNLI--
the Stanford Natural Language Inference Corpus--
MultiNLI, and Adversarial NLI.
I think they're interestingly different,
and they're all big benchmark tasks
that can support the training of lots
of diverse kinds of systems.
So let's begin with SNLI, which is the first to appear
of these three.
The associated paper is Bowman, et al., 2015.
Sam Bowman was a student in the NLP group,
and I was his advisor, along with Chris Manning,
and a bunch of us contributed to that paper.
An important thing to know about SNLI
is that the premises are all image captions from the image
Flickr30K data set.
So that's an important genre restriction
that you should be aware of when you think about training
systems on this data.
All the hypotheses were written by crowdworkers.
I'll show you the prompt in a little bit.
But the idea is they were given this premise, which
was an image caption, and then they
wrote three different texts corresponding
to the three NLI labels.
Unfortunately, as is common with crowdsourced data sets,
you should be aware that some of the sentences
do reflect stereotypes.
I think this traces to the fact that crowdworkers,
trying to do a lot of work, are faced with a creative block.
And the way they overcome that is
by falling back on easy tricks, and some of those
involve stereotypes.
Completely understandable, and this
is something that the field is trying
to come to grips with as we think about data set creation.
It's a big data set.
It has over 550,000 training examples.
And it has dev and test sets.
Each have 10,000 examples balanced
across the three classes.
Here's a look at the mean token lengths.
It's just sort of noteworthy that premises are a little bit
longer than hypotheses.
I guess that comes down to the fact
that crowdworkers were writing these sentences.
In terms of clause types, mostly, we talk about NLI
as a sentence task.
But in fact, only 74% of the examples
are sentences that is S-rooted in their syntactic parses.
It has a large vocabulary, but may
be modest relative to the size of the data set,
and that might come back to the fact
that the genre is kind of restricted.
We had about 60,000 examples that
were additionally validated by four other annotators.
And I'll show you the response distributions, which suggests
some sources of variation.
They had high interannotator agreement.
So given that validation, about 60%
examples had a unanimous gold label.
And we rate the overall human level of agreement
at about 91.2% for the gold labels.
And that's the measure of human performance
that's commonly used for SNLI.
And the overall Fleiss kappa measured
interannotator agreement was 0.7, which
is a high rate of agreement.
And then for the leaderboard, you
can check out this link here.
Sam has been good about curating all the systems that enter,
and you can get a sense for which approaches are best.
It's clear at this point, for example,
that ensembles of deep learning methods
are the best for this problem.
I mentioned before, the crowdsourcing methods
I think it's worth thinking about precisely what happened
here.
So here's the crowdsourcing interface.
There's some instructions up here.
Here's the caption-- that is, the premise
sentence in our terms-- a little boy
in an apron helps his mother.
And then the crowdworker had to come up with three sentences.
One definitely correct-- that's an entailment case.
One may be correct--
that is our gloss on neutral.
And one definitely incorrect, which is our
gloss on contradiction.
So you can see here that there's an attempt
to use informal language connecting
with informal reasoning, common sense reasoning in the prompt
here.
And then those get translated into our three
labels for the task.
And here are some examples from the validated set.
And I think they're sort of interesting, because you
get high rates of agreement, but you
do find some examples that have a lot of uncertainty
about them, like this last one here.
And I think that might be a hallmark, actually, of NLI
problems.
Now, one really fundamental thing
that I mentioned in the overview screencast
as definitely worth being aware of
relates specifically to the contradiction relation.
And there's discussion of this in the paper.
It's a tricky point.
What we say for SNLI, using these simple examples here,
is that both of them are in the contradiction relation.
The first one is "a boat sank in the Pacific Ocean."
It has premise and hypothesis, "a boat
sank in the Atlantic Ocean."
You might ask, of course, those could be true together.
They should be neutral, not contradiction.
The reason we call them contradiction
is because we make an assumption of event coreference,
that we're talking about the same boat in the same event.
And therefore, the locations contradict each other
in a common sense way.
And the second example is an even more extreme case of this.
Ruth Bader Ginsburg was appointed to the Supreme Court
and I had a sandwich for lunch today.
We say those are in the contradiction relation.
Of course, they could be true together.
But they couldn't, in our terms, be true of the same event.
They're describing very different events.
And for that reason, they get the contradiction label.
If a premise and hypothesis probably
describe a different photo, then the label is contradiction.
That's kind of anchoring back into our underlying domain
that you might have in mind.
We can mark progress on SNLI, because Sam has
been curating that leaderboard.
As I mentioned before, we estimate human performance
up here at almost 92.
And along this x-axis here, I've got time.
And you can see that very quickly, the community
has hill-climbed toward systems that are superhuman, according
to our estimate.
But down here at 78 is the original paper.
That was from an era when deep learning systems were really
not clearly the winners in this kind of competition,
but SNLI helped change that by introducing a lot of new data.
So a very rapid rise in system performance, and then
basically monotonic increase until 2019,
when we saw the first systems that
were, in these restrictive terms,
better than humans at the SNLI task.
Let's move to MultiNLI, which was a kind of successor
to SNLI.
This was collected by Idina Williams and colleagues,
including Sam Bowman.
The train premises, in this case,
are going to be much more diverse.
They're drawn from five genres--
fiction; government reports, and letters and things;
the Slate website; the Switchboard corpus; which
is people interacting over the phone;
and Berlitz travel guides.
And then interestingly, they have additional genres just
for dev and test.
And this is what they call the mismatched condition.
And those are the "9/11 Report," face-to-face conversations,
fundraising letters, and nonfiction
from Oxford University Press, as well as
articles about linguistics.
So this is noteworthy because in the mismatched condition
that MultiLNI sets up, you are forced
to train on those training examples
and then test on entirely new genres.
And you can just see how different, for example,
Berlitz travel guides might be from the "9/11 Report."
I think this is an interesting early example
of being adversarial and enforcing
our systems to grapple with new domains and new genres.
And I think that's a really productive step in testing
these systems for robustness.
It's another large data set, slightly smaller than SNLI.
But actually, the example lengths tend to be longer.
They did the same kind of validation,
and that gives us our estimates of human performance.
And once again, I would say that we
can have a lot of confidence.
There was a high rate of agreement.
92.6% is the traditional measure of human performance here.
For MultiNLI, the test set is available only
as a Kaggle competition, and you can check out the project page
here.
I love the fact that MultiNLI was distributed with
annotations that could help someone kind of do
out-of-the-box error analysis.
What they did is have linguists go through
and label specific examples for whether or not
they manifested specific linguistic phenomena, like do
the premise and hypothesis involve variation
in active-passive morphology?
Which might be a clue that the sentences are
synonymous or in an entailment relation,
but nonetheless hard for systems to predict because
of the change in word order.
We also have things like whether there are belief statements,
conditionals, whether coreference
is involved in a nontrivial way, modality,
negation, quantifiers-- things that you might think
would be good probes for the true systematicity of the model
you've trained.
And you can use these annotations
to kind of benchmark yourself there.
I think that's incredibly productive.
How are we doing on MulitiNLI?
So again, we're going to have our score over here
and on the x-axis, time.
We have that human estimate at 92.6%.
And since it's on Kaggle, we can look at lots more systems.
For SNLI, we just have the published papers.
But on Kaggle, lots of people enter
and they try lots of different things.
As a result, you get much more variance across this.
It's much less monotonic.
But nonetheless, you can see that the community is rapidly
hill climbing toward superhuman performance on this task,
as well.
And again, I would just want to reiterate, recalling themes
from our introductory lecture, this
does not necessarily mean that we have systems
that are superhuman at the task of common sense
reasoning, which is a very human and complex thing, but rather,
systems that are just narrowly outperforming
humans on this one particular very machine-like metric, which
gives us our estimate of human performance here.
Still, startling progress.
And then finally, adversarial NLIs,
kind of a response to that dynamic that looks like we're
making lots of progress.
But we might worry that our systems are benefiting
from idiosyncrasies and artifacts in the data sets,
and that they're not actually good at the kind
of human reasoning that we're truly trying to capture.
And that gave rise to the Adversarial NLI project.
The paper is Nie, et al., which also
involves some authors from earlier data sets, SNLI
and MultiNLI.
It's another large data set.
A little bit smaller, but you'll see why
it's special in some respects.
The premises come from very diverse sources.
We don't have the genre overfitting
you might get from SNLI.
And the hypotheses were again, written by crowdworkers.
But here, crucially, they were written not in the abstract,
but rather with the goal of fooling state-of-the-art
models.
That's the adversarial part of this project.
And this is a direct response to this feeling
that results in findings for SNLI and MultiNLI,
while impressive, might be overstating
the extent to which we've made progress on the underlying
task of common sense reasoning.
So here's how the dataset collection
worked in a little more detail.
I think this is a fascinating dynamic.
The annotator was presented with a premise sentence and one
condition, which would just correspond to the label
that they want to create.
They write a hypothesis, and a state-of-the-art model makes
a prediction about the premise-hypothesis pair,
basically predicting one of these three condition labels.
If the model's prediction matches the condition,
the annotator returns to step 2 to try again
with a new sentence.
If the model was fooled, though, the premise hypothesis pair
is independently validated.
So in this way, we're kind of guaranteed
to get a lot of examples that are
very hard for whatever model we have
in the loop in this process.
Here are some more details.
So it has three rounds, this data set,
for its first release.
Overall, that results in that large data set.
And you can see that in subsequent rounds,
the model is going to be expanded
to include previous rounds of data,
in addition, possibly, to other data resources.
And so what we're hoping is that as we progress
through these rounds, these examples
are going to get harder and harder in virtue of the fact
that the model is trained on more data
and is getting better as a result of seeing
all these adversarial examples.
In terms of the splits, the train set
is a mix of cases where the model's predictions were
correct and where it was incorrect,
because sometimes in that loop, the annotator was
unable to fool the model after some specified
number of attempts.
And we keep those examples, because they're nonetheless
interesting training data.
However, in the dev and test sets,
we have only examples that fooled the models.
So with respect to the best model for each round,
the test set is as adversarial as it could possibly get.
The model has gotten every single example wrong.
And Adversarial NLI is exciting because it's
given rise to a whole movement around creating
adversarial datasets.
And that's represented by this open-source project, Dynabench.
And we just recently published a paper
that's on the Dynabench effort, reporting
on a bunch of tasks that are going to use approximately
adversarial NLI techniques to develop datasets that are
adversarial in lots of domains.
And you've actually seen one of these in the Dynasent dataset
from our previous unit on sentiment analysis.
And here's the Dynabench interface.
And I guess I'm just exhorting you,
if you would like to get involved in this effort,
it's a community-wide thing to develop better benchmarks that
are going to get us closer to assessing how much progress
we're actually making.
And then finally, there are a lot of other NLI
data sets that I didn't mention.
So let me just run through these.
The GLUE benchmark has a lot of NLI tasks
in it, as does SuperGLUE, which is its successor.
I mentioned before, in the context of ANLI,
this NLI-style FEVER dataset.
FEVER is fact verification, and I've just translated
the examples into NLI ones.
Here's an NLI corpus for Chinese,
and here's one for Turkish.
The Chinese examples are all original,
and the Turkish one is a translation
with validation of SNLI and MultiNLI into Turkish.
XNLI is a bunch of assessment data sets
that is dev-test splits for more than a dozen languages,
drawing on the MultiNLI examples.
Those are human-created translations
that could be used to benchmark multilingual NLI systems.
And then there are a few others down
here kind of pointing out trying to get genre diversity,
and then NLI for specialized domains.
Here is medicine and science.
And those could be interesting for seeing how well a model can
grapple with variation that comes
in very specific and maybe technical domains.
So there's a wide world of tasks you can explore,
and I think that makes NLI a really exciting space
in which to develop original systems, and projects,
and so forth.