CHRISTOPHER POTTS: Welcome, everyone.
This is the first screencast in our series
on natural language inference, or NLI.
This is one of my favorite problems.
What I'd like to do is give you a sense
for how the task is formulated and then
situate the task within the broader
landscape of ideas for NLU.
As usual, we have a bunch of materials
that would allow you to get hands-on with this problem.
The core module is nli.py, and then there are two notebooks.
The first introduces the three data
sets that we'll be exploring in detail, SNLI, multiNLI,
and adversarial NLI.
And the second notebook offers a bunch of different modeling
approaches.
It really covers core approaches that people
have taken to NLI in the past, and I
hope it points to some avenues for modifying
those architectures, possibly in the service of developing
an original system for a final project.
There's also an associated homework and bakeoff.
I emphasize that this is not required for us this quarter.
I'm mentioning it because I think the problem is
an interesting one.
It's a word entailment task which
is an interesting small-scale problem
that I believe could be used to stress-test an NLI
system in interesting ways.
The core readings cover the three datasets
that will be in focus for us.
And then the final reading listed here,
was I think, the paper that introduced attention mechanisms
into the study of NLI and that had an impact that
went well beyond this task.
And then, for additional readings,
I'm kind of suggesting a range of things.
Some of these readings cover core fundamentals
for deep learning that I think will
be useful in the context of studying NLI.
Some of them help you with kind of a foundational understanding
of the NLI task and how you might think about it.
And then some of them are meant to push
us to stress test our system, think adversarially, and maybe
find artifacts in our datasets.
And those are going to be themes of later screencasts
in the series.
To begin getting a sense for how the task is formulated,
let's start with some simple examples here.
So in NLI, we have as our inputs a premise sentence
and a hypothesis sentence, and the task
is a classification one.
So in this simple example here, the premise sentence
is, "a turtle danced."
The hypothesis sentence is "a turtle moved."
Both of those are system inputs, and our task
is to assign one of three labels.
In this case, the correct label would be "entails."
The second example looks simple, but it actually
begins to suggest how the task is actually formulated.
We have as our premise, turtle.
And as our hypothesis, linguists.
And what we would like to do in the context of NLI
is assign that the contradicts label.
Now, you might pause there and think,
it's not a logical fact that turtles can be linguists,
so surely contradiction is too strong,
but it is a common sense that to kind
of natural inference about the world
we live in that no turtles are linguists,
and it's for that reason that we would choose
the contradicts label, and that begins to key into the fact
that fundamentally NLI is not a logical reasoning
task, but a more general common sense reasoning task.
"Every reptile danced" is neutral with respect
to "a turtle ate" which is just to say that these two
sentences can be true or false independently of each other.
And now, with entails contradiction neutral,
we have the three labels that are standardly used
for NLI datasets at this point.
Look at some additional examples.
"Some turtles walk" contradicts "no turtles move."
I think that's straightforward.
Here's one that shows how intricate this could get.
So the premise is "James Byron Dean refused
to move without blue jeans."
Entails "James Dean didn't dance without pants."
This highlights two aspects of the problem.
First, you might have to do some complex named entity
recognition on James Byron Dean and James Dean
to figure out that these are coreferring expressions.
And you also might encounter real linguistic complexity,
in this case, emphasizing things involving how negations
interact with each other.
This next example begins to show how much common sense reasoning
could be brought into the task.
So the premise is Mitsubishi's new vehicle sales in the US
fell 46% in June, and the hypothesis is
Mitsubishi's sales rose 46%.
And we would standardly say that that is in the contradiction
relation.
Now again, you might pause and think
it is certainly possible even in our world
that Mitsubishi could see a 46% rise
and fall in the same month.
So surely these should be labeled neutral.
But I think what you'll find in NLI datasets is that these
are called contradiction on the informal assumption
that the premise and hypothesis are
talking about the same event.
And in that context, we would say that these are common sense
contradictions.
Here's another example that highlights
how much pragmatics could be brought into the problem.
The premise is Acme reported that its CEO resigned,
and the hypothesis is that Acme's CEO resigned.
We would probably say entailment there.
Even though, in a strict logical sense,
the premise does not entail the hypothesis because, of course,
the company could be reporting things that are false.
But here, we kind of make an assumption
that the company is an authority and will likely
report true things about facts like this,
and therefore we allow that this would
be in the entailment relation.
Again not logical but much more like common sense.
So just to emphasize this, here is
kind of the fundamental question that we confront.
Does the premise justify an inference to the hypothesis?
Commonsense reasoning, rather than strict logic.
Two other characteristics of this task
in the modern era are, first, there's
a focus on local inference steps.
That is just one premise and one hypothesis,
rather than long deductive chains.
And the second is that the emphasis
is really on the variability of linguistic expressions,
so when people have created the large benchmark tasks
in this space, they have largely focused on just collecting
a lot of data and not placed any special emphasis
on collecting examples that have a lot of negations
or quantifiers or something that would really
shine a spotlight on linguistic and semantic complexity.
So that's worth keeping in mind about how
we're thinking about the task in the present day.
If you would like additional perspectives
on this, including some disputes about exactly how to think
about the problem, and what would be the most productive,
I would encourage you to check out of these three papers
by a lot of Stanford researchers.
I think the fundamental outcome of this
is that we do want to focus on common sense reasoning,
even though that's a kind of amorphous
and difficult to define concept.
It's nonetheless arguably the useful one for us
when we think about developing practical systems.
Now, in a visionary paper that really set the agenda
for NLI, Dagan et al, 2006, they make
a lot of connections between NLI and the broader
landscape of NLU.
So let me just read this opening statement here.
It seems that major inferences, as needed
by multiple applications, can indeed
be cast in terms of textual entailment.
Consequently, we hypothesize that textual entailment
recognition is a suitable generic task
for evaluating and comparing applied semantic inference
models.
Eventually, such efforts can promote the development
of entailment recognition engines
which may provide useful generic modules across applications.
So wonderful vision and a spin we might put on
in the present day is that what we might hope from NLI
is that since reasoning about entailment and contradiction
are truly fundamental to our use of language,
that pre-training on the NLI task
might give us representations that are useful in lots
of different contexts.
And Dagan et al actually continue
by showing that we can formulate a lot of traditional tasks
as NLI tasks.
And here are just a few examples of that.
If our task is paraphrased, we might
say that in the NLI context, that
means we want equality or mutual entailment between the text
and the paraphrase.
That is premise and the hypothesis.
For summarization, we would do something weaker.
We would hope just that the text, the original text,
entailed the summaries allowing that the summary might
be weaker or more general.
For information retrieval, we kind of do the reverse.
Here we want to find documents that entail the query.
And then for question answering, it's kind of similar.
We could formulate that as an entailment task
by saying that what we want is to find answers that entail
the question, and the way we might think about entailment
for questions is kind of illustrated here,
where we would informally convert a question like,
who left?
Into someone left, to give us a statement.
And then we could say that Sandy left is an answer to, who left?
In the sense that it entails someone left.
And I think there are many other tasks that we
could formulate in this way.
And it does show you just how fundamental entailment
and contradiction are to reasoning and language.
And finally, let me give you a sense for the model landscape
and how it has changed.
NLI is a pretty old problem in the field, and as a result,
we've seen a wide spectrum of different approaches.
In the earliest days, you had a lot
of systems that were kind of focused on logic and theorem
proving.
And I've characterized those systems here
as offering really deep representations,
but they weren't especially effective in the sense
that they worked only for the domains and examples
that the system designers had been able to anticipate.
So they're kind of brittle.
Following that, you have a kind of exploration
of what Bill MacCartney called natural logic approaches.
Bill was one of the early innovators in this space.
I think he actually coined the term natural language
inference.
And he explored natural logic, which
has some of the aspects of logic and theorem proving,
but it's kind of more open and easily amenable
to tackling a lot of data.
And so those systems were consequently
a little less steep but also more effective,
and a similar thing happened with these semantic graphs,
which is providing rich, conceptual representations
of the underlying domain that we want to reason about.
Another interesting thing here is that, until recently,
it was the case that clever hand-built features, which
I'll show you some a bit later in the screencast series.
They were really in the lead and simple n-gram variations,
traditional models with hand-built features,
they were the best models.
There was a kind of faith early on in the deep learning
revolution that eventually, those models would prove
to be the best at this task.
But at the time, we just didn't have the data sets
that would support--
that would provide evidence for that kind of claim.
And so as a result, for a while, deep learning systems
really lagged behind more traditional approaches.
And I would say that it was really in about 2017
that deep learning pulled ahead.
And that's a result of modeling innovations,
and also the arrival of some really large benchmark
datasets that would allow us to train systems that
were effective for the task.
And it's at that point you see that deep learning kind of took
over, and as a result, in subsequent screencasts,
we too will be focused on deep learning architectures
for the NLI problem.