CHRISTOPHER POTTS: Welcome back, everyone.
This is part 3 in our series on supervised sentiment analysis.
This screencast is going to focus on the Stanford Sentiment
Treebank.
Let me start with a quick project overview.
The associated paper is Socher et al.
2013.
I think this paper is a kind of model of open science.
At this website here, you can see
the full code, all the data of course,
as well as an API that will let you try out new examples
and kind of interact with the core models
that are motivated in the paper.
It's a sentence-level corpus.
It's got about 11,000 sentences in total.
And all of those sentences are originally
from Rotten Tomatoes.
So they are sentences from movie reviews.
The sentences themselves were originally
released by Pang and Lee in 2005.
It's a kind of classic data set.
And what the SST did was expand the data set
by labeling not only the full sentences but all
of the sub-constituents according
to a kind of traditional parse, syntactic parse,
of each of the examples.
And those are all crowdsourced labels.
So what this means is that we have vastly more supervision
signals all throughout the structure of these examples
than we would get from the original, where we just
had a single sentiment label for the entire sentence.
The labels themselves and the underlying corpus
are five-way labels that are extracted from workers' slider
responses.
So there's kind of an initial layer of aggregation.
They made a slider choice.
They were all grouped together into five labels.
And then we are going to work with a formulation that
is even more collapsed down to ternary sentiment.
I'll return to that a bit later.
The fully labeled tree thing is one
of the really exciting aspects of this corpus
that we will be able to take advantage of, especially
during training.
So the way that worked is, there were parses.
This is a simple constituent parse of a sentence,
NLU is enlightening.
And as I've indicated here, we have
labels in that space, 0 through 4,
on all of the lexical items, NLU, is, and enlightening,
as well as all the subconstituents in this phrase.
And so you can see that "is" is neutral.
But since "enlightening" is positive,
the whole verb phrase, "is enlightening" is positive.
We can say that NLU is neutral.
But in the context of this sentence,
the overall contribution is a highly positive one.
So label 4 on the root.
In the first screencast for this unit,
I motivated sentiment analysis with some cases
that I thought were kind of difficult
from a syntactic point of view.
This is one of them, they said it would be great.
I love how this is being handled.
We can see that down here, "be great"
is kind of clearly positive.
But by the time we have filtered that through this report,
they said just kind of displacing the sentiment
onto another agent.
The speaker is not necessarily endorsing
the claim of greatness.
What we get in the end is more like a neutral sentiment.
I think that's interesting.
And we can extend that even further, right?
These are actual predictions from the model that's motivated
in the underlying paper.
If we take that constituent that I just showed you
and conjoin it with "they were wrong,"
which is clearly negative, strikingly the model
is able to figure out that the overall sentiment is determined
by this second clause here, and assigned negatives
to the entire thing, despite the fact that there are obviously
subconstituents in here that are positive.
And that exactly the kind of mixing that I think
is correct for how language works
in the domain of sentiment.
And it's kind of encouraging to see
that this model is able to capture
at least some aspects of it.
Here's a similar case that I think is pretty good as well.
Although, maybe not as strikingly positive in the end
here.
I've just changed from the previous example,
"they were wrong," to "they were right."
It knows that right is correct.
And it seems to get that this is middle of the scale.
And I'd hope this was a 3 or a 4.
But I think that still we're seeing
some interesting interactions between what's
happening in subconstituents in these examples
and the prediction that's made at the root level.
So it's very encouraging.
There are a bunch of ways that you can formulate the SST task.
Kind of the raw one that comes from the paper would
be a 5-way classification problem where we have
these numerical labels here with the meaning of kind of 0
is very negative, 1 is negative, 2 is neutral, 3 is positive,
and 4 is very positive.
I think this is fine, but there are two gotchas
underlying this kind of scheme.
First, it's not really a fully ordered scale in the sense
that 4 is stronger than 3, but 0 is stronger than 1,
because we have kind of this polarity split
with neutral in the center.
So that's a kind of conceptual difficulty.
And then the other part is that, by and large, classifier models
that you pick will not give you partial credit for being close.
We might hope that a model that predicted a 1, negative,
was kind of right or certainly more right
if the true label is 0 than a model that had predicted 4.
But of course, if these are all treated
as independent classification bins,
then you're just equally wrong no matter
which prediction you made relative to the gold label.
And that seems unfair to our models.
We are going to work with what I've
called the ternary problem.
I think this is the minimal problem that really makes sense
conceptually.
For this one, we grouped 0 and 1 into a negative category,
3 and 4 into a positive category,
and reserved 2, as before, for what we're calling neutral.
And this kind of avoids the false presupposition
that every sentence is either negative or positive,
because it does allow us to make predictions
into this neutral or non-sentiment laden space.
It's very common, and you see this in the paper
as well as in a lot of work on the SST,
to formulate this as a binary problem.
For the binary problem, we simply
remove the middle of the scale and treat 0 and 1
as negative and 3 and 4 as positive, as before.
I think that has two drawbacks.
First, we have to throw away some data.
And second, then we're making this false presupposition
that every sentence is either classified as negative or as
positive when for a wide range of cases in the world,
that might be inappropriate.
Now, I focused here on the root level problem.
You can see that the numbers here for train and dev
are small, and the test set numbers are a little bit larger
than for devs.
So they're comparable.
But we can also think of this as the all nodes task,
because recall that a hallmark feature of the SST
is that every single subconstituent
in these examples has been labeled by crowd workers.
So we could treat each one of those
as a kind of independent classification problem.
We have the same range for all of the values
that they can take on.
So we can do similar kind of collapsing down
into the ternary problem or the binary problem.
And of course, here, we have a much larger dataset.
For us, we're going to, by and large,
work with the data in one particular way, which I think
is common in the literature.
As I said, we're going to have the ternary formulations.
Our labels will be positive, negative, and neutral.
When we do the dev and test step,
we are going to test only on full examples.
So for them, we will not make predictions
into the subconstituent space.
And then as a default for the code, as you'll see,
it is set up to train only on full examples.
So for these two cases, NLU is enlightening
and not enlightening here, if those
were two independent sentences in the corpus,
we would train just on those two independent examples,
one negative labeled positive and the other labeled negative.
However, you might imagine that you'll get a lot more strength
in training if you also trained on all the subconstituents,
which would mean essentially expanding
this example into its full root version,
NLU is enlightening, but also all
of the sub-pieces that are captured and labeled
in the corpus.
So that would give you many more examples and much more
diversity.
And then of course, "not enlightening"
would be split apart as well.
And then you could decide for yourself in addition
whether you want to treat this as two instances
of "enlightening" or one.
And the code facilitates all this.
So you can formulate it as a root only training scenario
or as a subconstituent training scenario.
And you can keep or remove duplicates.
This is going to impact the amount
of computational resources that you need for training models.
But of course, bigger could be better in this space,
because you're just seeing much more gold labeled information.
So that's the overview.
And for much more on this, and how
to work with our distribution of the corpus,
and so forth, I would encourage you
to work through this notebook that I've linked at the bottom
here.