Hello, everyone.
This video kicks off our series of screencasts
on supervised sentiment analysis.
I just want to provide you with an overview of the problem
and of the kind of work we'll be doing, and also
a rationale for why we'll be doing it.
So here's an overview of the entire unit.
I want to, in this screencast, motivate for you
the idea that sentiment analysis is
a deep problem and an important problem for NLU,
not only scientifically but also for industrial applications.
In the next screencast, I'll give you
some practical general tips for doing sentiment analysis.
Following that, we'll have two short screencasts that
introduce our core data sets, the Stanford Sentiment Treebank
and a new data set called DynaSent.
After that, I'll introduce the code base
that will be working with on the assignment in the bake-off.
That's sst.py, which is included in the course code
distribution.
And I'm going to use that code to illustrate
some important methodological issues surrounding
supervised analysis in general, which would be hyperparameter
tuning and comparison of different classifiers
to see whether they are different in some significant
statistical sense.
Then we'll talk about feature representation,
both for large sparse linear models with handbuilt features,
and also with more deep learning oriented
distributional representations.
And that will be a nice segue into the final unit, which
is on using recurrent neural networks as classifiers
for supervised sentiment analysis.
My hope is that this unit can provide a refresher
on core concepts in supervised learning,
introduce you to the problem of sentiment analysis, which
I think is, as I said, a central problem for natural language
understanding, and also set you on your way
toward doing the assignment and the bake-off
and possibly building projects in this space.
For the associated materials, as I said,
we've got a bunch of code. sst.py is the core module.
And then we have a notebook introducing
the Stanford Sentiment Treebank as a dataset.
We have a second notebook that's on what
I've called handbuilt features and mostly linear models,
and then a third notebook that's on using neural networks, which
more or less pushes you to using distributional representations
instead of handbuilt features, although, as you'll see,
the notebooks explore various combinations of these ideas.
The homework in the bake-off is in the notebook homework
sentiment.
And I'm going to introduce that probably
in a separate screencast.
The core readings are the two papers
that are oriented around our data sets, the Standard
Sentiment Treebank and DynaSent.
And as supplementary readings, you
might enjoy this compendium from Pang and Lee.
It's a kind of overview of the whole field of sentiment
analysis.
And it poses challenges and questions that
are still relevant to this day.
And then Goldberg 2015 is an excellent overview
of using neural networks in NLP, very generally.
But with lots of helpful notation and so forth that
we're aligned with.
And that might help you get a feel
for the landscape of modeling choices
that you might make in this space
and in subsequent units for this course.
So I want to start by just motivating
the idea that sentiment analysis is an interesting problem.
Because you often hear people say things like sentiment
analysis is solved or it's overly simplistic or just too
easy.
And I think none of those things are true.
And to motivate that, I just want to do a little data driven
exercise with you.
So for these examples, you should ask yourself,
which of these sentences expresses sentiment at all?
And for the ones that you think do express sentiment,
what is that sentiment?
Is it positive or negative or maybe neutral
or something else?
So you might think those are straightforward questions
but this is going to get difficult really fast.
Consider the first example.
There was an earthquake in California.
This is probably going to sound like bad news to you.
And many sentiment analysis systems
will assign this negative sentiment.
But we should ask ourselves, is this actually
a sentiment-laden sentence?
It is on the face of it merely stating a fact.
And we might hold that for sentiment to be expressed,
we need some kind of subjective evaluative perspective
to be included in here, like it was bad that there was
an earthquake in California.
And absent the "it was bad" clause,
this might just be a neutral statement of something
that had happened.
But the important point here is that unless we settle
these questions, we'll have continued indeterminacy
about what we're actually doing.
The team failed to complete the challenge.
Is that positive or negative?
We might agree that it's more than just a statement of fact,
although it's a borderline case even for that question.
But if we did decide it was sentiment-laden,
we would need to figure out the perspective of the speaker.
Is the speaker advocating for this team
or advocating for a different team, right?
We win.
We lose.
It's really going to depend on how the speaker is involved.
And that, of course, is going to have
to become part of our definition of what we're doing when
we assign sentiment labels.
They said it would be great.
On the face of it, this expresses no speaker
perspective at all.
This is merely reporting what somebody else said.
And we need to decide for those obviously different
perspectives what we're going to do
in terms of sentiment analysis.
Because after all, this could continue.
They said it would be great and they were right, which
is straightforwardly positive.
But it could also continue, they said it would be great
and they were wrong.
And I think that reveals that sentence 3 is not so obviously
encoding a particular speaker perspective,
whereas these clauses are what really tell the story for us
as sentiment analysts.
And then we get into things that you might call
non-literal use of language.
The party fat cats are sipping their expensive imported wines.
This has a lot of positive language in it.
Maybe only fat cats is the thing that
sounds like a direct smear.
But I think we could agree that overall, this is probably
negative in its valence.
And that will be a challenge for our systems
and also a challenge for us in just
characterizing precisely what was done here
in terms of sentiment.
Here's a similar example, oh, you're terrible.
This might be a criticism.
And it might therefore be straightforwardly negative.
On the other hand, it could be a kind
of teasing form of social bonding
that overall has a positive effect on the discourse.
How are we going to resolve that kind of context dependence?
Here's another one.
"Here's to ya, ya bastard."
It's got some negative language, even something
that's kind of like a swear.
But this could be a friendly, jocular phrase of some kind.
And we'll have to sort out whether it's friendly and fun
because of its negativity, or whether this
is straightforwardly just a positive sentence.
And then here's a case that's just
going to be a challenge for our systems.
This is out of the movie 2001.
This is from an actual review.
"Many consider the masterpiece bewildering,
boring, slow moving, or annoying."
There is lot of negative language there.
In fact, there's very little that's
positive except "masterpiece."
But I think we can all anticipate that overall this
is probably going to be a positive review of that movie.
So that just shows you that even if we're
clear about what we're doing in terms of sentiment,
the linguistic challenge here is significant.
And we could also extend that to sentiment
like long suffering fans, bittersweet memories,
hilariously embarrassing moments.
These are things that are going to blend positivity
and negativity and all sorts of other emotional dimensions
in ways that just make sentiment analysis very difficult to do
reliably.
And that's a nice segue into this topic of sentiment
analysis in industry.
Because of course, sentiment analysis
is one of the first tasks that was really transformed
by data-driven approaches.
And it was the first task to really make
an impact in industry.
There are lots of startups and companies
that offer sentiment analysis tools.
And it has obvious import for things
like marketing, and customer experience, and so forth.
And I would say the first thing I
would say is that to this day, the sentiment from industry,
so to speak, is that sentiment analysis tools still
fall short.
This is from an article from 2013.
And the gist of it is, anyone who
says they're getting better than 70% today
is lying, generally speaking.
From whatever notion of 70% we have here,
I think we can agree that that's too low and that we as a field
ought to be offering tools that are better.
This is another kind of equivocal headline.
Emotion AI technology has great promise when used responsibly.
Affective computing knows how you feel, sort of.
The sort of is kind of like the equivalent of 70% here.
I think it shows that there's a lot of work to be done if we're
going to have the kind of impact we
want to have in the technological sphere.
And then there's another dimension
to this which we're not going to really get to capture
but is worth planting in your minds
because this could become projects, right?
We're going to do classification of sentiment
into positive, negative, and neutral.
And that's often the starting point for these industry tools.
Many business leaders think they want
these pie charts to point out like 30% negative,
70% positive.
And then in Q2, the negativity is slightly up.
And that's surely a leading indicator of something.
It looks like negativity is on the rise.
But the issue is, what do you do?
How does this help with decision making?
Merely classifying these texts and showing change over time
is not enough for any business leader to take action.
But we need to know why this is happening, what the underlying
factors are, basically what are the customers saying
beyond these gross classifications
into positive, negative, and neutral.
And we should be pushing ourselves
to design tools that can offer that next layer of insight.
Affective computing, this is a kind of transition
into the wider world here.
We're going to focus on just sentiment analysis.
But you could think about emotional analysis
and all other kinds of kind of context dependent expression
in language, put that under the heading of affective computing.
This is a diagram from a paper I did a few years ago with Moritz
Sudhof.
It's a diagram of emotions and other kinds of moods
that people feel.
The arcs give you a transition.
So they show that people tend to transition
systematically from one emotional state to another.
So what we're seeing here is basically
just that this is a very high dimensional space.
It's not just positive, negative, neutral.
We have a wide range of feelings,
and moods and emotions and states that we go into.
And there's a lot of structure to how we experience
those moods in our lives.
And it would be great to break out
of the simple, positive, negative mode
and tackle all of these dimensions.
And in that spirit, here what I've done on this slide
is just list out a whole bunch of other tasks
that you might consider adjacent to sentiment analysis.
But they are meaningfully different from sentiment
analysis, things like subjectivity,
bias, stance taking, hate speech, microaggressions,
condescension, sarcasm, deception and betrayal,
online trolls, polarization, politeness,
and linguistic alignment.
These are all deeply social things that are influenced by
and shape our language.
And I've selected these papers in particular
because all of them have really nice either crisp statements
of the problem and/or really great public data sets
that you could use for experiments in this wide world.
I think that's a very exciting space
to explore as a kind of next step
from what we're doing in this unit.
But back down to Earth here.
Our primary data sets, as I said,
are going to be the ternary formulation of the Stanford
Sentiment Treebank which is just positive, negative, neutral.
And also the DynaSent data set, which has that same ternary
formulation.
The SST is movie reviews.
DynaSent is mostly reviews of products and services,
I think heavily biased toward restaurants
because the underlying data is from Yelp.
And then for the bake-off, we're going
to have a new dev/test split.
We'll use SST-3 as well as this new one
of a corpus of sentences from restaurant reviews.
So you can see that DynaSent might be an asset here.
They all have this ternary formulation.
And I'm hoping that the combination of these data sets
gives us a really interesting perspective,
not only on sentiment analysis but also on kind
of how we design systems that effectively transfer
across domains and maybe learn simultaneously
in multiple domains.