CHRISTOPHER POTTS: Welcome, everyone.
This is part 7 in our series on supervised sentiment analysis.
The focus of this screencast is on feature representation
of data.
There are really two things I'd like to do.
First, just explore some ideas for effective feature
representation in the context of sentiment analysis.
And, second, cover some of the core technical concepts
that surround feature representation that you'd
do well to have in mind as you write new feature functions
and optimize models.
Let's begin in a familiar place which
is N-gram feature functions.
To this point in the series of screencasts,
I've been just focusing on unigram feature functions.
That's also called the "bag-of-words" model,
and we can easily generalize that idea to bigrams,
and trigrams, and so forth.
All of these schemes will be heavily
dependent on the tokenizer that you've
chosen because, of course in the end, for every example
we represent, we are simply tokenizing that example
and then counting the tokens in that example.
This can be combined of course with preprocessing steps.
In part 2 in this series, I covered the preprocessing idea
of _NEG marking.
Which is essentially to mark words
as they appear in a heuristic way
in the scope of negative morphemes
as a way of indicating that, for example, "good"
is positive in normal context but might
become negative when it is in the scope of a negation
like "not" or "never."
We would handle that as a preprocessing step and that
would just create more unigrams that our tokenizer would turn
into tokens and then would be counted by these feature
representation schemes.
A hallmark of these feature approaches
is that they create very large, very sparse feature
representations.
You are going to have a column in your feature representation
for every single word that appears anywhere
in your training data.
And another important thing to keep
in mind about this approach is that, by and large, they
will fail to directly model relationships between features
unless you make some special effort to effectively interact
these features.
All you'll be doing is studying their distribution with respect
to the class labels that you have.
And it's very unlikely that you'll
recover, in any deep way, the kind
of underlying synonymy of words like "couch" and "sofa,"
for example.
And this is a shortcoming that we
might want to address as we move into distributed
representations of examples like deep learn.
So for our first technical concept,
I would like to just distinguish between feature functions
and features.
And to do this, I've just got a fully worked out example
here using tools from scikit-learn
that I think will make the importance of this distinction
really clear and concrete.
So in cell 1, I've just loaded a bunch of libraries.
In cell 2, I've got my standard, kind of lazy unigrams
feature function, which is taking in a stringed text,
downcasing it, and then simply splitting on whitespace.
And then the counter here is just turning that
into a count dictionary mapping each token
to the number of times that it appears in this example,
according to our tokenizer.
That would be fine for now.
In cell 3, I have a tiny little corpus that
has just two words, a and b.
In cell 4, I create a list of dictionaries
by calling unigrams_phi on each of the texts in my corpus here.
So that gives me a list of count dictionaries.
In 5, I use a DictVectorizer, as covered
in a previous screencast.
And what that's going to do is when I call fit_transform
on my list of feature dictionaries,
it will turn it into a matrix, which
is the input that all of these scikit machine learning models
expect for their training data.
And in cell 7, I've just given you
what I hope is a pretty intuitive view of that design
matrix.
Underlyingly, it's just an NumPy array.
But if we use pandas, we can see that the columns here
correspond to the names of each one of the features.
Because we have just two word types in our corpus,
there are two columns, a and b, and each of the rows
corresponds to an example from our corpus.
And so you can see that our first example has been reduced
to a representation that has 3 in its first dimension and 0
in its second, corresponding to the fact that it has three
a's and no b's.
Example 2, a, a, b is represented as a 2
in the first column and a 1 in the second column,
and so forth.
So that's a first distinction.
We have this feature function here, which is like a factory,
and depending on the data that come in for our corpus,
we're going to get very different features which
correspond to each one of the columns in this feature
representation matrix.
Let's continue this a little bit and think
about how this actually interacts with the optimization
process.
So in cell 7 here, I've just repeated that previous matrix
for reference.
In cell 8, I have the class labels for our four examples,
and you can see there are three distinct classes--
C1, C2, and C3.
I set up a logistic regression model,
although that's not especially important,
it's just a useful illustration.
And I call fit on my pair x, y.
That is my feature representations and my labels,
and that's the optimization process.
As part of that, and for a convention for scikit,
the optimization process creates this new attribute coef_
and this new attribute classes_.
coef_ here, these are the weights that we learned as part
of the optimization process, and of course classes_ corresponds
to the classes that inferred from the label y that we input.
And here I'm just using a pandas data frame again
to try to make this intuitive.
It's really just a NumPy array, this coef_ object here.
And you can see that the resulting matrix has
a row for each one of our classes
and a column for each one of our features.
And that's a useful reminder that what the optimization
process for models like this is actually doing
is learning a weight that associates class feature name
pairs with a weight, right?
So it's not just that we learn individual weights
for features, but rather we learn them with respect
to each one of the classes.
And that's a hallmark of optimization
for multi-class models like this one.
And then in cell 12, I've just shown you that you can actually
use the coef_ and this other bias term, intercept_,
to recreate the predictions of the model.
All you're doing is multiplying examples by those coefficients
and adding in the bias term.
And this matrix here is identical to what
you get in scikit, if you simply directly call predict_proba
for predict probabilities on your examples.
Let's turn back to what we're trying
to do to create good models, having those ideas in mind.
So let's just cover a few other ideas for hand-built feature
functions that I think could be effective for sentiment.
So, of course, we could have lexicon-derived features.
I earlier showed you a bunch of different lexicons.
And that could be used to group our unigrams.
So we could have these feature functions work in conjunction
with a "bag-of-words" or "bag-of-acronyms" model,
or we could use them to replace that model and develop
a sparser feature representation space.
We could also do the negation marking
that I mentioned before, and we could generalize that idea.
So many things in language take scope
in a way that will affect the semantics of words
that are in their scope.
So another classical example behind-- besides negation
is these modal adverbs like "quite possibly" or "totally".
We might have the idea that they are modulating
the extent to which the speaker is
committed to "masterpiece" or "amazing," in this case.
And keeping track of that semantic association
with simple underscore marking of some kind
might be useful for giving our model a chance
to see that these unigrams are different
depending on their environment.
We can also have length based features,
and that's just a useful reminder that these don't all
have to be count features.
We can have real valued features of various kinds
and they could signal something important about the class
label.
For example, I think neutral reviews,
three star reviews tend to be longer than one and five
star reviews, so might as well throw that in.
And we could expand that idea of float valued
features a little bit more.
I like the idea of thwarted expectations
which you might keep track of as the ratio
of positive to negative words in a sentence.
The idea being that very often if that ratio is exaggerated,
it's telling you the opposite story that you might expect
about the overall sentiment.
Many, many positive words stacked up together
might actually be preparing you for a negative assessment
and the reverse.
But the important thing about this feature
is that it wouldn't decide for you what these ratios mean.
You would just hope that it was a useful signal
that your model might pick up on as part of optimization
to figure out how to make use of the information.
And then, finally, we could do things, various kinds
of ad-hoc feature functions to try
to capture the fact that many uses of language
are non-literal and might be signaling exactly
the opposite of what they seem to do on their surface.
Like, "Not exactly a masterpiece."
is probably a pretty negative review.
It was "Like 50 hours long." is not saying that it was actually
50 hours long but rather, with hyperbole, indicating
that it was much too long or something like that.
And "The best movie in the history of the universe."
could be a ringing endorsement, but it could just as easily
be a bit of sarcasm.
Capturing those kind of subtle distinctions
is, of course, much more difficult.
But the hand-built feature functions
that you write where you could try to capture it,
and if they have a positive effect,
then maybe you've made some real progress.
And that's a good transition point
to this topic of assessing individual feature functions.
As you can see, the philosophy in this mode of work
is that you write lots of feature functions
and kind of see how well they can do at improving
your model overall.
You might end up with a very large model
with many correlated features and that
might lead you to want to do some feature selection to weed
out the ones that are not contributing in a positive way.
Now, scikit-learn has a whole library
for doing this called feature selection,
and it offers lots of functions that
will let you assess how much information your feature
functions contain with respect to the labels
for your classification problem.
So this is very powerful.
And I encourage you to use them, but you should be a little bit
cautious.
Take care when assessing feature functions individually
because correlations between those features
will make the assessments very hard to interpret.
The problem here is that your model is holistically
thinking about how all of these features
relate to your class label and figuring out
how to optimize weights on that basis,
whereas the feature function methods, many of them,
just look at individual features and how
they relate to the class label.
So you're losing all that correlational context.
And to make that a little concrete,
I just cooked up an example here, an idealized one, that
shows how you could be misled.
So I have three features, x1, x2, and x3
in a simple binary classification problem.
And I use the chi-square test from feature selection
to kind of assess how important each one of these features
is with respect to this classification problem.
And what I found is that, intuitively, it
looks like x1 and x2 are really powerful features.
And that might lead me to think, well, I'll
drop the third feature and include
just one and two in my model.
So far, so good.
However, if we thoroughly explore this space, what
we find is that in truth a simple linear model performs
best with just feature x1 and actually
including x2 hurts the model, despite the fact
that it has this positive feature importance value.
So what you really ought to be doing
is using just this single feature,
but these methods can't tell us that.
And even a positive feature selection value
might actually be something that's
at odds with what we're trying to do with our model,
as this example shows.
So, ideally, what you would do is
consider more holistic assessment methods
which scikit also offers.
This would be things like systematically removing
or perturbing feature values in the context of the full model
that you're optimizing and comparing performance
across those models.
This is much more expensive because you're
optimizing many, many models.
So it might be prohibitive for some classes of models
that you're exploring.
But if you can do it, this will be more reliable.
However, if this is impossible, it
might still be productive to do some feature selection
using simpler methods.
You should just be aware that you
might be doing something that's not
optimal for the actual optimization problem
that you've posed.
OK, and the final section of this screencast
is a kind of transition into the world of deep learning.
I've called this distributed representations as features.
This is a very different mode for thinking
about representing examples.
What we do in this case is take our token stream as before
but instead of writing a lot of hand-built feature functions,
we simply look up each one of those tokens in some embedding
that we have.
For example, it could be an embedding
that you created in the first unit of this course.
Or it could be a GloVe embedding or a static embedding
that you derived from BERT representations,
and so forth and so on.
The important thing is that each token is now
represented by a vector and that could be a powerful idea
because in representing each of these words as vectors,
we are now capturing the relationships
between those tokens.
We might now have a hope of seeing
that "sofa" and "couch" are actually
similar features in general and not
just with respect to the class labels that we have.
So that's the idea why this might be powerful.
So we take all those vectors and look them up.
However, for all these classifier models,
we need a fixed dimensional representation
to feed into the actual classifier unit.
So we're going to have to combine those vectors
in some way, and the simplest thing you could do
is combine them via some function like sum or mean,
right?
So take all these things, for example,
and take their average and that would give me another fixed
dimensional representation, no matter how many tokens are
in each one of the examples.
And that average vector would be the input to the classifier.
So if each one of these vectors has dimension 300, then so,
too, does the feature representation
of my entire example.
And I now have a classifier which
is processing feature representations that
have 300 columns.
Each dimension in the underlying embedding space
now corresponds to a feature.
And that's the basis for optimization.
And I'd say an eye opening thing about this class of models
is despite them being very compact--
300 dimensions versus 20,000 that you might have from
a "bag-of-words" model--
they turn out to be very powerful.
And this final slide here just shows you
how to implement those using tools and other utilities
for our course.
So I'm going to use GloVe, and I'm
going to use the 300 dimensional GloVe space which is included
in your data distribution.
In 4 and 5 here, we just write simple feature functions
and the hallmark of these is that they are simply
looking up words in the embedding
and then combining them via whatever function they
use or specifies.
So the output of this is directly
a vector representation of each example.
In cell 6, we set up a logistic regression, as before.
Of course, it could be a much fancier model
but logistic regression will do.
And then we use sst_experiment almost exactly as before.
The one change we need to remember
to make in operating in this mode is
to set the flag of vectorized equals false.
We already have each example represented as a vector,
so we do not need to pass it through that whole process
of using a DictVectorizer to turn count
dictionaries into vectors.
And as I said before, these turn out
to be quite good models despite their compactness.
And the final thing I'll say is that this model
is a nice transition into the recurrent neural networks
that we'll study in the final screencast for this unit, which
essentially generalize this idea by learning
an interesting combination function for all
the vectors for each of the individual tokens.