CHRISTOPHER POTTS: Hello, everyone.
Welcome to part 5 in our series on supervised sentiment
analysis.
The focus of this screencast is on the module sst.py,
which is included in the course code distribution.
It contains a bunch of tools that
will let you work fluidly, I hope,
with the Stanford Sentiment Treebank
and conduct a lot of experiments in service
of completing the homework and also doing an original system
entry for the bake off.
Let's say that my goals for the screencast are two-fold.
First, I do just want to get you acquainted with this code
so that you can work with it on the assignment in the bake off.
And in addition, I'd guess I'd like to convey to you some best
practices around setting up a code
infrastructure for a project, say,
that will let you run a lot of experiments
and really explore the space of ideas
that you have without introducing a lot of bugs
or writing a lot of extra code.
So let's begin.
We'll start with these reader functions.
At the top in the first cell here,
I just load in not only OS so that we
can find our files, but also sst which
is the module of interest.
We set up this variable here that's a pointer to where
the data set itself lives.
And then this function, sst.train_reader,
will let you load in a Pandas data frame that contains
the train set for the sst.
You'll notice that there are two optional keywords,
include_subtrees and dedup.
Dedup will remove repeated examples,
and include_subtrees is a flag that
will let you include or exclude all
of the subtrees that the sst contains.
By default, we'll include just the full examples.
But if you said, include subtrees equals true,
you get a much larger data set as we
discussed in the screencast on the sst itself.
In cell 4 here, I'm just giving you a look at one random record
from this.
So remember, it is a PANDAS data frame.
But we can get it as a dictionary for a little bit
of an easier look.
We've got an example ID.
We have the text of the sentence, the label,
which is either negative, positive, or neutral.
And then is_subtree is a flag on whether or not
it's a full root level example or a subconstituent of such
an example.
Since we have loaded this in with include
subtrees equals false, we get this distribution of labels
here.
This is just a distribution of labels on the full examples.
But of course, as we change these flags,
we get very different counts down here.
And then something comparable happens with the dev reader--
dev_df from sst.dev_reader, with a pointer to the home directory
for the data as before.
And here, the subtree distinction
and the dedup distinction, those are much less important
because these data sets consist just of root level examples.
And there are very few, if any, duplicate examples
in those data sets.
Now let's turn to feature functions.
We'll begin to build up a framework for doing
supervised sentiment analysis.
And the starting point here is what I've
called the feature function.
It's given in two, unigrams_phi.
It takes in a text that is a string.
And what it does is return a dictionary that is essentially
a count dictionary over the unigrams in that string
as given by this very simple tokenization scheme, which
just down cases all of the tokens and then
splits on whitespace.
So as an example text, if I have "NLU is enlightening,"
space, and then an exclamation mark,
and I call the feature function on that string,
I get this count dictionary here,
which is just giving the number of times
each token appears in that string
according to the feature function.
I'd say it's really important when you're
working with the standard version of this framework
doing handbuilt feature functions, that you just
abide by the contract that all of these feature functions
take in strings and return dictionaries,
mappings strings to their counts.
Or if you want to, they're Bools or floats or something
that we can make use of when we're doing featurization.
The next step here is what I've called a model wrapper.
And this is going to look a little bit trivial here.
But as you'll see as we move through more advanced methods
in this unit, especially the next screencast,
it's really nice to have these wrappers
around the normal, essentially the fit function down here.
So I'm going to make use of a scikit linear model called
LogisticRegression, a very standard sort
of cross-entropy classifier.
I've called my function fit_softmax_classifier.
And it takes in a supervised data set, so a feature matrix
and a list of labels.
And I set up my model down here.
And I've used some of the keyword parameters.
There are many more for the scikit model.
And then the crucial thing is that I call fit and return
the model, which is now a trained model, trained
on this data set xy.
It might look like all I've done is called fit on a model
that I set up.
But as you'll see, it's nice to have a wrapper function so
that we can potentially do a lot more as part
of this particular step in our experimental workflow.
So now let's just bring all those things together
into what is called sst.experiment,
which is like one-stop shopping for a complete experiment
in supervised sentiment analysis.
So we load in these two libraries.
We get a pointer to our dataset, and then call sst.experiment.
The first argument is the dataset
that it will be trained on.
So that's like train_df from before.
We have a feature function and a model wrapper.
And then these other things are optional.
So if I leave assess_dataframes as none,
then it will do a random split on this train reader
according to train size.
If you do specify some data frames here, a list of them,
then each one will be used as a separate evaluation
against the model that you train on this original data.
You can set the score function if you want.
Our default is macro F1.
And then we'll return to these two options later.
Verbose is just whether you want to print some information.
And Vectorize is an option that you can turn on and off.
And you'll probably turn it off when
you do deep learning experiments, which we'll
talk about later in the unit.
The result of all that is a bunch of information
about your experiments stored in this variable.
And because we had verbose equals true,
you're going to report here.
And this is just a first chance to call out
that throughout this course, essentially, when
we do classifier experiments, our primary metric is
going to be the macro average F1 score.
This is useful for us because it gives equal weight
to all the classes in our data, regardless of their size, which
is typically reflecting our value that we care even
about small classes.
We want to do well even on the rare events in our space.
And it's also perfectly balancing precision and recall
which is like a good null hypothesis
if we're not told ahead of time based on some other goal
whether we should favor precision or recall.
So that all leads us to kind of favor as a default
this macro average F1 score as an assessment
of how the model fit.
And here we've gotten 51.3.
The return value of sst.experiment, as I said,
is a dictionary.
And it should package up for you all the objects and information
you would need to test the model, assess the model,
and do all kinds of deep error analysis.
That is the philosophy here that you should, if possible,
capture as much information as you can about the experiment
that you ran in the service of being
able to do subsequent downstream analysis of what happened.
And so here I'm just giving an example
that we've got the model, the feature function, the train
dataset, whenever our assess datasets were used.
And if that was a random split of the train data,
that will be reflected in these two variables.
The set of predictions that you made
about each one of the assess datasets,
the metrics you chose, and the scores that you got.
And then if you do dive in, like if you look at train set,
it's a standard data set.
x is your feature space.
y is your labels.
Vectorizer is something that I'll return to.
That's an important part about how the internal workings
of sst.experiment function.
And then you have the raw examples in case
you need to do some really serious human level error
analysis of the examples as distinct from how they're
represented in this feature space.
So here is just a slide that brings all of those pieces
together.
This is one-stop shopping for an entire experiment.
We loaded all our libraries.
We have our pointer to the data.
And then the ingredients are really a feature function
and a model wrapper.
And that's all you need in our default setting.
Point it to the train data and it will do its job
and record all you would want for this experiment,
I hope, in this experiment variable here.
There's a final piece.
I want to return to that vectorizer variable
that you saw in the return values for sst.experiment.
And that is making use of what in scikit-learn is
called the DictVectorizer.
And this is really nice convenience function
for translating from human representations of your data
into representations that machine learning models like
to consume.
So let me just walk through this example here.
I've loaded the DictVectorizer, and I've got my train features
here in the mode that I just showed you
where here we have two examples and each one is represented
by our feature function as a dictionary
that maps like words into their counts.
It could be more flexible than that,
but that's like the most basic case that we consider.
And I set up my vectorizer in 3, and then I call fit_transform
on this list of dictionaries.
And the result here, x_train, is a matrix,
where each of the columns corresponds
to the keys in the dictionary representing a unique feature.
And the values are, of course, stored in that column.
So this feature space here has been
turned into a matrix that has two examples, 0 and 1.
There are a total of three features represented
across our two examples, A, B, and C.
And you can see that the counts are stored here.
So example 0 has 1 for A and 1 for B and 0 for C.
And example 1 has 0 for A, 1 for B, and 2 for C.
So that's recorded in the columns here.
You can, of course, undertake this step by hand.
But it's a kind of error-prone step.
And I'm just encouraging you to use DictVectorizer
to handle it all and essentially map you
from this, which is pretty human interpretable, into this,
which is something your models like to consume.
There's a second advantage here, which
is that if you use a DictVectorizer
and you need to now do something at test time,
you can easily use your vectorizer
to create feature spaces that are harmonized
with what you saw in training.
So as an example, if my test features
are another pair of examples with a different character,
then I can call transform on the original trained
vectorizer from up here.
And it will translate that list of features into a matrix.
Now, the important thing about what's happening here
is that it's going to package the test
features into the original training space
because, of course, those are the features
that your model recognizes.
Those are the features that you have weights
for if you've trained a model.
So it's important to call transform at this space.
And as an indication of one of the things that's
going to happen here is notice that in the test features,
my second example has a brand new feature D.
But D is not represented in the training space.
We have no weights for it.
It's simply not part of that original training data set.
And so the result is that when we call transform,
that feature is simply elided, which
is the desired behavior as we're translating
from training into testing.
And notice that the DictVectorizer has simply
handled that seamlessly for you, provided
that you remember at the second stage
not to call fit_transform.
That's the number one gotcha for this interface
is that if you call fit_transform a second time,
it will simply change the feature
space into the one that is represented in your test
features.
And then everything will fall apart.
And your model, as trained from before,
will be unable to consume these new matrices
that you've created.
But provided you remember that the rhythm is fit_transform
and then transform, this should be really a nice set
of interfaces.
And of course, this is what sst.experiment
is doing by default under the hood for you.