CHRISTOPHER POTTS: Welcome back, everyone.
This is part 8 in our series on supervised sentiment
analysis, the final screencast in the series.
We're going to be talking about recurrent neural network or RNA
classifiers.
I suppose this is officially our first step
into the world of deep learning for sentiment analysis.
This slide gives an overview of the model.
Let's work through it in some detail.
So we've got a single example with three tokens,
"the Rock rules."
These models are prepared for variable length sequences.
But this example happens to have like three.
And the first step to get this model started
is a familiar one.
We're going to look up each one of those tokens in what
is presumably a fixed embedding space here.
So for each token, we'll get a vector representation.
The next step is that we have some
learned parameters, a weight matrix, wxh.
And the subscript indicates that we're going from the inputs x
into the hidden layer h.
So that's a first transformation and that weight matrix
is used at each one of these time steps.
There is a second learned weight matrix
which I've called whh to indicate
that we are now traveling through the hidden layer.
And so we start at some initial state
h0, which could be an all 0 or a randomly initialized vector,
or a vector coming from some other component in the model.
And that representation is combined
with the representation that we derive going vertically
up from the embedding, usually in some additive fashion
to create this hidden state here h1.
And those parameters whh are used again
at each one of these time steps.
So that we have two learned weight matrices as part
of the core structure of this model,
the one that takes us from embeddings
into the hidden layer and the one
that travels us across the hidden layer.
And again, those are typically combined
in some additive fashion to create
these internal hidden representations.
Now we can do anything we want with those internal hidden
representations.
When we use RNNs as classifiers, we
do what is arguably the simplest thing, which
is take the final representation and use that as the input
to a standard softmax classifier.
So from the point of view of h3 going to y here,
we just have a learned weight matrix for the classifier,
maybe also a bias term.
But from this point here, this is really just a classifier
of the sort we've been studying up
until this point in the unit.
But of course, we could elaborate this model
in all sorts of ways.
It could run bidirectionally.
We could make more full use of the different hidden
representations here.
But in the simplest mode, our RNN classifiers
will just derive hidden representations
at each time step and use the final one
as the input to a classifier.
Couple of things I would say about this first,
if you would like.
A further layer of detail on how these models are structured
and optimized, I encourage you to look at this pure NumPy
reference implementation of an RNN classifier
that is included in our course code distribution.
I think that's a great way to get
a feel for the recursive process of computing
through full sequences and then having the error signals back
propagate through to update the weight matrix.
But for now, I think just understanding
the core structure of this model is sufficient.
I just want to remind you from the previous screencast,
that we're very close to the idea of distributed
representations of features that I introduced before.
Recall that for this mode, what we do
is look up each token in an embedding space,
just as we do for the RNN.
But instead of learning some complicated combination
function with a bunch of learned parameters,
we simply combine them via sum or average.
And that's the basis.
That's the input to the classifier here.
The RNN can be considered an elaboration of that because
instead of assuming that these vectors here
will be combined in some simple way like sum or mean,
we now have really vast capacity to learn
a much more complicated way of combining them
that is optimal with respect to the classifier
that we're trying to fit.
But fundamentally, these are very similar ideas.
And if it happened that sum or mean as in this picture
was exactly the right function to learn for your data,
then the RNN would certainly have the capacity to do that.
We just tend to favor the RNN because it can learn,
of course, a much wider range of complicated custom functions
that are particular to the problem that you've posed.
Now so far, we've been operating in a mode which
I've called standard RNN data set preparation.
Let's linger over that in a little bit of detail.
Suppose that we have two examples containing
the tokens a, b, a and b, c.
Those are our two raw inputs.
The first step in the standard mode
is to look up each one of those in some list of indices.
And then those indices are keyed into an embedding space.
And those finally give us the vector representations
of each examples.
So that really and truly, the inputs of the RNN
is a list of vectors.
It's just that we typically obtain those vectors
by looking them up in a fixed embedding space.
And so for example, since a occurs twice
in this first example, it is literally
repeated as the first and third vectors here.
Now I think you can see latent in this picture
the possibility that we might drop
the embedding space and instead just directly
input lists of vectors.
And that is one way that we will explore
later on in the quarter of using contextual models like BERT.
We would simply look up entire token streams
and get back lists of vectors and use those as fixed inputs
to a model like an RNN.
And that's a first step toward fine-tuning models like BERT
on problems like the ones we've posed in this unit.
So have that idea in mind as we talk next
about fine-tuning strategies.
Now another practical note.
What I've shown you so far is what you would
call a simple vanilla RNN.
LSTMs, Long Short Term Memory networks
are much more powerful models.
And we'll kind of default to them when we do experiments.
The fundamental issue is that plain RNNs
tend to perform poorly with very long sequences.
You get that error signal from the classifier
there at the final token.
But now information has to flow all the way back down
through the network.
Could be a very long sequence.
And the result is that the information
coming from that error signal is often lost or distorted.
Now LSTM cells are a prominent response to this problem.
They introduce mechanisms that control the flow of information
and help you avoid the problems of optimization
that arise for regular RNNs.
Now I'm not going to take the time here
to review this mechanism in detail.
I would instead recommend these two excellent blog
posts that have great diagrams and really
detailed discussions.
They can do a much better job than I
can at really conveying the intuitions visually and also
with math.
And I think you could pick one or both
and really pretty quickly gain a deep understanding of precisely
how LSTM cells are functioning.
The final thing here is just a code
snippet to show you how easy it is
to use our course code repository
to fit models like this.
In the context of sentiment analysis,
you can again make use of this sst library.
And what I've done here is a kind
of complicated version showing you
a bunch of different features.
So in cell 2, you can see that I'm going
to have a pointer to GloVe.
And I'm going to create a GloVe lookup using
the 50 dimensional vectors just to keep things simple.
The feature function for this model
is not one that returns count dictionaries.
It's important for the structure of the model
we're going to use that you input raw sequences of tokens.
So all we're doing here is down casing the sequence
and then splitting on whitespace.
Of course, you could do something more sophisticated.
The idea, though, is that you want to align with the GloVe
vocabulary.
Our model wrapper is doing a few things.
It's creating a vocabulary and loading it and embedding
using this GloVe space.
That'll be the initial embedding for our model.
And if you leave this step out, you'll
have a randomly initialized embedding space
which might be fine as well.
But presumably GloVe will give us a step up.
And then we set up the Torch RNN classifier.
And what I've done here is expose
a lot of the different keyword arguments, not all of them.
There are lots of knobs that you can fiddle with,
as is typical for deep learning models.
Maybe the one I would call out is
that we are using that fixed embedding
that we got from GloVe.
And I have set early_stopping equals true,
which might help you efficiently optimize these models.
Otherwise, you'll have to figure out
how many iterations you actually want it to run for.
And you might run it for much too long or much less time
than is needed to get an optimal model.
The early stopping options, and there
are a few other parameters involved in that,
might help you optimize these models
efficiently and effectively.
In the end though, having set up all that stuff,
you call fit as usual and return the trained model.
And in that context, you can simply use sst.experiment
with these previous components to conduct experiments
with RNNs, just as you did for simpler linear models
as in previous screencasts.
The one change which will be familiar
from the previous screencast, is that you need
to set vectorize equals false.
And that is important because, again, we're
going to let the model process these examples.
We don't want to pipe everything through some kind
of DictVectorizer.
That's strictly for handbuilt feature functions
and sparse linear models.
Here in the land of deep learning,
vectorize equals false.
And we'll use the components of the model
to represent each example as I discussed before.