CHRISTOPHER POTTS: Welcome, everyone.
This is part two in our series on grounded language
understanding.
Our task for this unit is essentially a natural language
generation task.
And I've called those speakers, the idea
is that speakers go from the world, that
is some non-linguistic thing that they're
trying to communicate about, into language.
Those are really the central agents that will be explored.
To ground all this, we're going to have a simple task.
I'm going to start with the most basic version of the task
that we'll ultimately tackle in our assignment and bake-off.
And that is color reference.
So these are examples taken from a corpus that was originally
collected by Randall Munroe of XKCD fame,
and processed into an NLP task by McMahan and Stone 2015.
And it's a simple formulation in that
the state of the world we want to communicate about
is a color patch.
And the task is simply to produce descriptions
of those color patches.
Now I've given some examples here.
And you can see that they range from simple one word
descriptions, all the way up to things
that are kind of complicated both cognitively and
linguistically, and I think point to the idea
that even though this is a simple and constrained domain,
it's a pretty cognitively and linguistically interesting one.
So our speakers, at least our baseline speakers,
are standard versions of encoder decoder models.
We're going to have for this initial formulation a very
simple encoder.
The task of the encoder is simply
to take a color representation, which
is going to be a list of floats embedded in some embedding
space, and then learn some hidden representation for that
color.
And that's all that needs to happen.
So it's just one step.
The decoder is where the speaking part happens.
So the initial token produced by the decoder, by the speaker,
is always the start token which is looked up
in an embedding space.
And then we get our first decoder hidden state,
which is created from the color representation
of the initial hidden state in the sequence we're
going to build, together with an embedding.
And both of those have weight transformations.
And it's an additive combination of them
that delivers this value h1 here.
Then we use some softmax parameters
to make a prediction about the next token.
Here we've predicted "dark."
And we get our error signal by comparing that prediction
with the actual token that occurred in our training data.
In this case, it was the word "light."
So since we made a wrong prediction,
we're going to get a substantive error signal that will then,
we hope, update the weight parameters
throughout this model in a way that
leads them to produce better generations the next time.
In a little more detail, just as a reminder.
So we have an embedding for that start token,
and indeed for all tokens.
The hidden state is derived from the embedding via a weight
transformation, and the color representation, which
is state h0 in the recurrence that we're building.
And that too has a transformation
applied to it, to travel through the hidden layer.
That gives us the state H1.
And then we have softmax parameters
on top of that h1 that make a prediction.
The prediction that they make is a prediction
over the entire vocabulary.
And the probability of the actual token
gives us our error signals.
So the probability of "light" is the error signal
that we'll use here to update the model parameters.
And then we begin with the next timestep.
I've called this teacher forcing,
because in the standard mode, which is the teacher forcing
mode, even though we predicted 'dark' at timestep 1
we're going to have as our second token,
the token "light" which is the actual token in the underlying
training data.
And we'll proceed as though we did not make a mistake.
So again, we do an embedding look up,
we get our second hidden state for the decoder, that's
a combination of the embedding x 37
and the previous student state, and we make another prediction.
And in this case, our prediction is "blue"
and that's the actual token.
And life is good for a little bit.
And then we proceed with a third timestep.
The actual token is "blue" h3.
We predict "green."
And in this case, we should have predicted
the stop token which would cause us to stop
processing the sequence.
We're just going to get an error signal as we steadily would
and propagate that back down through the model in hopes
that the next time, when we want to stop,
we'll actually produce this stop token that I've given up here.
At prediction time, of course the sequence is not given.
That doesn't change the encoder because the color
representation is part of the model inputs.
So then we have to decode and just
describe without any feedback.
So we proceed as we did before and we predict "dark" here.
And then "dark" has to become the token at the next timestep
because we don't know what the ground truth is.
And we proceed as before and say "blue."
And then that becomes the third time step,
and with luck, there in that third position,
we predict the stop token.
And the decoding process is completed.
That is the fundamental model.
Even though it's simple, it admits
of many interesting modifications.
Let me just mention a few of them.
First, the encoder and the decoder, of course,
could have many more hidden layers.
Mine just had one.
But they could be very deep networks.
We would expect that the layer counts
for the encoder and the decoder match
so that you have this even handoff from encoder to decoder
across all the hidden layers.
But even that's not a hard constraint.
I can imagine that some pooling or copying could
accommodate different numbers of layers and these two
components.
It's very common at present for researchers
to tie the embedding and classifier parameters, right?
The embedding gives us a representation
for every vocabulary item.
And the transpose of that can serve
as the set of parameters for a softmax classifier
when we predict tokens.
And tying those weights seems to be very productive in terms
of optimization effectiveness.
So you might consider that.
And finally, during training, we might
drop that teacher forcing assumption,
which would mean that in a small percentage of cases,
we would allow the model to just proceed as though its predicted
token was the correct token for the next time step,
even if that was a faulty assumption, on the idea
that that might help the model explore a wider
range of the space and inject generations
with some helpful diversity.
And then there's one other modification
that I want to mention, because you'll
see this as part of the homework and the system
that you're developing.
So we found that in Monroe et al.
2016, it was helpful to remind the decoder
at each one of its timesteps about what
it was trying to describe.
So in more detail, we had HSV color representations
as our inputs.
We did a Fourier transform to get an embedding.
And that was processed into a hidden state.
And then during decoding, we appended
to each one of the embeddings the Fourier transformation
representation of the color as a kind of informal reminder
at each timestep about what the input was actually like,
on the assumption that for long sequences where we get all
the way down to the end, the model might
have a hazy memory of what it's trying to describe.
And this functions as a kind of reminder at that point.
And that proved to be very effective,
and I'll encourage you to explore that in the homework.
And then I hope you can see that even though this task
formulation is simple, it's an instance of a wide range
of tasks that we might explore under the heading of grounding.
After all, for grounding in this sense,
we just need some non-linguistic representation coming in,
and the ideas that will generate language in response
to that input.
So image captioning is an instance of this.
Scene description, of course, is another instance.
Visual question answering is a slight modification
where the input is not just an image,
but also a question text and the idea
is that you want to produce an answer to that question
relative to the image input.
And then instruction giving would be a more general form
where the input is some kind of state description
and the idea is that we want to offer a complicated instruction
on that basis.
And I think we can think of many others that
would fit into this mold and benefit
not only from the encoder decoder architecture,
but also from conceptualization explicitly
as grounded natural language generation tasks.