Hi, everybody.
Welcome to Stanford's CS224N, also known
as Ling284, Natural Language Processing with Deep Learning.
I'm Christopher Manning, and I'm the main instructor
for this class.
So what we hope to do today is to dive right in.
So I'm going to spend about 10 minutes talking
about the course, and then we're going
to get straight into content for reasons
I'll explain in a minute.
So we'll talk about human language and word meaning,
I'll then introduce the ideas of the word2vec algorithm
for learning word meaning.
And then going from there we'll kind of concretely
work through how you can work out objective function
gradients with respect to the word2vec algorithm,
and say a teeny bit about how optimization works.
And then right at the end of the class
I then want to spend a little bit of time
giving you a sense of how these word vectors work,
and what you can do with them.
So really the key learning for today
is, I want to give you a sense of how amazing deep learning
word vectors are.
So we have this really surprising result
that word meaning can be represented,
not perfectly but really rather well
by a large vector of real numbers.
And that's sort of in a way, a commonplace of the last decade
of deep learning, but it flies in the face of thousands
of years of tradition.
And it's really rather an unexpected result
to start focusing on.
OK, so quickly what do we hope to teach in this course?
So we've got three primary goals.
The first is to teach you the foundation,
say a good deep understanding of the effect of modern methods
for deep learning applied to NLP.
So we are going to start with and go through the basics,
and then go on to key methods that
are used in NLP, recurrent networks, attention
transformers, and things like that.
We want to do something more than just that.
We'd also like to give you some sense of a big picture
understanding of human languages and what
are the reasons for why they're actually
quite difficult to understand and produce
even though humans seem to do it easily.
Now obviously if you really want to learn
a lot about this topic, you should enroll in
and go and start doing some classes
in the linguistics department.
But nevertheless for a lot of you,
this is the only human language content
you'll see during your master's degree or whatever.
And so we do hope to spend a bit of time on that starting today.
And then finally, we want to give you
an understanding of an ability to build systems in PyTorch,
that's some of the major problems in NLP.
So we'll look at learning word meanings, dependency parsing,
machine translation, question answering.
Let's dive in to human language.
Once upon a time, I had a lot longer introduction that
gave lots of examples about how human languages can
be misunderstood and complex, I'll
show a few of those examples in later lectures.
But since right for today, we're going
to be focused on word meaning.
I thought I'd just give one example, which comes
from a very nice xkcd cartoon.
And that isn't sort of about some
of the syntactic ambiguities of sentences,
but instead it's really emphasizing the important point
that language is a social system, constructed
and interpreted by people.
And that's part of how--
and it changes as people decide to adapt its construction,
and that's part of the reason why
human languages, great as an adaptive system
for human beings but difficult as a system for our computers
to understand to this day.
So in this conversation between the two women, one says,
anyway I could care less.
And the other says, I think you mean you couldn't care less,
saying you could care less implies
you care at least some amount.
And the other one says, I don't know
where these unbelievably complicated brains drifting
through a void trying in vain to connect with one another,
by plainly fleeing words out into the darkness.
Every choice of phrasing, spelling, and tone, and timing
carries countless sync signals and contexts and subtext,
and more and every listener interprets
those signals in their own way.
Language isn't a formal system, language is glorious chaos.
You can never know for sure what any words will mean to anyone,
all you can do is try to get better at guessing
how your words affect people.
So you can have a chance of finding the ones that
will make them feel something like what
you want them to feel.
Everything else is pointless.
I assume you're giving me tips on how you interpret words
because you want me to feel less alone.
If so, then thank you, that means a lot.
But if you're just running my sentences
to pass some mental checklist so you
can show off how well you know it, then I could care less.
OK, so that's ultimately what our goal
is, how to do a better job at building computational systems
that try to get better at guessing how their words will
affect other people and what other people are meaning
by the words that they choose to say.
So an interesting thing about human language
is, it is a system that was constructed by human beings.
And it's a system that was constructed relatively recently
in some sense.
So in discussions of artificial intelligence,
a lot of the time people focus a lot on human brains
and the neurons passing by, and this intelligence
that's meant to be inside people's heads.
But I just wanted to focus for a moment on the role of language,
there's actually--
this is kind of controversial but it's not
necessarily the case that humans are much more
intelligent than some of the higher apes like chimpanzees
or bonobos.
So chimpanzees and bonobos have been
shown to be able to use tools, to make plans,
and in fact chimps have much better short term memory
than human beings do.
So relative to that, if you look through the history of life
on Earth, human beings develop language really recently.
How recently, we kind of actually
don't know because there's no fossils that say,
OK here's a language speaker.
But most people estimate that language arose for human beings
sort of somewhere in the range of 100,000
to a million years ago.
OK, that's the way I let go but compared
to the process of the evolution of life on Earth,
that's kind of blinking an eyelid.
But that powerful communication between human beings
quickly set off our ascendancy over other creatures.
So it's kind of interesting that the ultimate power turned out
not to have been poisonous fangs or being super
fast or super big, that having the ability to communicate
with other members of your tribe.
It was much more recently again that humans developed writing,
which allowed knowledge to be communicated across distances
of time and space.
And so that's only about 5,000 years old,
the power of writing.
So in just a few thousand years the ability
to preserve and share knowledge took us from the Bronze Age
to the smartphones and tablets of today.
So a key question for artificial intelligence and human-computer
interaction is how to get computers
to be able to understand the information
conveyed in human languages.
Simultaneously, artificial intelligence
requires computers with the knowledge of people.
Fortunately now, AI systems might
be able to benefit from a virtuous cycle.
We need knowledge to understand language and people well,
but it's also the case that a lot of that knowledge
is contained in language spread out across the books and web
pages of the world.
And that's one of the things we're
going to look at in this course is, how
that we can sort of build on that virtuous cycle.
A lot of progress has already been made
and I just want to very quickly give a sense of that.
So in the last decade or so and especially
in the last few years, with newer methods of machine
translation we're now in a space where machine translation
really works moderately well.
So again, from the history of the world,
this is just amazing, right for thousands of years
learning other people's languages was
a human task which required a lot of effort
and concentration.
But now we're in a world where you could just
hop on your web browser and think,
oh, I wonder what the news is in Kenya today,
and you can head off over to a Kenyan website
and you can see something like this.
And you can go, huh, and you can then
ask Google to translate it for you from Swahili,
and the translation isn't quite perfect
but it's reasonably good.
So the newspaper, Tuko, has been informed that local government
minister links on [NON-ENGLISH SPEECH] and his
transport counterparts [INAUDIBLE] died within two
separate hours.
So within two separate hours, this is kind of awkward,
but essentially we're doing pretty well
at getting the information out of this page,
and so that's quite amazing.
The single biggest development in NLP
for the last year certainly in the popular media
was GPT-3, which was a huge new model that
was released by OpenAI.
What GPT-3 is about and why it's great is actually a bit subtle,
and so I can't really go through all the details of this here
but it's exciting because it seems
like it's the first step on the path to what we might call
universal models, where you can train up
one extremely large model on something like,
that library picture I showed before,
and it just has knowledge of the world
knowledge of human languages knowledge, of how to do tasks.
And then you can apply it to do all sorts of things.
So no longer are we building a model
to detect spam and then a model to detect pornography,
and then a model to detect whatever foreign language
content, and just building all these separate supervised
classifiers for every different task,
we've now just built up a model that understands.
So exactly what it does is it just predicts following words.
So on the left it's been told to write
about Elon Musk in the style of Doctor Seuss,
and it started off with some text
and then it's generating more text.
And the way it generates more text is literally by just
predicting one word at a time, following
words to complete its text.
But this has a very powerful facility
because what you can do with GPT-3
is you can give a couple of examples
of what you'd like it to do.
So I can give it some text and say, I broke the window,
change it into a question, what did I break?
I gracefully saved the day I changed it into a question.
What did I gracefully save?
So this prompt tells GPT-3 what I'm wanting it to do.
And so then if I give it another statement like,
I gave John flowers.
I can then say three, predict what words come next.
And it'll follow my prompt and produce
who did I give flowers to?
Or I can say I gave her a rose and a guitar,
and it will follow the idea of the pattern
and to who did I give a rose and a guitar to?
And actually this one model can then
do an amazing range of things, including
many that's [AUDIO OUT] at all, to give
just one example of that.
Another thing that you can do is get
it to translate human language sentences into SQL.
So this can make it much easier to do CS145.
So having given that a couple of examples of SQL translation
of human language text which I'm in this time not showing
because it won't fit on my slide,
I can then give it a sentence like how many users
have signed up since the start of 2020,
and it turns it into SQL.
Or I can give it another query what
is the average number of influences each user
subscribed to, and again it then converts that into SQL.
So GPT-3 knows a lot about the meaning of language
and the meaning of other things like SQL,
and can fluently manipulate it.
OK, so that leads us trying to this [INAUDIBLE] meaning,
and how do we represent the meaning of a word?
Well, what is meaning?
Well, we can look up something like the Webster's dictionary
and say, OK, the idea that is represented by a word, the idea
that a person wants to express by using
words signs, et cetera.
That Webster's dictionary definition
is really focused on the word idea somehow,
but this is pretty close to the commonest way
that linguists think about meaning.
So do they think of word meaning as being
a pairing between a word which is a signifier
or symbol, and the thing that it signifies,
the signified thing which is an idea or thing.
So that the meaning of the word chair is the set of things
that are chairs.
And that's referred to as denotational semantics.
A term that's also used and similarly applied
for the semantics of programming languages.
This model isn't very deeply implementable,
like how do I go from the idea that chair means
the set of chairs in the world to something
I can manipulate meaning with my computers.
So traditionally the way that meaning has normally
been handled in natural language processing systems
is to make use of resources like dictionary,
and thesaurus in particular.
Popular ones, WordNet, which organized words and terms
into both synonyms sets of words that can mean the same thing,
and hypernyms which correspond to ISA relationships.
And so for the ISA relationships we
can kind of look at the hypernyms of panda,
and panda as a kind of procyonidae,
whatever those are, probably with red pandas, which
is a kind of carnivore, which is a kind of placental, which
is kind a mammal.
And you sort of head up this hypernyms hierarchy.
So word that has been a great resource for NLP, that's
also been highly deficient.
So it lacks a lot of nuance, so for example in WordNet,
proficient is listed as a synonym for good.
But maybe that's sometimes true but it
seems like in a lot of contexts it's not true.
And you mean something rather different
when you say proficient versus good,
it's limited as a human constructed thesaurus.
So in particular there's lots of words and lots
of uses of words that just aren't there,
including anything that is more current terminology.
Like wicked is there for the wicked witch,
but not for more modern colloquial uses.
Ninja's certainly isn't there for the kind of description
some people make of programmers, and it's
impossible to keep up to date.
So it requires a lot of human labor,
but even when you have that, it has a set of synonyms
but doesn't really have a good sense of words
that means something similar.
So fantastic and great means something
similar without really being synonyms.
And so this idea of meaning similarity
is something that would be really useful to make progress
on, and where deep learning models excel.
OK, so what's the problem with a lot of traditional NLP?
Well the problem with a lot of traditional NLP
is that words are regarded as discrete symbols.
So we have symbols like hotel, conference,
motel are words, which in deep learning
speak we refer to as a localized representation.
And that's because if you are in a statistical machine learning
systems, want to represent these symbols that each of them
is a separate thing so the standard way of representing
them and this is what you do in something
like a statistical model if you're building
a logistic regression model with words as features
is that you represent them as one word vector.
So you have a dimension for each different word.
So maybe like my example, here are my representations
as vectors for motel and hotel.
And so that means that we have to have
huge vectors corresponding to the number of words
in our vocabulary.
So the kind of if you had a high school English dictionary it
probably have about 250,000 words in it, but there
are many, many more words in the language really.
So maybe we at least want to have a 500,000 dimensional
vector to be able to cope with that.
OK, but the bigger the even bigger problem
with discrete symbols is that we don't have this notion of word
relationships and similarity.
So for example in web search, if a user
searches for Seattle motel, we'd also
like to match on documents containing Seattle hotel.
But our problem is we've got this one word vectors
for the different words.
And so in a formal mathematical sense,
these two vectors are orthogonal,
that there's no natural notion of similarity between them
whatsoever.
Well, there are some things that we could do but try and do
about that, and people did do about that in before 2010.
We could say, hey we could use the WordNet synonyms
and we count things that lists of synonyms are similar anyway.
Or hey, maybe we could somehow build up
our representations of words that have meaning,
overlap and people did all of those things.
But they tended to fail badly from incompleteness,
so instead what I want to introduce
today is the modern deep learning method of doing that,
where we encode similarity in real value vector themselves.
So how do we go about doing that?
And the way we do that is by exploiting this idea
called distributional semantics.
So the idea of distributional semantics is again something
that when you first see it, maybe feels a little bit
crazy because rather than having something
like denotational semantics, what we're now going to do
is say that a word's meaning is going
to be given by the words that frequently appear close to it.
J. R Firth was a British linguist
from the middle of last century, and one of his pithy slogans
that everyone quotes at this moment
is, you shall know a word by the company it keeps.
And so this idea that you can represent a sense for words,
meaning as a notion of what context that appears in
has been a very successful idea.
One of the most successful ideas that's
used throughout statistical and deep learning NLP.
It's actually an interesting idea more philosophically,
so that there are kind of interesting connections.
For example in Wittgenstein's later writings,
he became enamored of a used theory of meaning.
And this is a sin in some sense, a used theory
of meaning but whether it's the ultimate theory of semantics,
it's actually still pretty controversial.
But it proves to be an extremely computational sense
of semantics, which has just led to it
being used everywhere very successfully in deep learning
systems.
So when a word appears in a text,
it has a context which are a set of words
that appear in the eye.
And so for a particular word, my example here is banking.
We'll find a bunch of places where banking occurs in text,
and will collect the sort of nearby words
that context words and we'll see and say
that those words that are appearing
in that kind of muddy brown color
around banking, that those contexts words
will in some sense represent the meaning of the word banking.
While I'm here let me just mention one distinction
that will come up regularly.
When we're talking about a word in our natural language
processing class, we sort of have two senses of word
which are referred to as types and tokens.
So there's a particular instance for words.
So there's in the first example, government debt problems
turning into banking crises.
There's banking there, and that's
a token of the word banking.
But then I've collected a bunch of instances of quote
unquote, the word banking.
And when I say the word banking and a bunch of examples of it,
I'm then treating banking as a type which refers to the uses
and meaning the word banking has across instances.
OK, so what are we going to do with this distributional models
of language?
Well, what we want to do is based
on looking at the words that occur in context as vectors
that we want to build up dense real valued
vector for each word, that in some sense
represents the meaning of that word.
And the way it will represent the meaning of that word
is that this vector will be useful for predicting
other words that occur in the context.
So in this example, to keep it manageable on the slide vectors
are only eight dimensional.
But in reality we use considerably bigger vectors,
so a very common size is actually
300 dimensional vectors.
OK, so for each word that's a word type,
we don't have a word vector.
These are also used with other names,
they refer to as neural word representations
or for a reason, they'll become clearer on the next slide,
they're referred to as word embeddings.
So these are now a distributed representation, not a localized
representation because the meaning of the word banking
is spread over all 300 dimensions of the vector.
These are called word embeddings because effectively when
we have a whole bunch of words, these representations place
them all in a high dimensional vector space,
and so they're embedded into that space.
Now unfortunately human beings are
very bad at looking at 300 dimensional vector
spaces or even eight dimensional vector spaces,
so the only thing that I can really display to you
here is a two dimensional projection of that space.
Now even that's useful, but it's also
important to realize that when you're
making it two dimensional projection of a three
dimensional space, you're losing almost all
the information in that space.
And a lot of things will be crushed together,
that don't actually deserve to be better.
So here's my word embeddings, of course you
can't see any of those at all.
But if I zoom in and then I zoom in further,
what you'll already see is that the representations
we've learnt distributionally to just a good job at grouping
together similar words.
So in this sort of overall picture,
I can zoom into one part of the space,
is actually the part that's up here and this view of it.
And it's got words for countries,
so not only are countries generally grouped together,
even the sort of particular sub groupings of countries
make a certain amount of sense.
And down here we then have nationality words,
if we go to another part of the space we can
see different kind of words.
So here are verbs and we have ones like come and go,
a very similar saying and thinking words,
say think expect, they kind of similar and nearby.
Over on the bottom right, we have sort of verbal exhilarates
and corpus, so have, had, has, forms of the verb to be.
And certain content for verbs are similar to corpus verbs,
because they describe states.
He remained angry, he became angry,
and so they're actually then grouped close together
to the word, the verb to be.
So there's a lot of interesting structure
in this space that then represents
the meaning of words.
So the algorithm I'm going to introduce now
is one that's called word2vec, which was introduced
by Tomas Mikolov and colleagues in 2013
as a framework for learning word vectors,
and it's sort of a simple and easy
to understand place to start.
So the idea is we have a lot of text
from somewhere, which we commonly refer to as a corpus.
Of text corpus as just the Latin word for body,
so it's a body of text.
And so then we choose a fixed vocabulary
which will typically be large but nevertheless truncated,
so we get rid of some of the really rare words,
so we might say vocabulary size of 400,000
and we then create for ourselves vector for each word.
OK, so then what we do is we want
to work out what's a good vector for each word,
and the really interesting thing is
that we can learn these word vectors from just a big pile
of text by doing this distributional similarity
task of being able to predict, well, what words occur
in the context of other words.
So in particular, we're going [AUDIO OUT]
through in the texts, and so at any moment
we have a center word C, and context words outside of it,
which we'll call O. And then based on the current word
vectors, we're going to calculate
the probability of a context word occurring,
given the center word according to our current model.
But then we know that certain words did actually
occur in the context of that center word,
and so what we want to do is then
keep adjusting the word vectors to maximize
the probability that's assigned to words that actually occur
in the context of the center word
as we proceed through these texts.
So to start to make that a bit more concrete,
this is what we're doing.
So we have a piece of text, we choose our center word which
is here in two and then we say, well,
if a model of predicting the probability of context words
given the center word and this model,
we'll come to in a minute, but it's defined
in terms of our word vectors.
So let's see what probability it gives
to the words that actually occurred
in the context of this word.
It gives them some probability, but maybe it'd
be nice if the probability it assigned was higher.
So then how can we change our word vectors
to raise those probabilities?
And so we'll do some calculations
with into being the center word, and then
we'll just go on to the next word
and then we'll do the same kind of calculations,
and keep on chunking.
So the big question then is, well
what are we doing for working out
the probability of a word occurring
in the context of the center word?
And so that's the central part of what we
develop as the word2vec object.
So this is the overall model that we want to use.
So for each position in our corpus, our body of text,
we want to predict context words within a window of fixed size
M, given the center word WJ.
And we want to become good at doing that,
so we want to give high probability to words
that occur in the context.
And so what we're going to do is we're
going to work out what's formerly
the data likelihood as to how good a job
we do at predicting words in the context of other words.
And so formally that likelihood is
going to be defined in terms of our word vectors.
So they're the parameters of our model,
and it's going to be calculated as taking the product of using
each word as the center word, and then
the product of each word and a window
around that of the probability of predicting that context
word in the center word.
And so to learn this model, we're
going to have an objective function, sometimes also
called a cost or a loss that we want to optimize.
And essentially what we want to do
is we want to maximize the likelihood of the context we
see around center words.
But following standard practice, we slightly
fiddle that because rather than dealing with products,
it's easier to deal with sums and so we
work with log likelihood.
And once we take log likelihood, all of our products
turn into sums.
We also work with the average log likelihood,
so we've got one on, t term here for the number of words
in the corpus, and finally for no particular reason
we like to minimize our objective function rather
than maximizing it.
So we stick a minus sign in there,
and so then by minimizing this objective function, J of theta,
that becomes maximizing our predictive accuracy.
OK, so that's the set up, but we still
haven't made any progress in how do we
calculate the probability of a word occurring in the context,
given the center word.
And so the way we're actually going
to do that is we have vector representations for each word,
and we're going to work out the probability simply
in terms of the word vectors.
Now at this point there's a little technical point,
we're actually going to give to each word, two word vectors.
One word vector for when it's used as the center
word, and a different word vector
when that's used as a context word.
This is done because it just simplifies
the math and the optimization, that seems a little bit ugly
that actually makes building word vectors a lot easier,
and really we can come back to that and discuss it later.
But that's what it is.
And so then once we have these word vectors,
the equation that we're going to use
for giving the probability of a context word appearing given
the center word is that we're going
to calculate it using the expression, the middle bottom
of my slide.
So let's sort of pull that apart just a little bit more.
So what we have here with this expression is
so for a particular center word and a particular context word
O, we're going to look up the vector
representation of each word.
So there U of O and V of C, and so then we're
simply going to take the dot product of those two vectors.
So dot product is a natural measure
for similarity between words because
in any particular mention of opposite,
you'll get a component that adds to that dot product sum.
If both are negative, it'll add a lot to the dot product sum.
If one's positive and one's negative,
it'll subtract from the similarity measure.
Both of them are zero, won't change the similarity.
So it sort of seems sort of plausible idea
to just take a dot product and thinking, well,
if two words have a larger dot product, that
means they're more similar.
And so then after that, we sort of really doing
nothing more than OK, we want to use dot products
to represent words similarity.
And now let's do the dumbest thing that we know,
how to turn this into a probability distribution.
Well, what do we do?
Well firstly, well taking a dot product
of two vectors that might come out
as positive or negative, but well we
want to have probabilities.
We can't have negative probabilities.
So a simple way to avoid negative probabilities
is to exponentiate them, because then we
know everything is positive.
And so then we are always getting a positive number
in the numerator, but for probabilities we
also want to have the numbers add up to 1.
So we have a probability distribution.
So we just normalizing the obvious way,
where we divide through by the sum of the numerator quantity
for each different word in the vocabulary,
and so then necessarily that gives us
a probability distribution.
So all the rest of that I was just
talking through what we're using there
is what's called the softmax function.
So the softmax function will take any R in vector
and turn it into things between 0 to 1.
And so we can take numbers and put them through the softmax,
and turn them into the redistribution.
So the name comes from the fact that it's sort of like a max.
So because of the fact that we exponentiate,
that really emphasizes the big contents
in the different dimensions of calculating similarity.
So most of the probability goes to the most similar things,
and it's called soft because well, it
doesn't do that absolutely.
It'll still give some probability
to everything that's in the slightest bit similar--
I mean, on the other hand it's a slightly weird name because max
normally takes a set of things and just returns
one the biggest of them, whereas the softmax is
taking a set of numbers and scaling them,
that is returning the whole probability distribution.
OK, so now we have all the pieces of our model.
And so how do we make our word vectors?
Well, the idea of what we want to do is we
want to fiddle our word vectors in such a way that we minimize
our loss, i.e. that we maximize the probability of the words
we actually saw in the context of the center word.
And so the theta represents all of our model parameters in one
very long vector.
So for our model here, the only parameters
are our word vectors.
So we have for each word two vectors,
its context vector and its center vector.
And each of those is a D dimensional vector,
where D might be 300 and we have V many words.
So we end up with this big huge vector which is 2DV long,
which if you have a 500,000 vocab times the 300 dimensional
at the time, it's more math than I can do in my head,
but it's got millions of millions of parameters,
it's got millions and millions of parameters.
And we somehow want to fiddle them all
to maximize the prediction of context words.
And so the way we kind of do that then is we use calculus.
So if what we want to do is take that math that we've
seen previously and say, well with this objective function,
we can work out derivatives and so
we can work out where the gradient is.
So how we can walk downhill to minimize loss.
So we're at some point and we can figure out
what is downhill, and we can then progressively
walk downhill and improve our model.
And so what our job is going to be
is to compute all of those vector gradients.
OK, so at this point I then want to kind of show
a little bit more as to how we can actually do that.
And then a couple more slides here
but maybe I'll just try and jigger things again and move
to my interactive whiteboard.
What we wanted to do, so we had our overall--
we had our overall J theta that we
were wanting to minimize our average neg log likelihood,
so that was the minus 1 on T of the sum of T equals 1 to big T,
which was our text length.
And then we were going through the words in each context,
so we were doing J between M words
on each side except itself.
And then what we wanted to do was in the side there,
we were then working out the log probability of the context word
at that position given the word that's in the center position T
.
And so then we converted that into our word vectors
by saying that the probability of O given C
is going to be expressed as the noun, this softmax of the dot
product.
And so now what we want to do is work out
the gradient, the direction of downhill for this [INAUDIBLE]..
And so the way we're doing that is
we're working out the partial derivative of this expression
with respect to every parameter in the model,
and all the parameters in the model
are the components, the dimensions of the word
vectors of every word.
And so we have the center word vectors and the outside word
vectors.
So here I'm just going to do the center word vectors
but on homework-- on a future homework assignment two,
the outside word vectors will show up
and they're kind of similar.
So what we're doing is we're working out
the partial derivative with respect
to our center word vector, which is maybe a 300 dimensional word
vector of this probability of O give C.
And since we're using log probabilities
of the log of this probability of O given C of this exp
of your U of OTVC, over--
my writing will get worse and worse, sorry.
I've already made a mistake, haven't I?
Sum, the sum of W equals 1 to the vocabulary
of the exp of UWTVC.
OK, well at this point things start off pretty easy.
So what we have here is something
that's log of A over B, so that's easy.
We can turn this into log A minus log B.
But before I go further I'll just
make a comment at this point.
So at this point my audience divides on in two,
there are some people in the audience
for which maybe a lot of people in the audience--
this really elementary math, I've seen this a million times
before and he isn't even explaining it very well.
And if you're in that group, well
feel free to look at your email or the newspaper
or whatever else is best suited to you,
but I think there are also other people in the class who
are, the last time I saw calculus
was when I was in high school, for which that's not the case.
And so I wanted to spend a few minutes going through this
a bit concretely so that to try and get over the idea
that even though most of deep learning and even word vector
learning seems like magic, that it's not really magic.
It's really just doing math and one of the things
we hope is that you do actually understand
this math that's being done.
So I'll keep along and do a bit more of it,
so then what we have it's use this way of writing the log.
And so then we can say that that expression above
equals the partial derivatives of VC
of the log of the numerator log XUOTVC
minus the partial derivative of the log of the denominator.
So that's then the sum of W equals 1 to V
of the X of UWTVC.
So at that point, I have my numerator here
and my former denominator there.
So at that point there is that's the first part is the numerator
part, so the numerator part is really, really easy,
so we have here that log on X just inverses of each other.
So they'd just go away.
So that becomes the derivative with respect
to VC of just what's left behind,
which is U0 dot product with VC.
OK, and so the thing to be aware of
is we're still doing this multivariate calculus.
So what we have here is calculus with respect to a vector,
like hopefully you saw some of the math 51
or some other place, not high school.
Single variable calculus.
On the other hand to the extent, you
can't remember some of this stuff.
Most of the time you can justify it perfectly well
by thinking about what happens with one dimension at a time,
and it generalizes to multivariable calculus.
So if about all that you remember of calculus
is that d dx of ax equals a, really it's
the same thing that we're going to be using here.
That here we have the outside word dot producted with the VC.
Well at the end of the day, that's
going to have terms of sort of U0 component 1 times
the center word component 1 plus U0 component 2 plus.
There were component 2, and so we're sort
of using this bit over here.
And so what we're going to be getting out
is the U0, and U01, and the U02, so this
will be all that is left with respect to VC1.
When we take its derivative with respect to VC1
and this term will be the only thing
left when we take the derivative with respect
to the variable VC2.
So the end result of taking the vector antiderivative
of U0 dot producted with VC is simply going to be U0.
OK great, so that's progress.
So then at that point we go on and we say oh damn, we still
have the denominator [AUDIO OUT] slightly more complex, but not
so bad.
So then we want to take the partial derivatives
with respect to VC of the log of the denominator.
OK and so then at this point, the one tool
that we need to know and remember
is how to use the chain rule.
So the chain rule is when you're wanting
to work out having derivatives of compositions of function.
So we have F of G of whatever X, but here it's going to be VC.
And so we want to say, OK what we have here is
we're working out a composition of functions.
So here's our F and here is our X, which is G of VC.
Maybe I shouldn't call it X, maybe it's probably better
to call it Z or something.
OK, so when we then want to work out the chain rule,
well what do we do?
We take the derivative of F at the point Z.
And so at that point we have to actually remember something,
we have to remember that the derivative of the log
is the 1 on X function.
So this is going to be equal to the 1 on X for Z.
So that's then going to be 1 over the sum of W equals 1 to V
of exp of UTVC multiplied by the derivative
of the inner function.
So the derivative of the part that is remaining,
I'm getting this, the sum of--
and this one trick here at this point
we do want to have a change of index.
So we want to say the sum of X equals 1 to V
of exp of U of X VC.
Since we can get into trouble if we don't change that
variable to be using a different one.
OK, so at that point we're making some progress,
but we still want to work out the derivative of this.
And so what we want to do is apply the chain rule once more.
So now here is our F and in here is our new Z equals G of VC.
And so, we then sort of repeat over,
so we can move the derivative inside a sum always.
So we then taking the derivative of this,
and so then the derivative of exp is itself,
so we're going to just have exp of the UXTVC times
there's is a sum of X equals 1 to V times
the derivative of UX TVC.
OK, and so then this is what we've worked out before,
we can just rewrite as UX.
OK, so now we're making progress.
So if we start putting all of that together,
what we have is the derivative, well the partial derivatives
with VC of this log probability.
All right, we have the numerator,
which was just U0 minus--
we then had the sum of the numerator, sum over X
equals 1 to V of exp UXTVC times U of X, then
that was multiplied by our first term that came from the 1
on X, which gives you the sum of W equals 1 to V
of the exp of UWTVC.
And this, the fact that we changed the variables
became important.
And so by just sort of rewriting that a little,
we can get that equals U0 minus the sum V equals sorry--
X equals 1 to V of this X, V of XTVC over the sum of W
equals to V of exp UWTVC times U of X.
And so at that point it's sort of interesting thing
has happened that we've ended up getting straight back exactly
the softmax formula probability that we saw when we started.
And we can just rewrite that more conveniently
as saying this equals U0 minus the sum over X equals 1 to V
of the probability of X given C times UX.
And so what we have at that moment is
this thing here is an expectation.
And so this isn't an average over all the context
vectors waited by their probability
according to the model.
And so it's always the case with these softmax style models,
that what you get out for the derivatives
is you get the observed minus the expected.
So our model is good if our model on average
predicts exactly the word vector that we actually see.
And so we're going to try and adjust
the parameters of our model, so [INAUDIBLE]..
Now we try and make it do it as much as possible,
I mean of course as you'll find you can never get close.
If I just say to you OK the word is croissant,
which words are going to occur in the context of croissant?
I mean, you can't answer that, there are all
sorts of sentences that you could say that involve the word
croissant.
So actually our particular probability estimates
are going to be kind of small, but nevertheless we
want to sort of fiddle our word vectors to try and make
those estimates as high as we possibly can.
So I've gone on about this stuff a bit,
but haven't actually sort of shown you any of what
actually happened.
So I just want to quickly show you
a bit of that as to what actually
happens with word vectors.
So here's a simple little IPython notebook
which is also what you'll be using for assignment one only.
So in the first cell I import a bunch of stuff.
So we've got NumPy for our vectors.
[AUDIO OUT] learns kind of your machine learning,
Swiss Army Knife GENESIM is a package
that you may well not have seen before.
It's a package that's often used for word vectors,
it's not really used for deep learning.
So this is the only time you'll see it in the class,
but if you just want a good package
for working with word vectors and some other application
it's a good one to know about.
OK so then in my second cell here
I'm loading a particular set of word vectors.
So these are glove word vectors that we made at Stanford
in 2014, and I'm loading a hundred dimensional word
vectors so that things are a little bit quicker for me
while I'm doing things here, sort
of do this model of bread and croissant, what I've just
got here is word vectors.
So I just wanted to sort of show you
that there are word vectors.
Well maybe I should have loaded those word vectors in advance.
Let's see.
Oh, OK, I'm in business.
OK, so here are my word vectors for bread and croissant,
and well you can see that maybe these two words are
a bit similar, so both of them are
negative in the first dimension, positive in the second,
negative in the third, positive in the fourth,
negative in the fifth.
So it sort of looks like they might
have a fair bit of dot product which
is kind of what we want because bread and croissant are
kind of similar.
But what we can do is actually ask the model
and these are Jensen functions, now
know what are the most similar words
so I can ask for croissant, what are
the most similar words to that.
And it will tell me it's things like brioche, baguette,
focaccia.
So that's pretty good.
Pudding is perhaps a little bit more questionable.
We can say most similar to the USA
and it says Canada, America, USA with periods, United States
that's pretty good.
And most similar to banana--
take it out-- coconut, mangoes, bananas, sort of fairly
tropical fruit grade.
Before finishing though, I want to show you something slightly
more than just similarity, which is one of the amazing things
that people observed with these word vectors,
and that was to say you can actually sort of do
arithmetic in this vector space, that makes sense.
And so in particular, people suggested this analogy task.
And so the idea of the analogy task
is you should be able to start with a word like king,
and you should be able to subtract out a male component
from that, add back in a woman component,
and then you should be able to ask, well,
what word is over here?
And what should like is that the word over there is queen.
And so this sort of little bit of--
so we're going to do that, with this sort of same most
similar function which is actually more--
so as well as having positive words,
you can ask for most similar negative words and you might
wonder what's most negatively similar to a banana,
and you might be thinking, oh it's--
I don't know, some kind of meat or something.
Actually that by itself isn't very useful
because when you just ask for most negatively similar
to things, you tend to get crazy strings that
were found in the data set that you don't know
what they mean if anything.
But if we put the two together, we
can use the most similar function with positives
and negatives to do analogies.
So we're going to say we want a positive king,
we want to subtract out negatively man,
we want to then add in positively woman
and find out what's most similar to this point in the space.
So my analogy function does that, precisely that
by taking a couple of most similar ones
and then subtracting out the negative one.
And so we can try out this analogy function.
So I can do the analogy, I show in the picture with man
is the king as woman is--
sorry I'm not saying that's right.
Yeah, man is the king as woman is to--
well, sorry I haven't done my cells.
OK, man is to king as woman is to queen.
So that's great, and that works well.
I mean-- and you can do it the sort of other way around,
king is to man as queen is to woman.
If this only worked for that one freakish example,
you maybe wouldn't be very impressed
but you know it actually turns out like it's not perfect
but you can do all sorts of fun analogies with this,
and they actually work.
So I could ask for something like analogy, here's
a good one.
Australia is to beer as France is to want?
And you can think about what you think the answer to that one
should be, and it comes out as champagne,
which is pretty good.
Or I could ask for something like analogy pencil
is to sketching as camera is to what?
And it says photographing.
You can also do the analogies with people, at this point,
I have to point out that this data was
and the model was built in 2014, so you can't ask anything
about Donald Trump in there.
Well you can, Trump is in there but not as president,
but I could ask something like analogy a bomb is to Clinton
as Reagan is to what?
And you can think of what you think
is the right analogy there, the analogy it returns with Nixon.
So I guess that depends on what you think of Bill Clinton
as to whether you think that was a good analogy or not.
You can also do sort of linguistic analogies with it,
so you can do something like analogy tall is
to tallest as long is to what?
And it does longest.
So it really just sort of knows a lot about the meaning
behavior of words, and I think when these methods were first
developed and hopefully still for you, that people were just
gobsmacked about how well this actually worked
at capturing enough words.
And so these word vectors then went everywhere
as a new representation that was so powerful for working out
word meaning.
And so that's our starting point for this class,
and we'll say a bit more about them next time.
And they're also the basis of what you're looking
at for the first assignment.
Can I ask a quick question about the distinction between the two
vectors per word?
Yes.
So my understanding is that there can be several context
words per word in the vocabulary,
but then there's only two vectors
I thought the distinction between the two
is that one it's like the actual word and one
is the context word.
But in multiple contexts words, how do you pick just two then?
Well, so we're doing every one of them,
so like maybe I won't turn back on the screen share
but we are doing in the objective function there
was a sum over--
so you've got this big corpus of text right,
so you're taking a sum over every word which
is it appearing as the center word,
and then inside that there's a second sum which is
for each word in the context.
So you are going to count each word as a context word,
and so then for one particular term of that objective function
you've got a particular context word and a particular center
word that you're then sort of summing over different context
words for each center word, and then
you're summing over all of the decisions of different center
words.
And to say just a sentence more about having two vectors,
I mean in some sense it's an ugly detail that
was done to make things sort of simple and fast.
So if you look at the math carefully,
if you sort of treated the two vectors as the same,
so if you use the same vectors for center and context
and you say, OK, let's work out the derivatives,
things get uglier.
And the reason that they get uglier is it's
when I'm iterating over all the choices of context word,
oh my God sometimes the context word
is going to be the same as the center word,
and so that messes with working out my derivatives.
Whereas by taking them as separate vectors that
never happens, so it's easy.
But the kind of interesting thing
is saying that you have these two different representations
sort of just ends up really sort of doing no harm,
and my wave my hands argument for that
is since we're kind of moving through each position,
the corpus one by one something--
a word that is the center word at one moment
is going to be the context word at the next moment,
and the word that was the context word
is going to have become the center word.
So you're sort of doing the computation
both ways in each case.
And so you should be able to convince yourself
that the two representations for the word
end up being very similar, and they are not identical
both for technical reasons of the ends
of documents and things like that, but very, very similar.
And so effectively you tend to get
two very similar representations for each word,
and we just average them and collect the word vector.
And when we use word vectors we just
have one vector for each word.
That makes sense thank you.
I have a question purely out of curiosity.
So we don't know when we projected the vectors,
the word vectors onto to the 2D surface
we saw like little clusters of squares
are similar to each other, and then
later on, we saw that with the analogies thing,
we kind of see that there's these directional vectors that
sort of get like the rule of or the C of O or something
like that.
And so I'm wondering, is there--
are there relationships between those
relational vectors themselves such as,
like is the rule of vector sort of similar to the C of O
of vector which is very different from like--
makes a good sandwich with vector.
Is there any research on that?
That's a good question.
You've stumped me already in the first lecture.
Yeah, I can't actually think of a piece of research.
And so I'm not sure I have a confident answer,
I mean it seems like that's a really easy thing
to check with once you have one of these sets of word vectors
that it seems like for any relationship that
is represented well enough by a word.
You should be able to see if it comes out kind of similar.
I mean, I'm not sure, we can look and see.
That's totally OK, just curious.
I'm sorry, I missed the last little bit to your answer
to your first question.
So when you want to collapse two vectors for the same work,
did you usually take the average?
Different people have done different things.
But the most common practice is after you've--
there's still a bit more I have to cover about running word2vec
that we didn't really get through today.
So I still got a bit more work to do on Thursday,
but once you've run your word2vec algorithm
and you sort of your output is two vectors for each word
and kind of when it's center and when its context, and so
typically people just average those two vectors and say OK,
that's the representation of the word croissant,
and that's what appears in the sort of word vectors file,
like the one I loaded.
Makes sense, thank you.
I think-- so my question is, if a word to have
two different meanings or multiple different meanings,
can we still represent it as the same single vector?
Yes, that's a very good question.
And actually there is some content
on that in Thursday's lecture, so I can say more about that.
But yeah, the first reaction is you kind of should
be scared because something I've said nothing about at all
is most words, especially short common words
have lots of meanings.
So if you have a word like star, that can be astronomical object
or it can be a film star, a Hollywood star,
or it can be something like the gold stars
that you got in elementary school.
And just taking all those uses of the word star
and collapsing them together into one word vector.
And you might think that's really crazy and bad,
but actually turns out to work rather well.
Maybe I won't go through all of that
right now because there is actually stuff
on that on Thursday's lecture.
Oh, I see, thanks.
You can look ahead of the slides for next time, oh wait.
I have a quick question, I know this might seem kind of strange
to ask, but I guess a lot of us were also
taking this course because of the hype between AI and speech
recognition.
And my basic question is, do we we look the stack of something
like Alexa or something to provide speech
to context or actions in this course,
or is it just primarily understanding?
So this is an unusual and unusual quarter,
but for this quarter there's a very clear answer which
is this quarter there's also a speech class being taught,
which is CS224S, a speech class being taught by Andrew Maas,
and you know this is a class that's
been more regularly offered, sometimes it's
only been offered every third year,
but it's being offered right now.
So if what you want to do is learn about speech recognition
and learn about sort of methods for building dialogue systems,
you should do CS224S.
So for this class in general the vast bulk of this class
is working with text and doing various kinds of text analysis
and understanding.
So we do tasks like some of the ones I've mentioned,
we do machine translation, we do question answering,
we look at how to parse the structure of sentences
and things like that.
In other years I sometimes say a little bit about speech,
but since this quarter there's a whole different class
that's focused on speech that seem a little bit silly.
[INAUDIBLE]
--focus more and more on speech [INAUDIBLE]..
I'm now getting a bad echo, I'm not
sure if that's my fault or your fault but anyway, answer,
so the speech class does a mix of stuff.
So I mean the sort of pure speech problems
classically have been doing speech recognition.
So going from a speech signal to text and doing text to speech,
going from text to a speech signal and both of those
are problems which are now normally done,
including by the cell phone that sits in your pocket using
neural networks.
And so it covers both of those, but then between that,
the class covers quite a bit and in particular it
starts off with looking at building dialogue systems.
So this is sort of something like Alexa,
Google Assistant, Siri, as to well assuming you
have a speech recognition, a text to speech system,
then you do have text-in and text-out.
What are the kind of ways that people
go about building dialogue systems like the ones
that I just mentioned.
I actually had a question.
So I think there is some people are noticing
that the opposites were really near to each other, which
was kind of odd but I was also wondering,
what about positive and negative balance or lack of it?
Is that captured well in this type of model
or is it not captured well with the opposite sides where
it really--
So the short answer is both of those.
So this is a good question, a good observation.
And the short answer is no, both of those
are captured really, really badly.
I mean there's a definition of--
when I say really, really badly I
mean what I mean is, if that's what you want to focus on,
you've got problems.
I mean it's not that the algorithm doesn't work so
precisely, what you find is that antonyms generally
occur in very similar topics because whether it's saying
John is really tall or John is really short,
or that movie was fantastic, or that movie was terrible.
You get antonyms occurring in the same context.
So because of that, their vectors are very similar
and similarly sort of effect and sentiment based words.
Well, like Mike creating terrible example,
the context is similar.
That if you're just learning this kind of predict words
and context models that no, that's not captured now.
That's not the end of the story, I
mean absolutely people wanted to use neural networks
for sentiment and other kinds of sort of connotation effect.
And there are very good ways of doing that, but somehow you
have to do something more than simply predicting words
and context because that's not sufficient to capture
that dimension, more on that later.
I just happen to like adjectives too,
like very basic adjectives, like so and like not.
Because they're sort of like appearing
like some context here.
What was your first example before not?
Like so.
This is so cool--
So that's actually a good question as well.
So yeah, so there are these very common words that are commonly
referred to as function words by linguists, which now includes
ones like so and not, but other ones like
and prepositions, like to, on, you sort of
might suspect that the word vectors
for those don't work out very well because they
occur in all kinds of different contexts.
And they're not very distinct from each other in many cases,
and to a first approximation I think that's true.
And part of why I didn't use those as examples in my slides,
but at the end of the day, we do build up vector representations
of those words too.
And you'll see in a few lectures time
when we start building what we call language models.
That actually they do a great job in those words as well,
I mean to explain what I'm meaning there.
I mean another feature of the word2vec model
is that actually ignore the position of words,
so it's said I'm going to predict
every word around the center word
but I'm predicting it in the same way.
I'm not predicting differently the word before me
versus the word after me, or the word two
away in either direction, they all just
predicted the same by that one probability function.
And so if that's all you've got, that's
sort of destroys your ability to do a good job at capturing
these sort of common more grammatical words
like so not an end.
But we build slightly different models
that are more sensitive to the structure of sentences,
and then we start doing a good job on those two.
OK, thank you.
I had a question about the characterization of word2vec.
Because I read [INAUDIBLE],, and it seems the character
architecture [INAUDIBLE].
It was slightly different from how
it's presented in the lecture, so are
like two complimentary ways together or--
Yeah, so I've still got more to say,
so stay tuned Thursday for more staff on word vectors.
So word2vec is kind of a framework
for building word vectors, and that there
are sort of several variant precise algorithms
within the framework.
And one of them is whether you're predicting the context
words or whether you're predicting the center word.
So the model I showed was predicting the context words,
so it was the skip grand model.
But then there's sort of a detail of how in particular
to do the optimization and what I presented
was the sort of easiest way to do it, which is naive
optimization with the equation, the softmax equation for word
vectors.
It turns out that that naive optimization
is sort of needlessly expensive, and people have come up
with faster ways of doing it, in particular the commerce thing
you see is what's called skip grand
with negative sound playing, and the negative sampling
is then sort of a much more efficient way to estimate.
Things and I'll mention that on Thursday.
Right, OK, thank you.
I was asking for more information
about how word vectors are constructed beyond the summary
of random initialization.
And then gradient based additive [INAUDIBLE]..
Yeah, so I sort of will do a bit more connecting this together
in the Thursday lecture, I guess to sort of--
I mean so much one can fit in the first class,
but the picture is essentially the picture
I showed the pieces of.
So to learn word vectors you start off
by having a vector for each word type
both for context and outside and those vectors you initialize
randomly, so that you just put small little numbers that
are randomly generated in each vector component.
And that's just your starting point,
and so from there on you're using an iterative algorithm
where you're progressively updating those word vectors,
so they do a better job at predicting which words appear
in the context of other words.
And the way that we're going to do
that is by using the gradients, that I was sort of starting
to show how to calculate and then once you have a gradient,
you can walk in the opposite direction of the gradient
and you're then walking downhill, i.e.
you're minimizing your loss and we're
going to sort of do lots of that until our word vectors get
as good as possible.
So it's really all math, but in some sense word
vector learning is sort of miraculous
since you do literally just start off
with completely random word vectors,
and run this algorithm of predicting words
for a long time, and out of nothing emergences word vectors
that represent meaning well.