If you want to turn words into numbers...
...and you want those numbers to make sense...
...then use word embeddings and similar words will have similar numbers! Hooray! StatQuest!
Hello, I'm Josh Starmer and welcome to StatQuest! Today we're going to talk about
Word Embedding and word2vec and they're going to be clearly explained. Lightning
makes using the cloud just as easy to use as your laptop and that's cool! This StatQuest
is also brought to you by the letters 'A', 'B', and 'C'. 'A' always, 'B' be, 'C'
curious. Always be curious! Also, I want to give a special thanks to Alex Lavaee
and the students at Boston University's Spark! for letting me test this StatQuest
out on them. Note: This StatQuest is focused on how neural networks can be used to
create word embeddings and assumes that you are already familiar with the basic ideas
of neural networks and backpropagation, as well as the main ideas about the softmax
function and cross entropy. If not, check out the Quests. Words are great. We can
use them to communicate all kinds of cool ideas. For example:
Hey Norm, isn't StatQuest awesome?
It sure is Squatch!
Unfortunately, a lot of machine learning algorithms, including neural networks, don't work well with words.
Hey, Neural Network, isn't StatQuest awesome!
So if we want to plug words into a neural network, or some other machine learning
algorithm, we need a way to turn the words into numbers.
24. 0.3 5.1
Yes, StatQuest is awesome!
Bam!
One super easy way to convert words into numbers is to just assign each word to a
random number. For example, if someone just saw the hit movie Troll 2 and said, "Troll
2 is great!", we can just give each word a random number. Note: For the purposes
of this StatQuest, were going to treat "Troll 2" as a single word. Anyway, if the
next person said, "Troll 2 is awesome!", then we could reuse the random numbers that
we already selected for "Troll 2" and "is", and assign a new random number to the
new word, "awesome!". In theory, this is fine, but it means that even though "great!"
and "awesome!" mean similar things and are used in similar ways, they have very different
numbers associated with them, 4.2 and - 32.1, and that means the neural network will
probably need a lot more complexity and training because learning how to correctly
process the word "great!", won't help the neural network correctly use the word "awesome!".
So it would be nice if similar words that are used in similar ways could be given
similar numbers so that learning how to use one word will help learn how to use the
other at the same time. And because the same words can be used in different contexts,
or made plural, or used in some other way, it might be nice to assign each word more
than one number, so that the neural network can more easily adjust to different contexts.
For example, the word "great" can be used in a positive way, like "StatQuest is great!",
and it can also be used in a sarcastic, negative way, like "My cell phone is broken,
great", and it would be nice if we had one number that could keep track of the positive
ways that "great" is used and a different number to keep track of the negative ways.
Hey Josh, deciding what words are similar and used in similar contexts sounds like
a lot of work. And using more than one number per word to account for different contexts sounds like even more work.
Don't worry Squatch. The good news is that we can get a super simple neural network
to do all of the work for us. Bam! To show how we can get a super simple neural network
to figure out what numbers should go with different words. Let's imagine we have
two phrases: "Troll 2 is great!" and "Gymkata is great!". Anyway, both phrases have
two words, "Troll 2" and "Gymkata", in similar contexts, because someone, and I'm
not going to name names, thinks that these two terrible movies are great! So, in
order to create a neural network to figure out what numbers we should associate with
each word, the first thing we do is create inputs for each unique word. In this case,
we have four unique words in the training data, so we have four inputs. Now we connect
each input to at least one activation function. Note: This activation function uses
the identity function, and thus, the input value is the same as the output value.
In other words, this activation function doesn't do anything except give us a place
to do addition. The number of activation functions corresponds to how many numbers
we want to associate with each word. And the weights on those connections will ultimately
be the numbers that we associate with each word. Now, in this example, we want to
associate two numbers with each word. So that means we'll use two activation functions.
And the weights on the connections to the second activation function will be another
number associated with each word. However, like always, these weights start out with
random values that we will optimize with backpropagation. Now, in order to do backpropagation,
we have to make predictions. So we'll use the input word to predict the next word
in the phrase. So if the phrase is "Troll 2 is great", then we can use the word "Troll
2" to predict the word "is". In other words, if the input word is "Troll 2", and
we indicate that by putting a 1 into the "Troll 2" input and 0s into all of the other
inputs, then we want the output for the word, "is", to have the largest value. And
if the input word is "is", which means that the input for "is" is is 1 and all of
the other inputs are 0, then we want the output for the next word, "great!", to have
the largest value. Lastly, if the input word is "Gymkata", which means that the input
for "Gymkata" is 1 and all of the other inputs are 0, then we want the output for
the next word, "is", to have the largest value. In order to make these predictions,
we connect the activation functions to outputs and we add weights to those connections
with random initialization values. And then we run the outputs through the softmax
function because we have multiple outputs for classification. And that means we can
use the cross entropy loss function for backpropagation. Again, the goal is to train
this neural network so that it correctly predicts the next word in a phrase. And
now, before training, if we plug the word "Troll 2" into the input and do the math,
we get lucky and correctly predict the next word, "is", but just barely. However,
when we plug the word "is" into the input and do the math, then we fail to correctly
predict the next word "great!" and instead predict "is". So we need to train this
neural network. However, before we optimize all of the weights, remember that I said
that these weights would be the numbers associated with each word. And since, in
this example, we have two weights for each word, we can plot each word on a graph
that has the weight values to the top activation function on the x axis and the weight
values the bottom activation function on the y-axis. For example, the word "Troll
2" goes here, because it's top weight is 0.38 and its bottom weight is 0.42. Likewise,
the word "is" goes here, and the word "great" goes here, and "Gymkata" goes here.
Now, with this graph, we see that the words "Troll 2" and "Gymkata" are currently
no more similar to each other as they are to any of the other words. However, because
both words appear in the same context in the training data, we hope that backpropagation
will make their weights more similar. So we start with these weights and we end up
with these new weights. The new weights on the connections from the inputs to the
activation functions are the word embeddings. And when we plot the words using the
new weights, which are now the embeddings, we see that "Troll 2" and "Gymkata" are
now relatively close to each other compared to the other words in the training data. Bam!
Now that we've trained the neural network, we can see how well it predicts the next
word. For example, if we plug the word "Troll 2" into the input and do the math,
then the output for the next word, "is", is 1 and everything else is 0, and that
is exactly what we wanted. And when we plug the word "is" into the input and do the
math, then the output for the next word in both phrases "great!" is 1 and everything
else is 0. Lastly, when we plug the word "Gymkata" into the input and do the math,
then the output for the next word, "is", is 1 and everything else is 0. Bam! Now,
before we talk about a popular word embedding tool, word2vec, let's summarize what
we've learned so far. First, rather than just assign random numbers to words, we
can train a relatively simple neural network to assign numbers for us. The advantage
of using a neural network is that it can use the context of words in the training
data set to optimize weights that can be used for the embeddings. And this can result
in similar words ending up with similar embeddings. Lastly, having similar words
with similar embeddings means training a neural network to process language is easier
because learning how one word is used helps learn how similar words are used. Double
bam! Now, so far, we've shown we can train a neural network to predict the next word
in each phrase. But just predicting the next word doesn't give us a lot of context
to understand each one. So now let's learn about the two strategies that word2vec,
a popular method for creating word embeddings, uses to include more context. The
first method, called "continuous bag-of-words", increases the context by using the
surrounding words to predict what occurs in the middle. For example, the continuous
bag-of-words method could use the words "Troll 2" and "great!" to predict the word
that occurs between them, "is". The second method, called "skip-gram", increases
the context by using the word in the middle to predict the surrounding words. For
example, the skip-gram method could use the word "is" to predict the surrounding
words "Troll 2", "great!" and "Gymkata". Lastly, before we're done, just know that
in practice, instead of using just 2 activation functions to create 2 embeddings
per word, people often use 100 or more activation functions to create a lot of embeddings
per word. And instead of using 2 sentences for training, they use the entire wikipedia.
Thus, instead of just having a vocabulary of 4 words and phrases, word2vec might
have a vocabulary of about 3 million words and phrases. Thus, the total number of
weights in this neural network that we need to optimize is 3 million words and phrases
times at least 100, the number of weights each word has going to the activation functions,
x 2, for the weights that get us from the activation functions to the outputs, for
a total of 600 million weights. So training can be slow. However, one way that word2vec
speeds things up is to use something called Negative Sampling. Negative sampling
works by randomly selecting a subset of words we don't want to predict for optimization.
For example, say like we wanted the word "aardvark" to predict the word "a". That
means that only the word "aardvark" has a 1 in it and all of the other words have
zeros, and that means we can ignore the weights coming from every word but "aardvark",
because the other words multiply their weights by zero. That alone removes close
to 300 million weights from this optimization step. However, we still have 300 million
weights after the activation functions. So, because we want to predict the word "a",
then we don't want to predict "aardvark", "abandon", and all of the other words.
So for this example, let's imagine word2vec randomly selects "abandon" as a word
we don't want to predict. Note: In practice, word2vec would select between 2 and
20 words that we don't want to predict. However, in this example, we just select
one, "abandon". So now word2vec only uses the output values for "a" and "abandon",
and that means for this round of backpropagation, we can ignore the weights that
lead to all of the other possible outputs. So, in the end, out of the 600 million
total weights in this neural network, we only optimize 300 per step. And this is
one way that word2vec can efficiently create lots of word embeddings for each word
in a large vocabulary. Triple bam! Now it's time for some shameless self-promotion.
If you want to review statistics and machine learning offline, check out the StatQuest
PDF study guides and my book, The StatQuest Illustrated Guide to Machine Learning
at statquest.org. There's something for everyone. Hooray! We've made it to the end
of another exciting StatQuest. If you liked this StatQuest and want to see more,
please subscribe. And if you want to support StatQuest, consider contributing to
my patreon campaign, becoming a channel member, buying one or two of my original
songs, or a t-shirt, or a hoodie, or just donate, the links are in the description
below. All right until next time Quest on!