hey everyone welcome to the semicolon in this video we'll be learning about a very popular
method for representing text called word Word2vec so machine learning and deep learning algorithms
cannot accept text directly we need some sort of numerical representation so that the algorithms
can process the data and in simple machine learning applications we use count vectorizer
or tf-idf both of which do not preserve any relationship between the words this is where
word embeddings come in they map all the words in a language into a vector space of a given
dimension so word2vec is a popular method to generate word embeddings this converts words
into vectors and with vectors we have multiple operations like add subtract calculate distance
and that is how the relationship among the words are preserved so one example of this relationship
is a very famous result of word2vec which says the vector of the king - man + woman
gives you the word vector of the word Queen and this relationship is preserved by word2vec
just by iterating through a large corpus of text like a Wikipedia or newspaper corpus so word2vec
is generally included in a lot of deep learning courses but there's no deep learning involved in
generating word embeddings using word2vec, it is just a simple two layered neural network now that
being said a lot of deep learning applications involving text have shown improvement after using
word2vec embeddings as features training the word2vec model is also very resource-intensive
as it requires large amount of RAM to store the vocabulary of the corpus so now the question is
how are these relationships formed? In simple terms word2vec tries to make the words with similar
contexts have similar embeddings so let's consider an example to understand it better so we have
two sentences the kid said he would grow up to be Superman and the child said he would grow up to be Superman
if you observe the words kid and child there are two different words with same context and
since word2vec tries to make words with similar context have similar embeddings kid and child will
have similar vectors when you iterate through a large corpus you would get a lot of sentences
where kid is replaceable by child and hence these vectors will have similar embeddings and how does
word2vec do the task of generating vectors from the words so there are two algorithms for
it continuous bag of words (cbow) and skip gram in continuous bag of words or cbow we predict the
target word from the context something like this and in skip gram we try to predict the context
words from the target word you may ask why are we trying to predict words when we need vectors for
each word I'll answer the question later on in the video for now let's look at the detailed working
of each of this method so for that we'll need a smaller example because English language has
around 13 million words in the dictionary which is quite huge for an example so let's consider a
language which has just five words so those words are can free hope set and you and nothing else so
we initially encode each word as one hot vector and to generate word vectors from this vocabulary
we iterate through a corpus and for simplicity let's assume our corpus has just one sentence that is
hope can set you free so now we need to select a window size for iterating over the sentences
so let's consider it to be three in this case and as discussed earlier in continuous bag of
words we try to predict this Center word from the context words so our window size is 3 the Center
word is the word we need to predict using the two words surrounding it so we use a simple neural
network for this and when the neural network has the context words hope and set as input we need
we need a neural network to predict can there is one more thing which we need to choose that is
the size of each vector let's say the size of each vector is three then we'll choose the hidden layer
to be of size three so our neural network now looks something like this one important thing to note
here is the 5 X 3 input matrix is shared by the context words and we pass the one hot
vector of hope and set and a neural network tries to predict this enter word we pass it through the
softmax function get the probabilities and compare it with the actual word and then the error is used
to update the weights then we slide the window and continue the same process this way the weights are
updated and once done we take the weight matrix of these weights and these are these set of our
vectors and this is how continuous bag-of-words (cbow) work the case of skip gram is very similar in
skip gram we choose the window size and give the Center word as input trying to predict the context
words the weights are updated in a similar way and here again the output matrix of size 5 x 3 is
shared by the context words this way both the algorithms help updating the weight matrix based
on the words and context then we generally take the input weight matrix and multiply it with the
one-hot vector of a particular world to obtain it's word vector now this word vector is what we call
word embeddings in real applications we generally train it over Wikipedia text choosing window size
around 5 to 10 and vector size is generally around 300 the number of words in the corpus is around 13
million and hence it takes huge amount of resource to train and generate word embeddings if you want
to skip this process you may very well go ahead and download the Pre trained word vectors you can
choose any one of the two algorithms skip gram or continuous bag of words (cbow) in general continuous
bag of words is preferred for smaller corpus and is faster to Train skip gram is slower but
works well on large corpus and large dimensions there are few other things to tweak to improve
accuracy like adding more data and this kind of obvious we can also increase the dimensions of
word vector and in this way more information will be preserved increasing the window size also helps
but it makes training little difficult so this is all about word2vec and word embeddings in
the next video we'll look at how we can implement this in python thank you