In this NLP playlist we have covered the
text representation techniques from label
encoding to TF-IDF Today we are going to
talk about word embeddings. There are
certain limitations of Bag of words and
TF-IDF which we have discussed in
previous videos, which is the vector size
can really be big for bag of words and
TF-IDF model. And it may consume lot of
compute resources, memory and so on. Let's
say you have vocabulary of 200,000 words
or 100,000 words each vector for each of
the documents would be 100
000 size and that that may be too much
and the presentation is sparse meaning in
that vector most of the values are 0.
So it is not a very efficient
presentation.
The other problem we saw was that let's
say you have 2 words I need help, I
need assistance these are similar
sentences. You expect that their vector
representation should be similar, but
since these are TF-IDF and bag of words
are count based methods, the vector
representation might not be similar. Here
you can see see there is 1 here 0
here 1 here 0 here Uh so these
techniques do not capture the meaning of
the words properly. You expect that if
you have two sentences which are similar,
their vectors will be very similar.
Word embedding tries to address both of
these shortcomings.
First
similar words will have similar vectors,
similar words or even sentences. For
example good and great these are two
similar words. When you will use word
embedding you will notice that their
vector will be very similar. See 3.1 3.1
4.4 4.2 there will be slight differences
but mostly they will be similar.
They will also be lower in dimension
which means the size of this vector will
be 300. 300 is a one of the popular
vector sizes in word embeddings. It could
be 50 size, 100 sizes, etc. But it won't be
like 100,000 because your vocabulary
size is 100,000 and these
representations are dense representation.
Meaning you won't notice too many zeros
as opposed to the previous techniques
where most of the values in the vectors
were zero. Now there are various word
embedding techniques such as Word2
vec, GloVe, fastText
These are the techniques uh which are
built on different approaches
and the common technique that we use to
build these word embedding which is Word
2vec GloVe and fastText is using
continuous bag of words and Skip Gram.
Now this might be a jargon so just hold
on. I will go over that in detail a
little later.
Then there are techniques such as BERT
and GPT
which are transformer based embedding
techniques. These are the latest
advancements in NLP. When you do Google
search it is using BERT. So you might
have heard all these jargons, so I'm just
going to go over these jargons in brief
and in the later videos, we'll go over
them one by one. So don't worry if you're
confused right now, just don't worry. Just
remember that these are various word
embedding techniques.
There is a technique called Elmo which
is based on LSTM. Now what do these
techniques do? See they convert a word or
a sentence into a vector representation.
So that it can capture the meaning of
that word properly. Not only that you can
do arithmetic with the words. Word2vec
has this famous example that you
can do arithmetic. You know how you do
arithmetic with numbers let's say 2 plus
3 is equal to 5 minus 1 is equal to 4.
You can do similar things with the text
and that's mind-boggling right? See King,
King has all these attributes: Authority,
King is Rich, it has power. But if you
remove the element of man from that and
add woman, it becomes Queen correct? Queen
also has power,
authority, lot of money but the gender is
different. So you subtract that gender, add
a new gender now you get Queen.
We'll look all of this in detail, we'll
look into this in in the detail uh
during the code, but for Word2Vec I
have already
made a different video. So if you go to
YouTube search for Codebasics Word2
Vec that that first video that you see
which is part of my deep learning
playlist please watch that because that
will give you a very good understanding
of the actual method or actual technique
behind Word2Vec. Since I already
have that video, I'm not going to repeat
the same thing here. I would highly
recommend you watch that okay? If you
don't watch it you're going to probably
miss few things in this NLP series. So
Word2Vec very very important concept,
you need to watch that video.
Now there are variations based on the
dataset that they are trained on. For
example Word2Vec you can train on
Google News dataset or some different
Corpus, and based on that you can get
different variations. So Word2Vec is like
a technique. But you can use the
technique on different dataset. It's
like you are training support vector
machine. So support vector machine is one
technique but you can train it on Amazon
reviews or movie classification.
Similarly Word2Vec the actual
embedding vector, you can get by training
it in a different corpus.
Same like GloVe. If you train it on
Twitter, let's say there is a GloVe
Twitter 50 this model, which is available
in Gensim library,
it will understand the tweets better. All
the slang and all the short form that
you use, it will understand that better.
And then there could be a Wiki gigaword
Glove version there are so many
different models. But the basic
techniques are GloVe, Word2Vec fast
text and so on. Similarly the
transformer based models such as BERT
also based on what type of dataset
they're trained on, you can get Bio BERT
or Fin BERT Fin BERT is trained on
financial dataset, Bio BERT is trained on
biomedical database.
And you can also do some parameter
tuning in BERT and get things like
Albert or Roberta. So you would hear all
these jargons, you know fear Fin BERT,
Albert, Roberta, Word2Vec So I'm just
throwing all the jargons so that you get
some overview. Don't worry if you don't
understand these techniques. We'll go
over them in detail in the later video.
But just to summarize what this
technique produces, see folks it's very
simple the whole purpose of these
techniques is to convert word into a
vector because we have seen previously
machine learning models don't understand
text. They need numbers.
So you can convert a single word such as
good to a vector, you can convert entire
sentence to a vector and that is called
sentence embedding or it can convert the
entire paragraph or a news article or a
document into a vector. So in the future
video, we will be writing code and we'll
be using this techniques to solve the
actual NLP problems. This was just a
short introduction on word embedding.
Once again watch my Word2Vec video,
and by the time you come to the next
video, you should have watched that video
okay? This video the first one, otherwise
you're going to miss out on few things.
All right? If you like this playlist, give
it a thumbs up and share it with your
friends. Thank you!
[Music]