In the last video I gave you an overview
of Gensim word embeddings. In this video
we are going to do news classification,
we have this CSV file where the news are
either fake or real and we are going to
build a classifier that can classify
these news into one of these categories
and I took the data set from Kaggle. So
thank you whoever provided this dataset
this is the link I will provide that in
the video description below. I have
already downloaded Gensim Word2vec
Google News model which has been trained
on Google News articles. We looked at
this in the last video
and we also looked at certain functions
of Gensim such as you want to find
similarity between great and good and
see almost 72 percent similarity. You can
also get the word vector of these
individual words
by doing this: so WV is your word vector
model
and if you simply treat that as a python
dictionary and give your word as a key,
it will give you the vector okay? So this
will be word vector let's say great
and we'll do same thing for good for
example
okay, and when you look at this vector uh
it the shape of that vector is usually
300. So the the size of the vector is 300,
you can obviously view the vector which
will not make any sense to you.
Now let's uh classify our news category.
So I just wanted to show you that there
is this functionality available where
you can treat these things as a just a w
v as a simple python dictionary. So I'm
going to
load my data set into pandas data frame
as usual so import pandas
and
as pd pd.read CSV what is the name of
my file?
Well here is the name
and
if you look at df.shape you'll find
I will while this is running I will just
print few lines here.
So see simple data frame two columns,
9900 samples. I want to know
if there is a class imbalance. So I will
quickly do df.label.value counts
if you don't know pandas by the separate
pandas tutorial
playlist
and you see that the there is no class
imbalance, there are almost similar size
so I don't have to worry about you know
treating the minority class and things
like that.
I will quickly create a numeric column
for this label because it is text
right our machine learning models they
need numbers. So I will say label number
and this is like df.label.apply
okay
or you can just use a map function
either way
and you will say
fake is zero and real is one.
So then if you print df what it is doing
is just creating a numeric
representation of your target variable
which is fake or real. I will now convert
this
into word embedding using Gensim's word
embedding method that we saw okay?
But before that I would like to do some
pre-processing because I want to remove
all the stop words,
I want to get the lemmatized version of
all these words. So I will write this
function called
preprocess and vectorize because I'm
doing two things- I am pre-processing it
and also vectorizing it okay?
And
before I write that function let me load
spacy's
large model and if you have not
downloaded this model when you run this
it will give you an error and if you
find an error, just run this command to
download that large model. We saw in the
previous videos that the large model
contains the word embedding. So you need
this okay
and through this playlist we have seen
how you can do pre-processing of your
text. It is simple you create NLP object
out of your text. Then you will say for
token in doc this will give you each
individual token and you will say if
token is
if the token is punctuation or token is
a stop word then continue.
Otherwise, I will just create
filtered
tokens list
and put the filter tokens into that okay?
So filtered tokens
.append and I will not put just the
token but the lemma
okay and
let me just show you how this method
works.
So when I
when I call this method which is some
sample sentence let's say this one
I don't know if you know but Andrew
Ang's favorite sentence if you if you
are doing his courses don't worry if you
don't understand.
So here it removed don't, then if and all
that and just just lemmatizing it okay? If
you have things like worrying or
something it will just still get your
lemma which is worry.
I also want to do vectorization. So how
do you do vectorization? We already saw
for one word
you can get the vector by just simply
treating WV as your python dictionary.
But what do you do with the
sentence? So for sentence
let's say if I have a sentence like this
what I will do is
I will get let's say worry and
understand right? This is my sentence
I will first get the 300 size vector for
this another Vector for this and then I
will average these two.
When I average these two what I get is
another vector of size 300 and this is
my
sentence embedding. Basically the numeric
representation for the sentence.
This is the common method that people
use in in the world of NLP. So let's
explore this method
called WV.
So this WV has this method called get
mean vector.
So see I have all these list of tokens
here. So if I do get mean vector here
and provide list of tokens right, so if I
say worry,
understand
it will give me
the average
of both these two and if you want to
verify it, you can do that by just saying
okay get individual vectors of course
right? So vector 1 is this, vector 2 is
um
understand and once you get these two
vectors
you can do np.mean so
import numpy as np np.mean
v1 v2 v1 v2
axis is zero. We have done numpy series
as well so you can look at it and that's
your mean. So I will just print since the
vector is I think 300 size right?
See if you look at the vector size.
shape it is 300. I will just print first
five maybe first five numbers or first
three first three numbers looks like
this,
and I will just prove that see this is a
mean of these two words. So get mean
Vector for worry and understand should
give you the same result.
So this is not giving the same result
because
get mean vector has this has if you read
the documentation of get mean vector
here it is
it has it is doing
normalization. So pre-normalization by
default is true. I'm gonna just make it
fall so that you get an idea okay? So I
will say this is false
and now you see this these vectors are
same.
So this is a test that get mean vector
is working as per your expectation.
If you normalize it it's good you know
in in machine learning we usually
normalize things. So normalization will
even give you better results, so I'm
gonna normalize it but
I just wanted to show you that get mean
vector
the way it works is it takes list of
words and it will just give you the mean
Vector of that. So you don't have to
manually do that.
So I will just use this function here
okay?
So I will say
get mean vector of this
and the reason I want to do this is
as we saw
that this filtered tokens is list of words
in a sentence and I am interested in a
embedding of the entire sentence and I'm
not going to say pre-process false
because a pre-processing is good I don't
see why we shouldn't do that.
And then when you call pre-process and
vectorize what you get is again a vector
of size how much, 300 exactly! Once we
have done this now comes
a part which requires your patience. So
I'm going to create a third column in my
data frame that will be the vector for
this, and how do you get a vector? Well we
call this function
and in pandas we all know we can do df
text.apply and just use Lambda. So
Lambda
text
okay?
When you run this it's gonna take 10
minutes, 15 minutes, 20 minutes it's a
heavy operation. So go for a walk and
when this is done running, you can come
back. I already have this notebook where
I have already run this and see the
vector will look something like this
okay? And after that we do our usual
process so we do train test split first
of all. So I have 20% test sample and 80%
training samples okay using see
vector.values label number
and then I'll have to reshape these
things because hold on uh let's see
okay I'm creating x train and x test
first and then I'm creating using np
stack I will create the 2D Vector. Why
I'm doing this? Well uh you can hold on.
So if you look at x train for example
right
look at first two samples, what happens
is it's a numpy array but each
individual element is also numpy array.
It is not a native two dimensional numpy
array.
See array and then array but when you do
np.stack what happens is it will
kind of flatten it.
So now you get
native numpy. See you don't see array
and array inside it's a native 2D numpy
array. And we have done this in previous
NLP tutorials that's why I'm not going
into details and I'm hoping that you are
following this tutorial in a sequence
because if you're just directly watching
this video some of these things might go
above your head.
And after that I'm using gradient
boosting classifier. This is again a
usual thing. In my machine learning
playlist, in my NLP playlist this this is
something we have done thousand times. So
I don't want to waste time typing all
this code,
and you will see that the
performance we get is amazing right? 98
percent see precision and recall for
both the classes. This is zero means fake
class, one means the real news class
this precision and recall, F1 score if you
don't know about this again I have
videos so just in YouTube Codebasics
precision versus recall you will get an
idea. You need to have all these concepts
clear
and then I took some news articles from
the same csv file and did prediction,
this was a real news and these two were
fake. So when I did prediction it
correctly predicted. See this is real one
means real, 0 0 means fake
and then I plotted a confusion matrix.
Again confusion Matrix we have done
thousand times in my machine learning
playlist NLP, I'm not going to go in
detail. Just to summarize whatever you
see in diagonal are the correct
prediction which means the news there
were 965 news articles in your test set
which were
true meaning which were real they were
not fake and our model predicted them to
be rare.
Same thing here there are 972 fake
articles in our test dataset and our
model predicted it to be fake. So we got
our prediction right on the diagonal. But
we got some errors here so there were 28
which were fakes because this is truth
which are fake, but our models say it's
real. There are 15 which were real but
our models said they are fake
okay? That's all I have. I have provided a
link of this notebook in the video
description below if you like this video
by the way give it a thumbs up and you
can subscribe to my channel because I
I'm seeing majority of the people who
are benefiting from these videos are not
subscribed and if you subscribe I it
will make me happy. So maybe that's your
fees I'm not asking you for any money so
just say subscribe, if you like my videos
if you don't like
don't worry I still want to say thank
you for spending time to watch this
video.
[Music]