What is going on everybody? Welcome to
part two of our NLTK or natural language
toolkit tutorial video with Python. In
this video, we're going to be talking
real briefly about stop words. So, what
are words? So, with a lot of data
analysis, you're going to find that
before you actually get to do any of the
good stuff, you're going to have to do a
lot of
pre-processing. And arguably, that's
basically what NLTK, the module, does
help you with. So NLTK is not going to,
you know, generate any insights for you
really. All it's going to be able to do
is help you kind of analyze, pull apart
text, uh, tag things either part of
speech, named entity, and so on, that
kind of stuff. But it can also help you
with, uh, the topic we're going to be
talking about today, and that's stop
words. So what are stop words? So
there's kind of like two notions of what
stop words can be. One, stop words can
be something that just like you
literally you you come across that word
and you just run away. You give it up
and you you like leave. So, a lot of
times people might use words that are
typically used sarcastically as a stop
word because they don't want to continue
attempting to analyze something when it
may or may not be the actual like
opposite meaning. So, that's one notion,
but another notion of stop words are
these words that you just pull out. You
just don't care about them. and they're
fluff and you don't need them. So, this
would be like a the and for the most
part, uh these kinds of things don't
really have any meaning to your text. Uh
therefore, you can just remove them.
They're filler words basically and they
make our language to us make a lot more
sense. But as far as data analysis is
concerned, they're useless. So, uh let's
go ahead and get started there. So, uh
first of all, we're going to go ahead
and do from
NLTK.corpus. again making a huge use of
Corpus. We're going to be you'll find
that we're using corpus throughout this
entire series. Um and I highly suggest
you check it out. Later on we'll
actually go visit the corpus and see all
the stuff that we have. Um anyway from
nltk.c corpus import
stockwords. Then from
nltk.tokenize we're going to import
word_tokenize.
Now we need to make an uh example
sentence. So let's just say uh example
sentence equals
um
this is an example
showing
off stopward
filtration. That may or may not work. Uh
we'll see. Hopefully there's enough in
there that will uh be good. So now we're
going to say
stop_words equals and we're going to say
the set of uh stop
words and the stop words has a few
languages. We're going to be using uh
the English language. So all of our stop
words is going to be b a set of English
stop words that have already been
predefined by NLTK. Uh we don't have to
do anything about that. Now you can add
your own words. You can append words to
stop words and so on. So for example,
let's just print. I don't believe it's a
huge uh set here. Let's just print them
and see what it is. Right. So here are
your stop words. It's a set of them.
We've got whom between yourself if so
own you or. So it's actually not that
big of a list, but these are just really
really common English words. you could
probably figure out a good list of stop
words by pulling up um any of those big
corpora and using you know from
collections import counter and then
taking every single word adding it to a
big list and using counter and like the
first I don't know 20 words are probably
stop words because they're just
useless. So anyway uh those are stop
words and uh those are the stop words
rather but now let's show like what we
can do with them. So uh let's tokenize
the sentence. So, let's do words equals
word do or word underscore. Oh my
goodness, I'm totally going crazy. Word
tokenize. And what do we want a word
tokenize? Example
sentence. So that's it. So now we have
the words and then now we're going to
filter the sentence. So
filtered underscore sentent. I keep want
to put a D there. Sentence.
Um, so filtered sentence is we'll just
have an empty list for now. And then
we're just going to say for W in words
or actually yeah words for W in words if
W not in stop
words
filtered sentence up
append W. Okay. Uh, simple enough. There
is a quick oneliner I'll show you after
this that most people use from examples
with NLTK, but it's confusing if you
don't know those oneliners. So, I'll
show you that in a moment though. So,
we'll save and run
that. And we're not printing the
filtered sentence, so nothing is
happening afterwards. Uh, print and
filtered sentence. Copy paste that in
there. Okay, this so is and was removed.
This example showing off was removed.
stop word filtration period. Okay, so
you can see that at least a few were
removed. Arguably showing off off maybe
should have remained because we were
actually like showing off is almost like
a you know a conjoined uh pair there. Uh
but arguably no the sentence meaning
still remains the same. This is this
example showing stop word filtration
totally fine. So um that's just a quick
quick example. Now as promised I'll show
you the oneliner. Uh, I tend to stay
away from those fancy oneliners just
because people don't really understand
what's happening in them. This is really
simple to follow along. Uh, but I'll
show it to you anyways. So, you could
say um instead of
this, I'll just comment that out. You
could say
filtered sentence
uh
equals. So this will be we'll use the
same parameters W for W in
words if not W in stop words. Okay. So
hopefully if I speak it out slowly you
know obviously you see exactly what's
happening here but this is just word for
word in words. So that's just iterating
through and allowing us to throw in this
down here. Uh, so for each word in
words, if not W, so if W isn't in stop
words, then it's part of the filtered
sentence list that we're making. And
then we can just at the end of this
print
filtered. Oh, there's that D again, man.
Uh, filtered sentence. And there you
have same thing. Okay, so that's stop
words. You'll find that it's useful um
especially if you're having any sort of
database or you're pulling out words and
you're say you have like articles on
articles on articles, stacks on stacks
on stacks. Uh this will save you a lot
of processing time to just not waste
your time with words that just simply do
not matter. Um so anyways, uh that's it
for stop words. We'll go ahead and stop
there. Um not too interesting, but also
pretty helpful to have nonetheless. Uh
if you do have any questions or
comments, please feel free to leave them
below. Otherwise, as always, thanks for
watching. Thanks for all the support of
the subscriptions, and until the next
video.