Today we are talking about Stemming and
Lemmatization, and these are the essential
steps that you need to perform in
pre-processing stage, while building NLP
application.
You might have noticed, when you search
in Google
with some word,
it might search for some
other words as well, which are matching
with that particular word, which is here
I'm talking about talking,
and then it will match a word talk.
So what we are doing here is,
essentially reducing the word to its
base word.
And not only,
like Google search is just one
primary example that I gave you for your
understanding, but even in text
classification. Let's say you are doing
sentiment classification, and you are
using some words such as talking, during
the
classification review, product review.
Then if someone has used the base word,
which is talk you know, talking, talk,
talked in past tense,
they all mean the same thing.
So you want to map all those words to
its its base word. There is a value in
reducing
a word to its base word.
I am giving some other examples here,
eating to eat,
ate is a past tense. But if you can map
that to eat,
which is the present tense the base word,
then it can have a value in your NLP
application.
Similarly, adjustable to adjust.
You'll notice that to get the base word
of a given word, sometimes you can use
simple rule,
such as
if the word has -ing as a suffix, just
remove -ing.
Talking, walking, running if you remove
-ing you get the base word.
Eating,
there are other
suffixes as well, such as adjustable,
amicable,
remarkable.
So if you remove -able, then you get the
base word.
So the point I want to make here is, by
using simple rules,
without using any language knowledge,
just by using simple rules, that
if the word has -ing, remove -ing, or
remove -able. You can have a list of these
suffixes, and you can use the rule that
remove these suffixes
from a given word, to get a base word.
Since we are using
fixed rules or heuristic, this process is
called stemming. It's like stemming,
you're just stemming
the
given word and getting the base word.
But on some other occasions,
applying just the fixed rule is not
going to work.
Ate to eat
you can't apply this rule. You have to
have that
knowledge of that language.
So whenever you want to use knowledge of
the language, or a linguistic knowledge,
jargon alert, linguistic knowledge is
nothing but a
knowledge of a language.
When you want to use that knowledge of a
language, and you get the base word,
that base word by the way
is also called lemma.
So eat
is a lemma for ate,
and this process is called lemmatization.
So stemming is just to summarize using
fixed set of rules, dumb rules
that remove this x number of prefixes or
suffixes to get a base word.
Whereas lemmatization is little more
sophisticated, where you need need to
know the rules of that particular
language. You need to have linguistic
knowledge.
You might be guessing already that with
stemming you can get into issues
sometimes. For example
if your word ability, if you just apply
the dumb rules,
it will remove -ity and you will get
this base word abil, which doesn't mean
anything.
Whereas if you apply lemmatization, you will
get the same word ability.
But
as you
learn more about NLP application, you
will realize that there is a value of
both. So stemming and lemmatization, both are
used.
And we are going to write some code.
By the way,
for stemming we will use NLTK because
spaCy doesn't have support for stemming.
There is an article
where spaCy authors have mentioned, why
they don't support stemming because
lemmatization as you can already see is
more sophisticated, more correct. So they
just
have lemmatization support but NLTK has
both, stemming and lemmatization both. They
support both. So I'm going to
show you
NLTK and I will demonstrate
stemming.
I want to take a moment and talk about
firstlanguage.in which makes
building NLP applications easier for you.
You can perform various NLP tasks which
are shown here, via this platform. If
you're using
a spaCy type of library, you download it.
You run your code locally. If you are
training a heavy model, you need higher
compute resources such as GPUs, whereas
here everything is in cloud. You just
make an HTTP call, things run on the
cloud. You don't need a local machine
with lot of compute resources.
There is a demo which you can see by
clicking on this link and I have
bought this bulb recently, and you see
there is a negative review on Amazon. And
when I copy paste this here
and I select text classification and
when I say submit,
it immediately classifies that as
negative. And if I have a positive review,
I just copy paste it here.
[Music]
And if I submit it, it will classify this
as a positive. See it's pretty powerful.
So it makes uh things much easier for
you. You don't need to have like too much
like detailed NLP knowledge, you can
just call APIs and get things done
quickly for yourself. The free tire
is something you can try today. You can
just sign in and you can get an API
access. Once you sign in if you go to
dashboard, you will get your own API key
and you can use that API key to make the
calls.
They have SDKs available both in Python
and TypeScript. Go check it out. The link
is in video description below, and thanks
firstlanguage.in for sponsoring
this video. So let's go to a notebook. So
I imported
NLTK and spaCy and by the way, if you don't
have NLTK installed, you can in your
command prompt,
where is my command prompt?
In your command prompt, you can just run
pip install NLTK, very basic right,
to install it. I have already
imported both of these libraries, and now
I'm going to
[Music]
import
PorterStemmer from NLTK So from NLT
[Music]
NLTK.stem
import
Porter you can just hit tab to auto
complete,
and Stemmer This is a class, so I 'm
creating an object of this class in
Python. When you do this,
this is a class and you're creating an
object of this class, storing it here.
There is other
uh,
stemmer as well, called SnowballStemmer
but I'm just going to show you
one.
And I have a couple of words here,
and you can say for word in words.
Print so I'll print a word,
and then
some separation okay,
and stemmer.stem
that particular word.
And it prints the
the, it does the stemming and it prints
the base word. So you you see it applied
fixed set of rules. So from it removed
-ing got eat.
Then when it has eats, it will remove -s
and eat. If you have a base word, it will
not do anything. So eat is eat. But not as
this. Ate it did not do anything,
because stemmer doesn't have
a knowledge of a language. It is just
using fixed rules.
Adjustable it is saying adjust.
Rafting, raft. Look at this, ability abil
Abil doesn't mean anything. But still
you will notice that many applications,
even production applications, they use
NLTK
and
using stemmer, could be efficient because
it is faster. It just applies bunch of
rules. It doesn't require any language
knowledge, and it can have a value
in your NLP application. So although it
is dumb, it can have value. Now let's do
lemmatization in spaCy. So this was a
quick demo of
stemming in NLTK
So
as usual, so I'm going to now create NLP
object,
and say spaCy.load
en
core
web
sm is small, you can use medium. I'm going
to use small version of
english language processing pipeline.
If you don't know, what I'm talking about
you have to watch previous video.
And then I will create
some document. So
let's
use the same word.
Here
and see in spaCy it's pretty
straightforward. You can just say for
word or for token
in a doc
print token
some separator
and you can say token.lemma
underscore
Notice this ate it reduces to eat.
Eating, eat. See, it worked fine.
Adjustable it did not say adjust because
the lemmatization, the rules which are set
by
default english model, says that the base
word for adjustable is adjustable.
Ability is ability. But look at this,
when you have better it reduced that
word too
well.
So it has this mapping rules in the
trained model that you just loaded. so
based on the
language-based on what kind of model
you're loading, you might be loading a
different english model, which will have
a different rule.
So there is
this is a quick demo of lemmatization
in spaCy. If you by the way, let me show
you one more thing, which is
this this thing called token.lemma
So see there is lemma and there is lemma
underscore So lemma is just printing a
hash.
So it's a unique hash of your lemma
which is a base word, because this is a
train english model with a fixed
vocabulary. It has few thousand words and
for each word it will have a unique
identifier sort of
and that is this hash. And you'll see
that for eat the hash is same. This is
just for your knowledge by the way. I
just wanted to quickly show you
different uh sentence.
So I'll just do copy paste, and I will
try.
I don't know if you all watch
Mandalorian. Mandalorian is my favorite
series on Disney.
So Mando doesn't talk that much. But
Mando talked for three hours! Hurrah!
Mendo is becoming talkative!
Mando
for three hours, although talking isn't
his thing.
He became
talkative,
and I will remove
so see
talked it reduces to talk. Talking is also
reduced to talk.
ut talkative is still
talkative because this is,
this is how as I said this is how, this
particular model behaves.
Sometimes you might want to modify a
behavior, right? You want to customize it.
So how do you do that? So
if you remember our pipe names, so we
have
for this particular model
the pipeline that we have is this:
there's a tok2vec, tagger, parser
attribute ruler and then there is a
lemmatizer. Now attribute ruler
assigns attribute to a particular token,
and you can customize it. So let me
explain
what,
what I mean here. So let's say you have
this sentence.
Now bro,
brah, bruh, all these are
the
[Music]
you know you mean brother basically, when
you say this.
But by default, the language model
doesn't understand this is slang, right?
So it doesn't understand. So it will say
bro is bro,
brah is brah, right? And if you look at the
first token itself,
so
like the first token is this, correct? Bro
And if you look at the
lemma of that,
it is bro,
because it doesn't understand the slang.
But let's say you want to customize this
model. You know that bro and brah means
brother, you can customize it by doing
this. So what you do is,
you go here, you get this particular
attribute ruler
by doing this. So you say
nlp.get_pipe attribute ruler. So it will
give you that particular component from
the pipeline,
and then you can customize it by adding
that custom rule, okay? So see, here I am
adding a custom rule, where I am saying
this.
So what I want to say is
I want
lemma to be brother
for few words. So for bro and brah, I want
their lemma to be brother, okay? So here
you are assigning attribute and
attribute is lemma. So basically, you're
saying the lemma
for the words that I'm going to specify
here,
and the way you specify those words, I
mean that is this,
I'm just going to copy paste
using
for brother bro and brah,
make their base word to be brother.
So you notice previously bro was bro, and
brah was brah. But when I run this again
bro is brother, and brah is also brother.
And
for the bro
see the lemma was bro, but now the
lemma became
brother.
So I hope this gave you uh
some understanding of
stemming and lemmatization. I don't have
an exercise for this particular video,
but I might add exercises in the future
as well. Do always
check the video description below,
because sometimes if I don't find time
for the exercise while recording this
video,
and if I find time later on, I add those
exercises. And I
I
talk about those exercises in the video
description below. So always when you are
watching my videos, on my channel any
video, make sure you check video
description, because I might have
included more information. After I
publish the video, if I find some
correction,
if I want to add some exercises, some
additional information, articles, I always
add them in the video description. So
make sure you check it out. I hope you're
liking this series. If you do,
please give it a thumbs up and share it
with your friends. One of the way you can
share this series is, by posting on
LinkedIn.
LinkedIn there are some serious folks. So
you can post about this series, so that
other people learning NLP can also
benefit. Thank you very much for watching!
If you have any question post, in a
comment box below.