Today we are going to talk about
language processing pipelines in spaCy,
and this is little different than
your usual NLP pipeline.
In last
video, we created
a blank NLP object and that had a
tokenizer.
But it was missing
a language processing pipeline. So just
imagine,
that after tokenizer
there is this pipeline, pipeline is
basically a bunch of components,
and this pipeline right now is blank,
when you do spacey.blank
So you take text as an input do
tokenization after that there are a
couple of component you can add in this
pipe.
What are those components?
And after that, you get your ultimate
dock object. Well those components could
be this tagger, parser
NER stands for named entity
recognization. Don't worry! I know you are
confused right now.
We will cover all of this in detail, but
a pipeline is something that comes after
tokenizer,
and you can have number of processing
steps.
Let me show you directly in the code
here,
I have run jupyter notebook, I'm gonna
create a new notebook,
import spaCy
as usual.
There is a typo. Okay so import spaCy and
whatever we did in the
last video, I'm going to do the same
thing,
where
we created this
sentence,
and we created first of all a blank
language processing pipeline,
and
you can see it can tokenize. So it can
tokenize because
as we saw
that although my pipeline is blank, I get
tokenizer component by default. But I
don't have a pipeline. Now what does that
pipeline do really?
In order to understand that,
see let me first show you that the pipe
names are blank,
you know this blank array
shows you this blank component which has
nothing.
And now,
we are going to use some pre-trained
pipeline with different components, and I
will show you what those NER what
those jargons you know those, those magic
words, what does it mean really.
So if you go to
spaCy documentation,
there are different languages you have
correct?
And for each language
you can download a
pre-trained pipeline, by
running this command. So you can just run
this command, this is for English
if you want something for Chinese, it
will say this.
It just changes this component. So it is
something that you have to install.
So
let me just copy,
go to my command prompt.
Here I have Git Bash. You can use Windows
command prompt also,
and just hit enter. I have already
downloaded it, so maybe it will not
download again. But
this is downloading a
trained pipeline for English language.
This
en means english
sm means small
okay? But let's let's first see what does
it do?
So once
you have downloaded,
now instead of creating a blank pipeline
I will do something like this: See, here I
had NLP is equal to space.blank
Here I am doing spacey.load en_core_web
sm So en_core_web_sm
I want to take a moment and talk about
firstlanguage.in which makes
building NLP applications easier for you.
You can perform various NLP tasks which
are shown here via this platform if
you're using
a spaCy type of library, you download it
you run your code locally. If you are
training a heavy model, you need higher
compute resources, such as GPUs. Whereas
here everything is in cloud. You just
make an HTTP call,
things run on the cloud you don't need a
local machine with lot of compute
resources.
There is a demo which you can see by
clicking on this link, and I have
bought this bulb recently, and you see
there is a negative review on Amazon, and
when I copy paste this here,
and I select text classification and
when I say submit,
it immediately classifies that as
negative. And if I have a positive review,
I just copy paste it here,
and if I submit it, it will classify this
as a positive. See it's pretty powerful!
So it makes uh things much easier for
you. You don't need to have like too much
like detailed NLP knowledge. You can
just call APIs and get things done
quickly for yourself. The free tier
is something you can try today. You can
just sign in, and you can get an API
access. Once you sign in, if you go to
dashboard, you will get your own API key
and you can use that API key to make the
calls.
They have SDKs available both in Python
and Typescript. Go check it out! The link
is in video description below, and thanks
firstlanguage.in for sponsoring
this video.
Now when I do the same thing NLP pipe
name,
see it has all this components okay?
You can also
say pipeline, and you have all this
component. Now what does this component
do really?
Let me show you,
so when I do
just when I print
this tokens, now instead
along with tokens,
I'm going to print few more things.
So I'm printing this,
I'm hitting this pipe just to kind of
separate things apart.
You can do token.
pos
part of speech
okay? And we will look into part of
speech, and all of these in details. Don't
worry, okay? Right now you just watch what
I'm doing.
Sometimes you know you have to be
patient. You have to see what exactly I'm
trying to show you, and then you can
understand.
So
POS is part of speech. So we will cover,
if you are not very familiar about
English language, we'll cover the English
fundamentals like language one on one in
in future videos. But I will just simply
explain, part of speech is when you're
speaking an English sentence,
every word
has some meaning right? When I say, "Dhaval
eats apple", Dhaval is a noun.
'Eats' is
a verb
correct? So similarly see here, 'ate' is a
verb. 100 is a number.
Then there is proper noun
and common noun. Proper noun is,
Dhaval is a proper noun, and when I say a
person eats apple, a person is a noun.
Person is for a general group. But it is
a noun. We can say person eats apple, so
person is a noun, eat is a verb. But when I
say, "Dhaval eats apple", Dhaval is a
noun, but the type of noun is proper noun.
So Dhaval is a proper noun, eat is
verb.
So it is showing
that
part of speech.
The third thing it is showing is lemma,
which is the base word. For example ate is
a past tense. But the actual word is eat.
Said is a past tense.
The actual word, the base word is say.
So
you got
part of speech because of this tagger
component in the pipeline. See you can
print pipeline or you can just print
pipe names.
So the tagger component gives you,
this tagger component gives you proper
noun.
Lemmatizer gives you lemma.
Then there is NER okay? What does NER do
really?
So in NER
you can have a sentence like this,
and you can say
for ent like entity
in
doc.ents
print
ent.
text
and
ent.
So see
here
it,
okay let me just put
pipe here just to kind of separate
things out.
So here it says
Tesla is an organization and 45 billion
is money. So it is
recognizing the
entities. You can also,
let's see
there's something called
spacey.explain
and it can I think explain the label,
the entity label.
So see or get org means companies,
agencies, institutes, money means monetary
values.
So now you understand when you are
loading a trained pipeline with all the
components, you get some inbuilt features.
And all these components that you are
seeing,
NER, lemmatizer
it is nothing
but this,
correct? So this is your language
processing pipeline. You have tagger, parser,
NER You can have n number of
components. You can even customize these
components,
and you can
use some of the
predefined pipeline which I showed you
here. See,
these are the ones.
Now see if you don't see anything yet,
for example for Hindi language,
I don't have a pipeline. So I have basic
language support with a tokenizer. But I
don't have a pipeline. But I have
pipeline for
all these languages, which I can download.
That's why you don't see Hindi in
this particular
drop down,
okay?
All right!
And by the way, to show
these entities little in a fancier way,
you can use
this module called from spacy
import
displacy So this is just for display,
and you can say
display.render and you are rendering
the
document and you're saying the style is
equal to entity,
and see it kind of,
it's just a nice visual display of the
same thing.
It is showing Tesla inc is a
organization and 45 billion is money. So
you can have organization, money, you can
have people.
For example,
if I say
Bloomberg founded data analytics company
Bloomberg.
Now Bloomberg is a person name. So that's
your person entity,
and when I say founded Bloomberg,
the other Bloomberg is actually a
company name,
correct? I hope you're getting point.
What I'm saying is
Bloomberg
founded
data company called
Bloomberg.
Here
the first
term
is person.
Whereas the last term is
org correct?
So
entity, named entity recognization NER
that's the component, you know. Here that
component is called
NER, named entity recognization,
that
allows you to recognize the entity from
your text.
And we will look into NER
and part of speech
later in detail. This video is mainly
done to give you an overview
of
this whole
language processing pipeline. Now see,
I did all of this here correct? See
same sentence,
when I use this particular pipeline, I
get
POS, lemma, everything. But if I copy paste
this same thing here,
where I'm using a blank pipeline, then
what will happen is I don't get any POS,
lemma, nothing. Blank pipeline is blank
pipeline. It doesn't give you anything. I
hope you're getting the point.
See when you have a blank pipeline, after
tokenizer you don't have anything. It's a
blank pipeline.
Whereas, when you use
something like
you know load that particular English
trained pipeline, you get tagger, parser
you get all these components,
created for you automatically
correct?
So when I use this same thing,
you can notice the difference in the
output here, right? Here I am saying
spacy.load english here spacey.blank
correct?
And see,
same thing like, if you're using blank
pipeline you won't be able to recognize
entities.
Now you can use
pipelines in different language. For
example, French.
French
so for using French trained pipeline, you
have to run this.
If you don't run this, you'll get an
error correct? So
I have already
installed that, so I'm not gonna get any
error,
and it will download now the French
pipeline.
If you're not downloaded that, you will
get an error like this. See
like cannot find whatever. So just just
run this command,
run this command first, and then
download the
French
pipeline.
And
I have a trans, I use Google translate to
translate the same sentence into French,
the same sentence into French.
And I can
now do this,
see Tesla Inc is going to acquire Twitter
for 45 billion dollar.
I'm just making this up. I know Elon Musk
wants to buy Twitter. I don't know that's
gonna happen in the future. Maybe in the
when you're watching this video, Twitter
might be controlled by Mr. Musk
Writing some crazy tweets at four
o'clock in the night!
All right!
So see, this works even for,
like even though sentences in
French, it
it kind of worked.
Now,
when you have a blank pipeline, let me
just,
okay so I have,
okay I have this black pipeline okay?
All right!
I have this blank pipeline,
and I want to,
let's say print the entities. So where
are the entities?
So I want to say
this,
and I will use
this particular sentence.
Yep! So you'll notice that this code is
not printing anything because I don't
have any entities. But let's say I want
to use a blank pipeline. I add my custom
components. I don't want
let's say, all these components
you know. I just want,
my the problem that I'm solving, I only
care about NER.
I don't care about all these components.
So what you can do is,
you can first load
the name entity recognizer, you want to
use it from your English pipeline. So you
can first load the English pipeline
and then,
you can do something like this. So you're
creating a blank pipeline, and in that
you're adding NER,
you're adding NER and you're saying
source is this, which means from my
English trained pipeline, add NER
component. And when I do NLP.
pipe names it will show you see it has NER
And now,
when I print this,
see it works!
It says like this is,
okay this did not work
well because I used a French sentence.
Maybe I should use an English sentence.
Let's see.
See initially when I was using French, it
said Tesla as a person, because Nikola
Tesla it thought it is a Nikola Tesla, a
person who invented electricity. But here
um,
I have English pipeline, and I want
supplying English sentence, so it can
understand better.
So you you see
you can
customize a blank pipeline, and add your
custom components, and that those
components could be sourced from
a
different uh trained pipeline itself. All
right? So that's all we had for today's
video.
We don't have any exercise for today. But
what I'll do is uh in the next few
videos, as we learn more concept, maybe
after 2 or 3 videos, I will give
you consolidated exercise that covers
the concepts you've got you learned in
last few videos, all right? If you're
liking this video give it a thumbs up!
Share it with your friends and,
um I'm hoping you are enjoying this NLP
tutorial playlist. The link of the
playlist is in video description below.
Thank you! Bye!
[Music]