our cat teacher is going to teach you
cosine similarity and cosine distance.
how it is used in data science. We'll
look at some theory and then we'll move
into python code.
Let's say you are data scientist working
for some financial company
where on a Google drive you have a bunch
of financial documents.
Now you don't know for what is the
company associated with each of these
documents but when you open the document
when you read it you can kind of figure
out that
this is probably about Apple. Why?
because iPhone is mentioned so many
times
so when you're reading about apple
financial report
they might mention galaxy as well
because that's their main competitor
but here the ratio of iphone to galaxy
is iPhone have three iPhone
is mentioned three times galaxy is
mentioned one time
so you know overall you see iPhone
occurrence more much more than Galaxy
If your Samsung document of course
Samsung company's document
will have more mention of galaxy than
iPhone.
Now let's say new document comes in
and you don't know what is the company
associated with it.
Here you can again count iphone and
galaxy and you find that iPhone is six
times mentioned in the document galaxy
is mentioned two times.
Looking at the situation as a data
scientist you can
figure out a simple formula that
whenever a ratio of iphone to galaxy
is three to one it should be
an Apple document so now you can auto
annotate Apple as a company this is a
common problem in financial
institute where you have a document and
you want to
tag some metadata to it and we can use
some automation,
some rules, some coding to auto annotate
you know you can manually annotate it
but using this particular formula
you can automate the annotation process.
Unfortunately things in real life are
quite different
in our document. You know there might be
mention of iPad
Google Pixel is another competitor so
now how do you come up with your formula
maybe you say okay iPhone to Galaxy ratio
has to be three to one and iPhone to
Pixel ratio has to be three to one and
iPad to Pixel ratio has to be two to one
well.
That's too complicated! My baby got
confused!
Well vector mathematics comes at risk.
This thing can be presented as a vector
here
and using vector mathematics we can
figure out document similarity. Let me go
back to the simple case that we saw
before
how do you represent this as a vector on
x-axis. I have iphone word count y-axis
has galaxy
word count so you see 3 2 1
and this is the vector. Vector has
magnitude and direction both.
And when I have my yellow document
iPhone is 6, galaxy is 2.
Now look at the angle between
these two arrows blue and yellow the
angle is zero.
So the angle determines the document
similarity.
If the angle is zero it means documents
are very similar
you might have another green document
You know iphone 5 galaxy
one time here the angle is still
not that high you know. It's a little bit
of an angle so you can say these
documents are still similar
but when you have a Samsung document
where iPhone is mentioned only once
galaxy is four
you see the angle is much bigger. So when
you have a bigger angle you can say
these documents are not similar meaning.
If for yellow arrow I know companies
Apple
for blue arrow, I can say company is
definitely definitely not
Apple versus if I have a green arrow
where the theta 2 angle is much you know
much closer to yellow arrow
you can say green and yellow documents
are similar whereas blue is
not similar let's say the angle between
these two is 17 degree
you can use this angle to define
document
similarity. One thing you can say is
document similarities this is 17 degree.
I mean when I say that it doesn't sound
that
obvious or intuitive. It's not a
good way to represent similarity what if
i can present the similarity between
some number between let's say zero to
one.
When I say document similarity is 0.9
which means they're 90 similar you know
if the document similarity is one they
are very similar if it is 0 they are
very different. Ok. So how can you
transform this
degree angle into a range?
Because that range seems like a good
convention
you just take cosine if you don't know
about sine cosine, I made a previous
video previously in the same series
watch that but cosine of 17 is 0.95
now see my convention is much more
obvious, much more easier. I can say
document similarity
here between green and yellow vector is
0.95. Hooray!
AQll right now cosine similarity is
nothing but a cosine
of an angle between the two vectors. And
the great part about
math is we looked at very simple
scenario of
two dimensional vector in real life you
will have
hundred dimension vector and these
dimensions are basically the features.
So your document might have 100 features
and
you can present those as a vector
and you can still do the math. See that's
the beauty of math you cannot visualize
100 dimension
but the math will continue to work, Okay
the academic formula is between
a and b these are the two vectors and
the cosine similarity between these two
vectors
is dot product which is a dot b
divided by magnitude of a magnitude of b
and a dot b is usually b cos theta so
you know it is cosine similarity is
nothing but
cos theta. All right
now if you have arrows pointing in the
same direction then the cosine
similarity is 1 which means those
arrows those vectors those documents are
quite similar.
If they are at 90 degree means the
similarity is zero they are very
different
and if it is 180 degree similarities
minus one which means
they represent opposite con concept
now cosine distance is a very simple
concept just it is used to represent the
same thing it's one minus cosine
similarity
so here when two arrows are pointing in
the same direction when you're talking
about
same vector the cosine distance will be
zero.
So when you say distance is zero which
means they are similar they are closer
that's the only idea behind cosine
distance when
you have two vectors at 90 degree cosine
distance is 1, which means they are very
different and cosine distance is all
represented only in a positive space.
That's why
I'm not talking about that 180 degree
case that we saw in a previous slide.
Let's write some python code now we'll
be using python's escalant module for
importing cosine similarity method here
and when you
have this method it expects two vectors.
And those two vectors are going to be
the case that we looked at for Apple
document was
three one and six and two so let's see
what is the similarity between three one
and six and two so three one.
Six and two now this method expects
two dimensional array so you have to put
one additional array here. I mean that's
the signature of the method
you see similarity is one which means
the documents are very similar.
And if you find a cosine distance of the
same thing
is going to be be zero.
Cosine distance so this is two
dimensional okay
See one e raised to minus 16 this is
very close to zero.
Okay now if you
find out a distance between
let's say these two documents you know
three one three two.
See there are iPhone three times the
galaxy one times
iphone three time galaxy two times still
they both are apple documents
so there is an 0.96 per
similarity one similarity means they're
very very similar.
Now let's look at some real documents so
I'm going to
to create some variable
variables. Here with financial document
string.
So you can see the first one is iPhone
apple document. Apple document
samsung samsung okay and
what you can do is you can create a
Pandas data frame so I'm going to import
Pandas here
and we create a Pandas frame
and the data in the data frame we are
going to hard code
some arrays okay? So what are those
records in the first document you see
iPhone came one two three times galaxy
is one time
okay so iPhone three times galaxy one
time.
Similarly, I have pre-counted the iPhone
and galaxy word count in
remaining documents and
that's that's what it looks like now in
the index.
I want to supply the document as an
index so
instead of 0 1 2 3, I will say okay doc 1
1.2.3
and my data frame will look like this so
it says document one has
iPhone three times galaxy one times and
so on
you can clearly see first two documents
are Apple documents
and second and third documents are
Samsung documents. Now
when you do something like this it
returns you
first document but it is two dimensional
array or a data frame
and our Cosine similarity function
expects
that data frame and that's why we are
doing this you know.
otherwise I would have done simply this.
I want to compile let's say
document one and document two so I would
have done this
right but see it expects two
dimensional array and that's the only
reason
I am doing this kind of range
okay we are getting some error let's see
oh yeah this is a syntax error actually.
It has to be this way
see they're very similar 0.94 so we can
say both are Apple documents.
But if you do the same thing with let's
say document one and document three
they are not that similar see point six
why point six well let's compare doc one
and rock three see three one and one
three.
This some similarity you know but
if you do
dog three and four let's say then they
will be
more closer to one.
3.98
and now for the same documents if I do
cosine
[Music]
distance
it will be 0.01 so cosine distance is 1
minus similarity. So if you add these two
numbers they will add up to one
that's all pretty much I had if you want
to do the same thing in tensorflow then
tensorflow has
this particular function that you can
use. I hope you like this video if you
did please
share it with your friends. Thank You.