Welcome everybody, you're watching mr fugu data science. Today we're doing something pretty cool,
cosine similarity usually found in a data mining course. If you're doing any natural language
processing, you'll come across this at some point. Feel free to hit me up here on instagram and
twitter as well as if you like to help support the channel hit me up right here on buy me a coffee.
Let's get into this. Before i forget thanks to the recent subscribers and viewers i greatly
appreciate it. Here's the imports that you're going to need right here pause the video if you
want to follow along. Otherwise go in the link in description below and get the code from the
Github. Cosine similarity, is something you may remember from your trigonometry class. Where we're
doing a dot product and using the denominator to create a unit vector for our A and our B.
What you're essentially doing is looking at the angle measure between the two vectors and you're
looking to see if there's a higher value of theta. Then you'll have a lower value for your cosine of
theta which would mean that the similarity value would be lower. So the value you get back from
the cosine theta would be lower. Meaning that if we were comparing two documents for example,
document A and document B and if there was a large angle between those two vectors then
that would mean that we have a dissimilar or not likely similar text. Essentially what we're doing
if you expand this out we're doing the sum A and B divided by the square root of the sum of A
squared multiplied by square root of the sum of B squared. Let's look at an example real quick. What
this would look like if we did it by hand if we had these two vectors A and vector B of length. I
think it's seven or eight eight. We would first do the dot product to get our numerator of A and B.
Then we would do the squaring of each of the terms, then add those together and then take
the square root of that. Do the same thing for a multiply them together and then divide both of
them and see what you get in this case we would have a similarity of 0.417. You're going from 0
to 1 on your scale. That means it's almost halfway there. Let's see what this looks like when we're
actually using text documents. Because, what you're doing is using word counts. What we're
going to use today are some tweets. I have 50 tweets that i got a few months ago, and here
they are right here. We're going to extract those out and start manipulating those from
those 50 tweets. I did some regular expressions to show you something that we need to get rid
of. Which is going to be highly useless, not all of the entries will have links. But we need to
take care of these. So i'm going to extract those and replace them with an empty space,
so i take those tweets. I use our pattern and i use the regular expression compile and i
iterate through our panda series that contains our 50 tweets. And i'm substituting with an
empty space here and appending everything as a list. Here's the first two entries of what we
got back. Remember, the first one that we had we had a link to it and you'll notice this ellipsis,
where the text was continuing or ended abruptly. For some reason that's not going to help us in
our analysis. But i want you to just be aware of this, all right . One thing that you can do,
if you're trying to standardize these texts. I would consider making everything lower case
beforehand and that would be one thing i would consider. In terms of taking out your
stop words for instance or useless words. i'm not sure if that's necessary when you're doing
cosine similarity or not. We'll call in from sklearn, our feature extraction text to import,
the account vectorizer which is allowing us to create our frequency counts for each of the words.
And we'll be basing those words off of columns. If we're throwing that into a data frame and
our rows will be the number of entries we had or number of tweets which would be 50. We're
using the count vectorizer with our stop words for english, and then we're doing a fit transform on
that to create our sparse matrix. Let's print these off real quick zoom out a little bit
,and see what these look like, all right. We can see each one of these steps
convert this into our matrix and flatten it out and then chuck it into our panda's data frame
allowing us to take our new matrix. Our document matrix and the names from our features and
that's what's printed off here. You'll see a bunch of zeros. But it is working because,
you'll see these counts here. You just have a lot of words which is right there. We'll move on to
our cosine similarity doing pairwise comparisons here. We'll throw in our data frame and then you
have the option of doing a dense output or not. And i'll just print this off so you can see what
this looks like. You'll notice that this first entry so as 0 0 1 1 2 2 3 3 etc, all the way to
50 50 on the diagonals. They're all gonna equal one because, that's doing your first row tweet,
your first column tweet or the same exact tweet. So if this was from the first row for your
original data frame that's the same thing you're doing a comparison with here. For zero zero,
then for this next row it'll be the second one versus the second one comparison that's why you're
getting a one. This is going to kind of mess this up later for what i'll show you. Something that we
overlooked, all right. You'll see examples online that are showing you how to actually
calculate this and print off what we just had, and it basically stops there. But what i want to
show you is taking the original tweet finding the highest value for the cosine similarity per row
and extract that value, and then return the cosine similarity and the actual tweet that corresponds
with that number which would be the index number. You'll see in a second, we have quite a few steps
to manipulate this process all right. The first step here, i decided i need to iterate inside
of our data frame which i converted into a list of values, of nested values. And what i needed to do
was figure out how do i actually let me print this off for you. For this part one it's printing off.
This is the index or this is the row number this is the index position for that row. You have 50
rows. You have 50 index positions per row. This is the cosine similarity for zero zero, this is the
cosine similarity for row zero index position one. That's what this is. So at zero zero we need to
take care of at one one. We'll scroll down at one one two two etc. We're gonna change these to zeros
because, what i'm going to do is put these back into a data frame and allow us to do comparisons
to find the maximum value per row. With this one here it's going to mess you up. So you can't
really do what i'm trying to prove. That's what allows us at the second part here to do. If the
zero zero occurs, or the one one occurs basically just append a zero to that and replace that one
is what's going on here. And just leave your row number your index position the same. Otherwise,
if you're not at a one one two two 3 3 etc. Just append those values in and that's what's
going on here. You'll notice this is what's changed. The next thing we're going to do
is another two-step thing. You could pause this and read it real quick. But, i need to do a few
things i need to create a default dictionary where i'm going to store those values that we had above.
Let's go back, i'm going to take each one of these index or these row positions that's going to be my
key. Then my value is going to be the cosine similarity values here. I iterate through and
create my default dictionary with my new values that you're going to see. Here's the row number,
and here's the 50 50 values that correspond to that row number. Which now, has given me
the ability to start creating a data frame later. You'll be able to see that so i create that data
frame and you get a 50 by 50 matrix and everything that was on the diagonal that you see here.
That zero were the ones. We take that and iterate through it and we take the largest value by
row and save that in here. But that's going to give me not the value but, the index position of
where the largest value is. This is going to correspond to our original data frame where,
now we can iterate through this list and just append a new list with the tweets that we want
from here. I iterate inside of the positional argument and store the new list with our tweets.
Our new tweets corresponding to their largest similarity values here. That's what's getting
printed out. What i did was store the actual values here which are the similarity values
for all the positions that had the highest value and that's what this is. Now we got another piece
to our puzzle. Let's go down to the next spot where we put everything together. This is the
data frame of the tweets where i took out all the html stuff. The links that were unnecessary
and i concatenate everything together, print out the first eight entries so you can see what's
going on here. We parse this through we have our parse tweets. Our tweets based on similarity which
were their max similarity per row, and its actual value. Then we could see here's this particular
tweet and here's the value that was most similar to it for that row same thing, for all of these
so you'll say okay. Well what do you mean similar. Well each row and column you have a 50 by 50. You
originally had 50 rows or 50 items but, then you're doing a comparison wise adjustment for
each. So that's why you have a 50 by a 50. And then i did this one had i think the highest value
for similarity which was 90 basically 0.91 out of one. You'll see that it's it's fairly similar with
what it's talking about. What i wanted to also do, was show you make this a little smaller. With a
heat map so you can see what this actually looks like. Here's all of our entries. Here's our 50 by
50. You'll notice that there's some spots in here that seem like they're equaling one or very close
to one, which would mean that you have something that's extremely similar to the next tweet. You're
gonna see where this was kind of an issue and one of the things that i need to mention in this code.
There's a few things that we missed, and that's what i want to bring to your attention. Now that's
what you look at for exploring your data and understanding what's really going on. And giving
an explanation of when things go awry. A few things that we should consider due to the cosine
similarity here. There were seven entries that had three tweets which were the same exact data,
not all seven were the same. But you may have like two lower same and two in the same and maybe three
were the same, something like that. In the future what you want to look for is are there duplicates
in your data check to see if they're actual valid entries. Did these tweets come from the
same person, did they come from different people? Since they're retweets you know for instance,
in this case it may have came from different people retweeting it. That's what would give
you the ability to have the same tweet just by a different person. Understanding what's going on
with your data is highly important, and you should always look and investigate. Instead of blindly
do something. And if there's a circumstance maybe not in this case you may be able to get rid of or
discard data. That just depends on a case-by-case situation. I noticed that there were 43 out of 50
unique entries, seven of which were the retweets that were the same this tweet low was parsed it
had the lowest cosine similarity of zero. And it's just pure nonsense, it looks like one this may be
either a troll or somebody just giving junk, or someone possibly not knowing how to use twitter
and just throwing people's handles for some weird reason. I don't really know what's going on there.
But that's an explanation of why this one would have a complete zero, and not look like anything
else. But i'd like to say please like, share and subscribe. And if you subscribe turn on that
notification bell. I hope this very utility to someone. Feel free to go here on my instagram and
twitter. As well as if you'd like to help support the channel hit me up here. Thanks for watching
see you in the next video. Bye