Welcome everybody. You're watching Mr. Fugu Data Science.
Today, we will use one hot encoding, principle component analysis with K
modes and we'll do some plotting. As always the code is in the link in
description below. We need a few new libraries today. If you do not have
them please install K lar (klaR), matrix , cluster. We are also using markdown,
knitter, tidy verse, and quad tee da (quanteda). If you're new to this: we're dealing with
sales data. Today, we're interested in the description, item description, quantity,
and unit price of that item. So each item description is just a string though we
had to pre-process using natural language processing in the previous
video where we created word clouds. If you're interested in that. Please, look at
the video in the cart above. Now, we created tokens last time. Where we
separated a string into individual words and broke that down. To take out
punctuation, digits and any word symbols.There were still words that
we had; that were either useless or didn't have meaning. Such as these, that I
further took out. If you notice here we have almost one and a half million
individual words. But, these are not unique words. These are just words that
occur outright. What we need to do is break up these words. And if you notice
we still have further processing that needs to be done. For instance, these
digits right here need to be taken out, as well as individual characters. Such as
these that are just plain useless. In order to do that. You call a function
which is from quan teda (quanteda) called tokens underscore remove. I'm not coding this
out today. Because, I'm having issues with memory and my machine
and processing these data while recording. We call this function
called tokens underscore remove. Where we take our list of words but I call it
tokens with no stop words. And I'm using the pattern for regular expressions of
taking digits between 1 and 5 in length separated by a word. Then I need to take
at least a minimum character length of each string of 3. Where, I call the
separate function of tokens underscore select. Where I take my new data that I
store. If you notice here we lost a little over 70 words out of our 2001.
From this you can still do further processing: such as word stemming to get
rid of further words. So if you have like toys, toy box, toy. And you only wanted to
keep toy instead of all the different versions. And then and then you take
another unique and that shrink this list down even more to be more concise. I
didn't do that. But, that that's something I would suggest is do word stemming next.
Then to do the 1 hot encoding. What we're basically doing is having rows which are
the item description which is a word string. And the columns are the list of
individual words. And you're trying to find out does this substring exist? So
you say does this individual word exist within this string of words or sentence.
You're basically creating a matrix of this with three hundred eighty seven
thousand rows by almost two thousand columns. So that's pretty big. You got
a figure this is gonna take a long time with doing the looks. It took me about
10 minutes with 16 gigs of ram in a i7 processor, on my older macbook pro. So what
you're doing here is you're basically taking your list of words that you're
doing your comparison with. While you're calling a function you're creating with
grep and you're looking at each individual string for this particular
word. Then if that word is there you get a true otherwise you get a false.
So you have 1930 columns by 387 thousand rows. Since, this s apply returns a
matrix. I convert it into a data frame. From this data frame; I then need to
manipulate it. Where I'm saying I am using the s apply what which takes in as
an input a list a data frame, or a vector and it returns a matrix or vector.
Then the L apply takes in a function and returns a vector. So this is that
logical returning a true or false which is all of these columns here. So I'm
saying this call what we just stored our variable in and take all rows with the
logical columns of true and false and change that true and false into a
numeric which would be your binary variable for your encoding. From there
since I'm returning either a matrix or a vector. You throw it into a data frame
once again. Then I chose to add a new column for unit price. What does this
even look, like right? So let's look at this real quick. Here's, what it looks
like. We have our binary encoding so we did our 1 hot encoding. We have this
separate column right here. This is what I'll do for some pre-processing. So I can
do my PCA. Now, you can take this as it is and do PCA. You could also do any kind of
regression if that's what you're doing for that analysis. From there I stored it
as distinct values. But, this is a little misleading. Because, here if I wanted to
do the counts of each one of these columns. You're going to have a problem.
So let's negate that. And I want to mention something. If you want to do any
kind of analysis that allows you to do this kind of setup. Like one hot encoding
or label encoding. And let's say you wanted to create a sparse matrix. You
could use the library matrix with the capital M and create a sparse matrix.
I showed this for two reasons the amount of data you're saving is 100 fold. So
it's 100 times Moll are almost in size by converting
this exact data into a sparse matrix. That you could still input into whatever
analysis you're doing that's relevant. So consider that if you're using a large
data set and memory is a problem. Now, I want to show something here. Where is it?
There it is. I would like to take each one of these unit prices
and put them in a range of values that I could use such as this. The reason I
wanted to do this is because, I wanted to link similar items by prices and use
these prices here in this column and transform my data. Where this row is
actually a column. So I can use for my clustering, okay. What's going on with
this? It's kind of goofy, right? Well if we separate this. You have just a regular
if else statement. But it's highly nested. What would you do with this? We'll just
make up a variable name, so we could print this out. So I need to do it's
called rest of data, and create a new column called WoW X. Let's just do
that. Okay, so what happens here? I created this
new column called Wow X. So anything less than $1 gets its new label and anything
else becomes just an O. So what you do is you do the same similar thing. But,
instead of the second parameter. You throw in another if else statement. And
continue this in order to create your threshold. And so I did five different
ones. This may look intimidating but if you break it apart just pay
attention: that each time you have your second parameter, that you would call in
after this you just throw another if-else in there, and then at the end you
just have the last value that you would like. Unless, you wanted to go further. So
that's what's going on with this. So I have to subset these data.
I'm only interested like I said in the beginning is these three columns.
Where I want the quantity, the unit price and the description of my item. And from
there; I need to take the data subset. Where I'm saying I would like to take
this price range and make that my columns. Instead of having
long data which is a bunch of rows I want to have it go wide. Which is in
column wise instead. And then I want everything on the rows. Think of like an
ID. You know if just rows just think of it like that. And that's all my
descriptions and then since this is basically pivoting everything; you end up
with a bunch of Na's that I have to fill with zero. And so that's
what's going on here. If you notice here: is my description
which is basically like an ID column. All these columns are my quantity based on
one to three dollars, three to five dollars, less than a dollar,
etc. And this is what I use to plug in for my principal component analysis. So I
call PR comp. Which is the function in the cluster library, and I'm excluding
the first column. Because, it's non numeric. This right here and then I call
K-modes. Because, originally I was doing categorical data. And so I
said arbitrarily let's look at calling that same data iterating eight times. I
don't want any weighted values here. And then I want to do seven for my I think
it's centers? I can't remember oh so be modes. There's a thing called a
scree plot. That you can create to help you dictate how many clusters are ideal
for your data. So what I did here after you you do your PCA. You have a few
things that are available to you, okay.
So you have a few things that are related. I called the standard
deviation; this is all of your principal components, and then this is rotation. I
didn't scale in this circumstance and those are your centers. what I did was I called
standard deviation squared which will give me my variance proportion, here.
This is showing me that with this plot roughly 60% of my variance is already
covered before I had the second principal component. Which is pretty good.
Because, usually most of your variance is explained within the first two principal
components. Then if you don't do that squared for the variance. What you end up
with is your eigenvalues. If you pay attention. And it's just a little bit
different, not exactly the same. Then you could plot each principal component
relative to each other, all right. Since, I have five principal
components here. There since I have five principal components. I could just kind
of browse through and look at what these look like.
I colored them, with the K modes cluster. So pay attention to that.
Here we are: showing that x-axis deals with all of our principal components.
Okay, so I'm taking that column-wise and I'm taking the variance and I'm dividing
by the sum of that. And that's what's giving the proportion of variance that's
explained it within our data. Now, there's something interestingly you can note. So
the scroll back here to the top. And then say scale equals true. Do the same thing.
But, this pay attention that just changed. Scaling our data. Now, it's saying that
only 22% of our data is explained. and Now, this pay attention here and you look
at this and you say wow that's really interesting. Everything changed as soon
as I scale the data and centered. So that's what's going on with this. Also, if
you pay attention the explained variance for everything changes and it's almost
all the same. Now, I'm not going into the explanation of what these data mean
today. I'm just showing you how you code it and explain how you can use in
essence, a custom one hot encoding. if you had to make it. As always I would like to
say thank you for watching. Please, subscribe and turn on that notification
bell. And leave a comment, let me know what I could do to make this channel
better for all of us. See you in the next video. Bye