Welcome everybody. You're watching Mr. Fugu Data Science.
Today, we're using R studio to do label and one hot encoding for categorical
data. As always the code is in the link in description below. You'll need to
do a few installations of dummy's, cat encoder, and matrix. After you finish that.
You will need to import the libraries of markdown, tidy verse, dummies,
matrix, and cat encoders. Let's refresh on our data. We have some online sales data
though we've been using lately, for our online sales data project. Today,
our focus will be to create one hot and label encoding from the country column.
Let's get into that label encoding. I will call a country label encode. Where
we need to first convert our data into a factor; and then you will need to convert
it to numeric. Let's we get an idea what that looks like.
Let's make sure we did this correctly. Where we first compare the unique values
of the original data to what we just created. So that checks out. Let's do a
second example, using the cat encoders library. Where I will just call it
country with an abbreviation basically. So I'll call it label and code two. And
for this one, label encoder dot fit and then you just call in your data that you
used before. Then we will need to transform that data of our country
encode two that we're creating. You need to also call in a matrix of the
data. As your other parameter and then this should be good. And there we go.
And that's what we just created above two different variations. Now, let's get
into the one hot encoding. So this one will use a design matrix or you're a
model matrix. So what you need to do is we'll call this variable DF cat.
Where I need a data frame and I will take the model dot matrix and then you
use the tilde. Which is this symbol and then we'll call
the rest of our data. And then you need to do minus one. Because, if you
don't do the minus one you end up having something goofy happen. Where you will
create a column called intercept. And I'll show you what's going on with that
in a minute, okay. So that should be good. See what this looks like,
there we are. Here's our encoded data. Let me show you what it looks like when you
don't do that minus one. There you are. It takes away the Australia and you have
this, right here. So let's now do a sparse matrix, where I will do sparse matrix
encode. Where now I'm calling the matrix function from the library we imported
called matrix. And then I have to say it as dot matrix and I'm taking in the DF
cat we just created. Then I could say sparse equals false, right. We'll do
that real quick so you see what it looks like.
That's what it looks like. So let's make it true,
and that's our sparse matrix. Then we can look at what the DF cat looks like, if I
take all the unique values. and I did this sort of illustrate better for us.
Instead of having just one column of ones which is the Australian columns
what happened first. So now you can see on the diagonal these are all of our
columns based on country. So that illustrates nicely. We will now notice
something here: that when you create a sparse matrix, I mentioned this in a
previous video. You'll notice that the amount of memory that's used decreases
immensely. Here we are saving one order of magnitude so it's 1,000 times smaller,
okay. And another thing that I would like to say is that: when you're dealing with
multiple columns of categorical data, it tends to explode in size. So when you're
doing any kind of encoding such as one hot. This can be an issue, so consider
matrices If you can. Now, let's do another example with one hot encoding but, we're going
to use our design matrix once again for a little bit of a different example. So
we will create a design mat2 variable. Where i will do the model dot matrix
again, and then I'll use the tilde symbol. Where I'll call the country minus one and
then call the rest of the data. Okay, so do that and then I can get the head of
that. Oh, we need to do something first. I need to create a data frame. I'll
create a data frame from our design two. Where I say data frame of the design
matrix two, call that in. And then I can show you the unique values for the head
of that. So you can get the gist of what it is. So here you go, that's what it
looks like as the second example of that. Then let's make sure that everything
looks good. So I did this real quick here, to make sure that if I didn't check out
their unique values of this to make sure it was working. This is another way you
could verify that its actually working. Because, if I did not do that. When you
print this off without the unique you just see Australia with one one one one
one and then these walls look like zero. So you won't really know. Unless, you do
something like this unique or you take the column sums to know that it's not
just one column populating your data, and it's working properly. It's like a safety
check. Then I want to use the dummy library form to use one hot encoding
here. That's pretty cool. I'll create a dummy and then I'll just call it Lib
encode. Where I'm going to bind my columns for these data. Take in my
country, and then I'm going to call the dummy. Which is part of that package
for our data; where we're taking the country. And then I'm using a separator
and it's gonna be this underscore, okay. So then you do that, and then okay that's
fine. Then we'll see what this looks like.
Perfect. So we're on the right track once again. Then I can use some one hot
encoding. Where we're gonna instead of just print out only the country columns. We're
gonna have this whole thing I appended to our data frame. So let's do this.
There's only one deviation we do. Instead of using country here. We're
calling the full data frame. So we'll do those both. Now, you notice you have the
whole data frame with all the countries that are attached to that. We can also
use the cat encoders to also do one hot encoding. Where you first convert your
data to a matrix, and then you have to use the transform once again. And you
take what you just created and do a matrix of your data. And you could do
sparse equals true or false. I just did false here so to illustrate this.
Then as a safety check I ended up summing the columns. So I made sure that
it was working properly. And finally I would like to mention as a side note:
there's a few other libraries you can also use for encoding. And after you do
the encoding you could also do your analysis. Depending on what it is. And
that would be correlation funnel, and ml tools. A few things to be aware of: if
you're doing label encoding you may have issues depending on your analysis with
ranking your data. For instance if you label encoded here and there's 37
countries you may have issues with your analysis thinking let you know
country 37 is better or country 36 is better than 37 or 1 is better than 2 or
something like this right. So you got to be careful with that. Also,
there's situations where you can use the label encoding and then further process
using one hot coding, depending on the circumstance. Now for one hot encoding I mentioned
just above. But, I'm gonna say one more time you have memory issues. If you're
dealing with for instance, the description column for the items in this
sales data is like a couple thousand different items sold. So if you made that
wide data you would have issues with memory storing these matrices. So
something like label encoding may be useful there. If not consider the sparse
matrices to help save yourself some memory allocation and speed up your
processes. As always I would like to say thank you for watching. I hope this by
utility to somebody. Throw a like and please subscribe. And if you do turn on
that notification bell. Feel free to leave a comment let me know what I can
do better or if you like this. Have a good one. See in the next video. Bye