Welcome everybody, you're watching Mr Fugu Data Science.
Today we're using sk learn (sklearn), to do some one hot and label
encoding. One and multiple columns. Let's look at
what our data consists of. The original data set will be country,
device, and version number. It's just dummy data that
i created and the end result should look something like this.
Where we have our label encoding for the version, and country
as well as one hot for the devices. Before, we get into this: i would like to
say thank you to the new subscribers and viewers recently. I greatly appreciate it.
For the label encoding, let's start with doing
one column at a time, the second example will be doing
multiple columns at once. We're going to do the
countries first and then the version number.
This is what the result will look like. So let's create our label
encoder: i will call this label encoder as our variable name.
Take in the label encoder; let's call the label encoder.
We have to do a fit transform, and call in our data. Which
would be the df country and here's what it looks like.
But, let's actually store this as a new column for a data
frame and let's call it country label encoder
see what that looks like. Here's what we look like for
the label encoding of our country. Let's move on to the
next one we're going to do the same thing except: this will be for the
version. Let's print that out. Now, we got the first two columns.
That's looking promising. Let's investigate these data real quick.
So what's going on with this? It looks like from our version labels here.
It looks like it's encoding the number two label
as 12 4 the first label as 10 3 or the zero label i should say. First
one has three four. But, if you notice here this
is looking kind of strange. Because, it's this is seven this is five this is six,
And you have to consider a few things of how this is
actually ordering these. It's calling in the one
and then it's doing any comparison from one
to the other one. The first digit. It's not looking at this as a ten. Then it
would proceed and look at if one and one are tied.
Look at the second position: if these are tied look at this third
position; if that's tied look at these positions right here. And that's how
that's working. Let's do some reverse encoding.
For, instance if this was your label encoding this particular version number.
We can do something here: where we go basically backwards
from our encoding. We can look at our label encoder
and do the same thing that we did before. Let's just
call in the version and only take the first five entries.
Then we're going to call the label encoder. But we're going to now do an
inverse transform; so we can get back all of our information.
Let's just cut and paste that. And then let's take the
first five entries and do a comparison. There we are.
It's looking like
we're able to convert it back, from our labels so this would be the labels for
the first five entries. And this is the actual entries that we
encoded. That's what it looks like. And you can
do a comparison here to see how they match up if you're skeptical or
if you're just verifying that this was actually doing what you thought it was
doing. We can go forwards and backwards. But
we need to do some considerations, all right. Let's think about this
in terms of ordinal data: where you have some form of ordering.
You may have a label that corresponds to actual ordered data which is kind of
a concern. So let's assume that the version; instead
of doing this as a label what if it actually had some
form of ordering. In the circumstance of these data
instead of doing label encoding; we have to account for an
order, okay. And because of that you also have another
thing that you need to bear in mind: Which is right here.
There are circumstances where you have to switch encoding methods.
For instance, if you do label encoding and your data did not
have some form of ordering and your algorithm
thinks that your labels now have some form of ordering. You may need to
also convert that into further one hot encoding and expand that as a matrix. So
that's two different things that you need to think about.
Let's look at this real quick. If i was just comparing these not as
strings: but, as numeric literals where we have an
underscore separating this; can we do a comparison?
And this comparison looks like it's printing off correctly. But, then you
have an issue this 12 is actually bigger than this 10.
But, you have an issue because it's saying that this 10 is bigger than this
12. and then if we skip down to this one we see that this
10 is actually looking like it's smaller than this 9; which is interesting.
Because, it's actually reading this string as this
1 versus this 9. Because, it does not have a second number behind it. So it's only
comparing this with this; instead of the 10.
Okay, now we have this printing out correctly.
Because, you're able to go through and match all of these sequentially and it
finally does a comparison of the one versus the two. So if we're dealing with
the ordinal data: we need to figure out how we can actually map this correctly.
So you would think that converting these into integers
would work, right. And you would think okay this
this looks like it's promising. This looks like something that i could
actually use for ranking. So how would we deal
with this? If we're doing should we do some one-hot encoding first
how should we deal with this? Let's get into that at the end.
First let's get into one hot encoding: just call this one hot enc for encoding.
And let's call in our encoder and that's do something called
handle unknown equals ignore. The description is above right here.
You can read that while i'm typing this in.
Let's create a data frame, and take in what we're calling right above.
We're going to do a fit transform as usual.
And what are we calling in? We're going to take in the devices now.
Since, that was only ipad and... What does this look like? Ipad and
your iphone. Here's what we look like, okay.
Not very interesting. It looks like it's just creating two
separate columns. This seems kind of nonsense: but, it's actually relevant.
In the case that you have many variables that you're mapping. It'll spread this
out into many columns, that can be formed into a matrix. Which is pretty useful.
Just join these data together in order to see what it actually looks
like. There we go so now we have a one hot
encoded devices. Sure that seems useful. Let's
look at inversing this to make sure that
everything works correctly. We can always print everything off
so we can do a comparison.
We want to... I'm going to take in this. Let's take this in.
Let's go and paste this, and put it in and let's take the
first five values. Then create something called
encode. Where we're going to do our fit transform
and then we're going to call in our data frame with our devices.
And then how do we deal with this now? We can do a
inverse inverse transform of the data that we just encoded.
Let's take the first first five entries; and there you go.
You can see that you have your iphone you have your ipad.
Iphone, ipad. So it changes all right. Here, the second entry of these tuples. All
right that looks fine and dandy. We can now multi encode: taking in
as many columns as you have for your data that's able to get
encoded. But we need to now isolate some of our columns. Since, we've been
manipulating our data and doing some encoding. We're only taking the last
three. We can look at that real quick. We're only taking these. Because if i
did the whole thing you can see that we have our encoding.
Let's call this encoding all columns: where we need our new
isolated data frame and we're going to use an
apply function. Allowing us to take in the
label encoder. We're going to do a fit transform as
usual. And what's this look like? Okay cool.
So we got back our encoding. Now, let's take that encoding;
and rename these columns. So just create a new variable and let's just say
df encoding is going to be the data frame with the encoding.
We need to concatenate our data frame
that we just isolated and what we just used. But we need to pay
attention to the axis that we're working with. And let's print this off,
if i pay attention. Let's print this off. We
worked out taking in as the label it was able to read
each one of these and encode them. And the binary or one hot was taken care of
by itself. So that's perfect and i guess there's a distinction by calling this
binary encoding versus this create your array your numpy array.
Now what else can we do? Let's do a second example of this to see
how it works. But, let's do this a little bit different. So let's call in
our collections module and import a default dictionary.
Let's name this d. Run apply function: where we're going to
call in our lambda, create our dictionary. Where we're taking the name of our
variable. And then we're going to do a transform
on this variable. If i paid attention, oh.
And i'm not paying attention. Because, we need to call in the label encoder within
our dictionary so we can actually use this apply function.
So here we are. We're able to use a different method where we call in a
default dictionary. And here if you've ever used these you
can use a list where you basically append
for your keys and all your values will be list. But in this case
you're taking in your values as your label encoding. Which works perfectly. We
can...
Make this a little more appropriate. Do an inverse of the same thing. But this is
going to be a little bit different. let's do an apply function on
our fit. And we're going to do very similar to
here. But we're going to take in an inverse of
this now. and let's see what this looks like.
There we go we got everything back. We can go forwards and backwards.
Lastly, since we're dealing with that ordinal data for our version:
if we were considering it not as just a labeling. But, actual ordering. We should
deal with that now. If we were to say that these were strings of
numeric literals separated by underscores; let's figure out how we can
actually encode this. And it's not too bad. All we have to do
is just iterate within our data frame for the version what we're dealing with.
And then we're going to make a temporary variable
taking in our label encoder, doing our fit transform
and we need to do something interesting. So let's
see what happens if we convert this into an integer.
And let's also create a separate a temporary variable and let's take in
our label encoder again and instead call in the data frame
column and let's print everything out and see what happens.
Okay, so there's a lot going on here. This first one right here
is what happens when we're label encoding
the values as integers. So if we were to take
this and consider it an integer. This is how these would look like
for the first five entries, okay. Then if we were doing a comparison of these
strings. We mentioned before that it says this
is not bigger than this. But we're taking it as an integer that we just did.
You say: well this doesn't look really good. The original data that we took in
doing the transform is this one. Which is represented here
and you'll see the difference in encoding numbers. And what happens is
it's printing off this. This is getting printed off
if we're doing a comparison of taking those numbers as
integers. But, notice this right here where'd the zero come in?
That didn't come in, right here. There's no zero. But, what it's doing is is
replacing this okay. But, that doesn't work. That's not
really what we need. So it's replacing these first
underscores with zeros. So this is close but it's not
quite correct. We need to do some further investigation.
I wanted to do a number literal conversion
and look into this. Because if we do this as a comparison of a float
instead, what happens? So as a float you still have this issue: where it does that
conversion of that first underscore into a zero. But we need to
then refine that. what i decided to do is first iterate through each one of our
versions here. Each one of those independently and
replace each underscore with a dot so making it into a float.
And only taking the first occurrence of it. The second
or subsequent occurrences of this will be replaced
instead of an underscore with an empty space and that's what you end up with
here. So then now we have actual ordering and it's correct,
all right. You can take this and use it as ordinal data now. Which is what you
want: if you're not considering this just as a plain label.
Now i would like to say further work what can be done with this:
would be looking into memory management if you have a larger data set for label
encoding. Because, there's tricks or different
types of label encoding you can do to make sparse matrices,
to cut down on the amount of space you're using if you had
hundreds or even thousands of variables to do comparisons against.
Which starts making an explosion of your matrix if you had one
and slows down the computational time of reading in
and evaluating everything. Now i would like to say that:
i took in these data and i looked at how these are reading in and comparing
it to its actual label. So if you notice; here's all of the
labels on the first or the zero index of each tuple.
Here's its actual value that i converted it to. And you notice that it's correct
if you're comparing it here to the label number that's printed off corresponding
to its actual value. You could see that you can
actually label and code this correctly now. Lastly, i put
everything together so you can see what it looks like from
your version of label encoding and as a final installment versus
the label encoding version of the string literal. And i also concatenated the
actual float values that we have so you can see what this looks like as a whole.
But, i'd like to say thank you for watching. Please, like share and subscribe.
and if you subscribe turn on that notification bell. As always i hope this
brought utility to someone. Thank you for watching
see you in the next video. Bye