one hot label Target encoding yeah
yeah stack Quest
hello I'm Josh starmer and welcome to
statquest today we're going to talk
about one hot label and Target encoding
and they're going to be clearly
explained you don't have to worry about
the details of scaling your stuff up in
the cloud cause lightning will take care
of it for you bam
this stat Quest is also brought to you
by the letters a b and c a always b b c
curious always be curious
imagine we had this data and we wanted
to use favorite color and height to
predict if someone loves Troll 2 which
is a movie that some people love and
some people don't
in this case favorite color has three
discrete values blue red and green
now in theory discrete features like
favorite color are fine for most machine
learning algorithms but in practice a
lot of popular machine learning
algorithms including neural networks do
not work well with them
as a result discrete data are often
converted into numerical values before
being used for machine learning
one popular method for converting
discrete variables or features into
numbers is to use something called one
hot encoding
when we have three or more options for a
discrete variable and in the case of
favorite color we have three options
we start by creating a new column for
each option
in this case that means creating three
new columns blue red and green
now in the blue column we set the value
to 1 if we had blue in the original
favorite color column and we set the
remaining values to zero
likewise for the red column we set the
value to 1 the one time we had read in
the original favorite color column and
we set the remaining values to zero
lastly for the green column we set the
value to 1 if we had green in the
original favorite color column and we
set the remaining values to zero
note the last column loves Troll 2 is
also discrete but it only has two
options yes and no so we simply replace
yes with one and no with zero
and now all of the columns in our new
data set are numeric and can be used
with algorithms that don't do well with
discrete data like neural networks or XG
boost
bam
using one hot encoding to convert
discrete data into numeric data works
fine when we don't have too many options
in this case we only have three options
for favored color so we replace favorite
color with three new columns
but when we have a lot of options for
example if we had a column of postal
codes and there are 41
683 postal codes in the United States
then we would end up replacing the one
postal code column with 41
683 new columns which might make the
data difficult to work with
so when we have tons of options for a
discrete variable one alternative to one
hot encoding is to Simply assign numbers
from low to high to each option
so in this case we might set blue to
zero red to one and green to two
and just like before we could convert
Love's Troll 2 to be numeric by setting
yes to one and no to zero
simply converting the discrete values to
random numbers like what we did here is
called label encoding and again just
like before all of the columns are now
numeric and we can run the data through
a neural network double bam
note one thing that people don't like
about using label encoding is that the
numbers we use are just arbitrary and
some machine learning algorithms will
treat the order of the numbers as if
they might mean something and that can
cause problems
for example a decision tree splitting on
favored color would be forced to group
red and green together or blue and red
together
simply because of the random numbers we
assign to each color
so instead of just picking random
numbers to represent the options blue
red and green
we can calculate the mean value of the
target the thing we want to predict
which in this case is loves Troll 2 for
each option
for example of the three people that
like the color blue only one of them
loves troll 2.
so the mean value for blue is 1 divided
by 3 or 0.33
so we replace blue with 0.33
likewise because only one person likes
red and they do not love Troll 2
the mean for red is zero so we replace
red with zero
lastly because two of three people who
like green also love Troll 2 we replace
green with 0.67
because we use the target the thing we
want to predict to determine what values
to replace the discrete options this
method is called Target encoding
that being said we've only talked about
the simplest type of Target encoding
a more commonly used version of Target
encoding deals with the fact that we
only had one person who liked the color
red
and that means we only used one person
to determine the mean value for red and
thus we don't have a lot of data
supporting the use of zero to replace
red
in contrast both blue and green have
more data three people each supporting
the values we use to replace them
because less data supports the value we
replaced red with we have less
confidence that we replaced red with the
best value than we have for blue and
green
so in order to deal with this target
encoding usually is done using a
weighted mean that combines the mean for
a specific option like red with the
overall mean of the target which is
Love's troll 2.
for example in order to use the fancier
Target encoding with our data we start
by plugging in the mean of the target
for blue 1 divided by three
then because three people were used to
calculate the mean for blue we plug in 3
for n
then we plug in the overall main for the
Target loves troll 2. 3 divided by 7
because overall three of the seven
people love troll 2.
now we just need to pick a value for M
the weight for the overall mean
m is a user-defined parameter or hyper
parameter and in this example we set m
equal to 2. setting m equal to 2 means
we need at least three rows of data
before the option mean the mean we
calculated for blue becomes more
important than the overall mean
now we just do the math and get 0.37
so we plug in 0.37 for blue
now we calculate the weighted mean for
red beep beep boop
and we get 0.29 so we plug in 0.29 for
red lastly we calculate the weighted
mean for green beep
[Music]
and we get 0.57 so we plug in 0.57 for
green bam
now let's compare the target encoding
when we use the weighted mean to the
Target encoding without the weighted
mean
the target encoding for blue and green
are similar to what they were before
and this makes sense because we had a
relatively large amount of data for both
blue and green
in contrast with the weighted mean the
value for red is much closer to the
overall mean than before
and this also makes sense because we
have so little data for red only one row
in a way we can think of the overall
mean as our best guess given no data
however as we get more data more rows
for each option we use the data more
rather than our best guess to determine
the target encoding
note if you're familiar with Bayesian
methods this approach may look familiar
because a lot of Bayesian methods boil
down to calculating a weighted average
between a guess and the data as a result
some people call this Bayesian mean
encoding triple bam
note some of you may have noticed that
we are using the target the thing we
want to predict to modify the values in
favored color
and doing this sort of thing is a data
science no no
that we call data leakage
data leakage results in models that work
great with training data but not so well
with testing data in other words data
leakage results in models that are over
fit
the good news is that there are a bunch
of relatively simple ways to avoid data
leakage or at least reduce the amount of
data leakage so that you can use Target
encoding without overfitting your model
one of the most popular methods to
reduce leakage is called k-fold Target
encoding so let's go back to the
original data set that had blue red and
green categories for favored color
and talk about k-fold targeting coding
note the word fold in k-fold Target
encoding refers to splitting the data
into equal sized subsets and the K
refers to how many subsets we create
for example if we did two-fold Target
encoding then we would divide the data
into two equal sized subsets
note because we have an uneven number of
rows we just made the subsets as similar
in size as possible
now to make it easier to keep track of
things let's label the first subset a
and the second subset B
now to Target in code blue in subset a
we ignore the target values in this
subset
in other words we ignore the values for
Love's Troll 2 in this subset
and instead plug the target values from
subset B into the weighted mean equation
we start by plugging in the subset b
mean of the target for blue zero divided
by one because the one person in subset
B that likes blue does not love troll 2.
then because there is only one person in
subset B that likes blue we plug in 1
for n
then we plug in the overall mean for the
Target in subset b 1 divided by 3
because overall one of the three people
in subset B loves troll 2. and just like
we did before we'll set m equal to 2.
now we just do the math and get 0.22
so we plug in 0.22 for the two rows and
subset a with blue
now we need to Target and code the one
row with blue in subset B so we ignore
the target values in this subset and
instead plug the target values from
subset a into the equation for the
weighted mean
[Music]
then we do the math and get 0.5
so we plug in 0.5 for blue but only in
subset B
note you may have noticed that the
different subsets have different values
for blue this is okay because favorite
color is becoming a continuous variable
just like height
and now let's encode the color red in
subset a
so we ignore the target values in subset
a
and instead plug the target values from
subset B into the equation for the
weighted mean
now because subset B doesn't have anyone
who likes the color red the mean for red
is zero and n equals zero
the other values are the same as before
and we end up replacing red in subset a
with 0.33
likewise Green in subset a uses the
target values in subset B and turns into
0.42
and green in subset B uses the target
values from subset a and turns into 0.67
now that each color and each subset has
been encoded we merge the subsets back
together and we're done note this
process reduces data leakage because the
rows do not use their own Target values
to calculate their encoding bam
now going back to the original data with
blue red and green
if we set k equal to 7 then we would
divide the data into seven subsets
now Target encoding the first subset
which consists of a single row with
favorite color equal to Blue
means we ignore its Target value
and use the target values from all of
the other subsets to calculate the
weighted mean
likewise encoding the other subsets
would use all of the other Target values
except their own
note when we use all of the target
values except 1 to do the encoding it's
called leave one out Target encoding a
quick scan of the internet shows that
some people are successful with leave
one out Target encoding and other people
are successful with setting k equal to
five
triple bam
now it's time for some Shameless
self-promotion
if you want to review statistics and
machine learning offline check out the
statquest PDF study guides in my book
The stackquest Illustrated guide to
machine learning at stackwest.org
there's something for everyone
hooray we've made it to the end of
another exciting stat Quest if you like
this stat Quest and want to see more
please subscribe and if you want to
support stack Quest consider
contributing to my patreon campaign
becoming a channel member buying one or
two of my original songs or a t-shirt or
a hoodie or just donate the links are in
the description below alright until next
time Quest on