hello and welcome back to my Q&A video
series about the pandas library in
Python and the question for today is how
do I create dummy variables in pandas
okay
so if you work in machine learning or
statistics or econometrics you might be
familiar with the concept of dummy
variables also known as indicator
variables and as an example let's
pretend you have a column in your data
set called sex that's either male or
female an example of dummy coding or
dummy encoding would be to mark a female
as zero and male as one okay now this
process is also used to include
unordered categorical features in
machine learning models which is why I'm
most familiar with it okay so if you're
interested in doing that how do we do it
in pandas alright so as always I'll pull
up an example dataset we can work with
so we'll import pandas as PD and the
data set I'm going to use is the
training data set from Carol's Titanic
competition and if you're not familiar
with Kegel I've got a previous video
that discusses that but we're gonna say
train equals PD dot read CSV and it's a
bit dot Li slash Kegel train okay and
let's look at the head so these are
passengers that were aboard the Titanic
and some information about each of them
and so let's pretend we want to create a
dummy variable for the sex column shown
right here so it's currently specified
as an object column because it is
strings okay so one way we can do it is
to use the series map
method now I'm gonna create a new column
in the data frame called um I'll call it
sex underscore male equals train dots X
dot map and with the map method I'm
gonna pass it a dictionary and I'm gonna
say I want female to map to 0 and male
to map to 1 okay so that's how I specify
this and I'm making a new column called
sex male and let's take a look at the
result okay so here's our new column on
the right and I don't I can scroll over
a bit here so you can see it and sex
male it is a 1 if they are male and a 0
otherwise okay so that's how we've dummy
encoded this particular this particular
column and I called it's X underscore
male because that implies even if you
didn't know what it originally looked
like you'd probably assume 1 mean
because I called it sex male 1 means
male 0 means female that's kind of the
convention okay all right
so I'm gonna show you that's one way to
do it but I'm gonna show you a way that
is more flexible it is a bit more work
but it's more flexible and I think
you'll like it
so the next way I'm gonna do it is I'm
gonna say I'm gonna use the top level
function called PD get dummies and I'm
gonna pass it the series train dot sex
okay and what it does is create one
column for every possible value okay so
there are two possible values for sex in
this data set so it created one column
for each and then for each row it tells
you whether it
was female or male by putting a 1 in the
appropriate column okay
now generally speaking if you have K
possible values for a categorical
variable you use K minus 1 dummy
variables to represent it
in other words we have two possible
values for sex so we only use one dummy
variable to represent it because that
will capture all the possible
information about the feature okay so we
want to drop in other words we want to
drop the first column because we don't
need it
and that will define it as what's known
as the baseline okay so the way I'm
gonna do that is I'm going to use the
eye lock method we've seen a couple
times so I'm gonna say dot I'll oak and
then we say bracket and the question is
what rows do I want I want all rows
that's : and what columns do I want I
want columns starting in position 1
through the end okay
and now I've dropped my first column the
column in position zero and defined
female as the base line okay um now
there's one more thing I want to do I
like to because I'm gonna add this
column back to the original data frame I
like to identify where it came from
um and you'll see why this is relevant
in a little bit but I'm gonna use this
cool prefix argument so I'm going to say
prefix equals sex and when I do that it
it adds the prefix and then it adds an
underscore you can you can use a
different character if you like there's
a way to specify it but by default it
puts an underscore and now we can see
that with this get dummy's I've
duplicated the same thing
we did with the the map method
okay so we've achieved the same result
as above just using git dummies now this
seems like more work but as I said I
want to show you how we can use this and
it's a bit more flexible okay so now
let's try get dummies with a categorical
variable that has more than two possible
values for instance there is embarked so
trained embarked dot value counts to
check it's different values and we can
see there are three possible values of
train dot embarked okay
so let's use get dummies with that
so we'll say PD get dummies train
embarked prefix equals embarked okay and
as we can see only it sets up a column
for each of the possible values it gives
there's only a single one in ever in
every row so we know the first row is an
S the second Rose is C then it's s SS q
s and so on and if you look at the data
frame you'll see it matches what we were
expecting there now as before if you
have three possible values you need
three minus one or two possible dummy
variables to capture all the information
about that feature so just like before
we're gonna say dot I lope all rows and
columns one through the end okay and now
we've dropped the C column and
established it as the baseline okay so
now we can still identify what the
embarked value is for every row we if we
if it's zero one we know that's an S if
it's zero zero we know its
c01 s01 that's an s10 is aq
so those three combinations 0 1 0 0 and
1 0 are the C the Q and the ass okay
now you might be thinking well I've
still got to attach this to the data
frame how do we do that so what I'm
gonna do is I'm gonna say I'm gonna
creep I'm gonna save this embark'd
dummies equals I'm gonna save it like
that and then I'm gonna use another top
level function called concat PD concat
and the way contact works is you can
concatenate columns side by side or rows
on top of one another so in this case
I'm gonna concatenate train and embarked
dummies and if you want to concatenate
rows you use axis equals 0 columns axis
equals 1 so we're gonna say axis equals
1 and we're gonna overwrite the original
train and then let's take a look at it
okay so let me scroll over and we can
see that we've now got this embark'd
queue and embarked s columns on the
right added to the original data frame
okay so as always I'm going to end with
a bonus and to do this bonus I'm gonna
just reset the data frame okay to how it
is when I started it so let's just rerun
it and rerun head okay and the trick I
want to show you is how to pass a data
frame to PD get dummies okay so up to
now we've been passing a series to PD
get dummies but we're gonna pass a data
frame so let's say PD get dummies
and we're gonna pass it train okay and
we're gonna use the argument columns
equals and we pass columns a list and we
tell it which columns do I want to dummy
sex and embarked for example okay and
when we do that super cool
it drops check this out
you'll notice the original sex column
and the original embarked column are
gone and they have been replaced with
the dummy columns sex female sex male
embarked C embarked Q embarked s okay
but you might be thinking we'll wait a
minute I can't use this I'll oak trick
to drop like sex female and embarked see
how am I gonna drop those well you could
use like the drop method but what if you
had specified 40 different columns that
you were dummying okay it'd be a lot of
manual manual work to drop those exact
columns well starting in pandas version
0.18 which released in March 2016
there's a new parameter called drop
first and all we do is set it to true
and check this out it has now dropped
the female level and it has dropped
embarked C so we've accomplished exactly
what we were going for without using AI
Lok and without using contact and got it
done in one line so once you've got
pandas 0.18 you can do this methodology
too okay so that is it for today thank
you so much for joining me as always
please click Subscribe if you'd like to
see more videos like this feel free to
leave me a comment or a question below
and I'd love to hear from you but that
is it so thanks again and I hope to see
you again
soon