what is going on everybody and welcome
back to another data science with Python
and pandas tutorial video in this video
we are gonna be kind of building on
everything up to this point we're going
to continue with the minimum wage data
set for one more video this one and kind
of combine two other data sets just to
see more about this whole relationships
thing and learn a few more cool features
of pandas before I finalize this little
mini series with doing machine learning
basically a machine learning workflow
with pandas in the next video with that
let's get into it so we're to be using
two new data sets here we're gonna be
using the 2016 presidential election
vote by county as well as unemployment
by county so pretty cool so it'd be nice
to combine all of these data sets into
one major data set and see if voting
proclivities the unemployment rate and
the minimum wage have any relationships
with any with each other right cool so
that's the idea and really we're just
going to see if there's any relationship
with it's pretty hard to based on that
data alone draw any real conclusions
we're just doing it to see to learn more
about pandas really really so if you
find yourself offended by the results of
this video not my problem
so anyways here we go import import it
import pandas as PD and then we're gonna
bring in unemployment County as PD dot
read underscore or CSV and that is data
sets slash and this one is unemployment
by county - Us / output dot CSV so this
one came with its own little directory
basically and then we can do on M County
dot head okay and there we get a rate
that's our unemployment rate and then
it's by county
and then you're so cool so now we want
to do is combine this plausibly with our
minimum wage so to do that we're just
gonna come over here and basically take
all of this data copy
come over here paste boom awesome okay
so now we want to do is we still have
that missing data so let's go ahead and
fix that real quick and in fact can I
just do this yes I sure can
no wait I just want to do this I don't
want the core I just want to drop it so
instead we'll say ax min wage equals ax
min wage dropping in and then act waged
on and fix that capital W okay great so
now what do we want to do so we want to
take the minimum wage and we want to
kind of add that to a new column in our
unemployment County data set so the way
I think I want to do that is by creating
a new column and then mapping the values
I there's probably a better way than
this but it gives me the opportunity to
show you guys at the most basic level a
method that will always work for
creating new column values in pandas
this was not the most efficient and in
fact it's going to probably take a
moment probably a minute or two to run
but I'm gonna show it anyways because it
always works and then we'll talk about a
couple will show a couple other examples
that are gonna be like way faster but
anyway yet so let's do define get get
get wage and this is a function that's
going to take two parameters it's gonna
take a year and by state
so recall minimum wages by state but our
unemployment data is by
County our voting data is also by county
so unfortunately camp by county the
minimum wage actually does kind of defer
so right away we've got a slight
violation here but that's okay the show
must go on and I'm really just trying to
show you guys an example anyways so
whatever but sure that's that's a
problem we just I don't have a minimum
wage data by county so I'm sure there's
a file out there that exists feel free
to use that if you're if you're trying
to make an actual report here just take
note that this isn't quite fair anyway
what we want to do
we'll do a try and then accept and if we
hit an exception we're gonna return n P
dot nan also have we do we we did import
it right because we use nan right using
in here wait everybody just relax a
second where when did we we never
imported numpy not on this one
what okay well you're gonna need numpy
[Applause]
mmm twilight zone okay whatever
yeah okay I'm still okay whatever here
we go um Road term act min wage because
we just want to get the value right so
we're trying to get a minimum wage by
year and state so return Axman wage and
we can't just say year's state because
that action act min wage is indexed by
year so actually we're going to do dot
Lok and then we grab the year so dot Lok
whoops dot Lok year okay and then Oh
actually so it would be here
but no we're gonna say that Lok year
okay there's a reference to the index
column and then we want to get that
specific state so that's a function that
will do that we can quickly
that function by doing something like
this I get min wage 2012 and Colorado
and in fact 2012 shouldn't be a string
cool so that's eight point three three
probably impossible to see but hopefully
on your screen you see it okay so now
we've got that now we're going to map
this function to a new column in pandas
and the way that we can do that is just
like a regular mapping function in
Python in general so we can map a
function to a list or array in Python
using map and this is like I said before
not going to be the most efficient way
but this is a way that will always work
especially when you have things like
functions with multiple parameters or
you have a fun like you want it you're
not just mapping like one value to
another place maybe you want to run some
sort of true function or calculation or
some sort of algorithm right you need
something more advanced this is your
this is your guy okay so this is how
you're gonna do it so first of all I'm
gonna throw in time just to measure time
here that is a ipython notebook specific
command so don't do that if you're you
know writing PyCharm or something I
don't even know that might work in by
German but definitely got an idol in
sublime text okay so we are gonna map a
new column so it's uh NAMP County and
we're gonna say min wage or min wage and
that is going to be equal to I think I'm
just gonna write map for now we have to
encase this as well but we're gonna say
we want to map get min wage we're gonna
map that function and then you just pass
all the parameters of that function so
year
what year are we gonna map on M County
year and then what's the state well same
thing right on on m p-- county
State and actually year is a capital y
year okay so we map that unfortunately I
don't think this will work I'm gonna run
it just because I'm on fresh pandas and
all that and I'm curious if it does did
that work
everybody relax a second um I have to
check this now on unint
County I didn't think that was gonna
work because as I think like python it
used to work in Python 2 and then with
Python 3 you actually had to do list map
no way that worked no no you get it
that's hilarious
that's adorable okay so yeah we do have
two Calicut so in Python 2 that actually
worked you didn't get a map object so
actually we have to convert this to a
list oh that was so exciting so fast
okay so it actually has to be calculated
here we go and then whenever that is
done I just want it to it we could also
see when it gets a number but I'm just
gonna do nm County dots head here cool
okay so again all that's doing is we're
just we're converting it to a list at
the very end because that you know with
with pandas you you you can just say a
new column is a list of things and it
doesn't matter how you generated that
list it doesn't have to be via a thon or
well it doesn't have to be violin
initially I guess but doesn't have to be
if I am pandas okay but yeah so this is
gonna be relatively slow if you can do a
dot map that's great otherwise a dot
apply or something like that
there are some other options for us and
I'll kind of I'll show an option anyway
down the line but once that's done cool
we are good to go so the first thing
that I would do once that is done is I
would just check to see on F County dot
and in fact let's just do this on empty
rate and then
min wage whoops did that capital W again
so don't forget rate is the unemployment
rate we price you to change the name at
some point but that's okay and then this
is minimum wage and the question is is
there a relationship there do they Co
relate together and then the next
question would be pretty much the exact
same thing except to see do they vary
together so covariance and we are still
waiting on that calculation are oh my
gosh it's taking forever
so probably while we wait on that we'll
just load in the presidential data while
we wait and then if someone knows a
quicker way to do multiplied cup to a
quicker way to map a function with
multiple parameters so I can't just have
one parameter cuz I know yes there are
quicker ways if it just only had one but
what about when it has two parameters I
don't know the cool thing is though that
it just knows to map you know by row
like it it's still pretty magical even
though it takes a lot of time okay but
anyway we are done onam county had done
and then here we can see is there any
correlation so it doesn't look like
there is much of a true relationship but
it does look like they actually do very
together pretty significantly so that's
interesting I guess that would probably
suggest that there's other factors at
play and those other factors have a high
correlation to these things but these
things between each other don't that's
kind of how I would read into it I could
be totally wrong I'm just here to teach
you guys pandas not to actually uh find
true answers so anyway let's load in the
presidential data now so let's say Prez
16 equals PT dot read CSV and this would
be datasets slash Prez 16 results dot
CSV now Prez 16 head
and what I get here is County Phipps I'm
not sure what Phipps is candidate st
which I believe is state but I don't
know for sure what s T stands for
percent report that's probably how many
places reported I don't know
votes that's I guess total Ville oh this
is how many votes for that person this
is how many votes in total were made
yeah and then this is what percentage
for what which candidate so as you can
see so for each for each county and each
candidate we get all this data but to
keep things simple let's just do Donald
Trump and then the value we're gonna
track is the percent so for each county
that we find our question will be two so
how many people in that county voted for
Donald Trump and we'd like to check
correlation and covariance between the
unemployment rate and the minimum wage
in their relationship with the people
voting for Trump okay so let's do that
so how do we do that because now we
actually need to combine all this data
on county and state and then share the
percentage vote so that's that sounds
like a really complex thing that's
really hard to jumble in your head to
how you might do that but it's not too
too too hard curiously I think I think
I'll just continue on this guy but okay
so so what we want to do now is let's
just grab because our our unemployment
data on amp on amp but I'm sorry
unemployment county we're gonna call it
2015 in a second item County let's just
do a Len of on amp
County right that's a very big data
frame so I think what I'd like to do is
just get the latest unemployment County
data which I believe is February 2015
and
yeah so we can make that much smaller so
I will do the following so we have
unemployment county so just just to make
sure because we're we're gonna be
dealing with so many data sets here it's
gonna be easy to get them mixed up right
so we have a year and month so now we
want to filter this data frame down to
just 2015 February okay
so how do we do that so the way that
we're gonna do that is similar to what
we have before but we need like to two
things so we're now what we're gonna say
is County underscore 2015 is equal to 1
M underscore County and then this is
probably long summers actually I think
mi zoomed out no I wasn't anyway this is
probably long so I'll probably mount but
for now unab County mayor and then we
have two clauses basically so we're
gonna say Clause number one and you know
clause number two so under what
scenarios do we want County 2015 well we
want it to be the case where this and
then we're gonna say year is 2015 and
then we also like I'm just gonna copy
this whole thing that and then we're
gonna say month equals February February
awesome now that we've done that ooh I
don't know if I accidentally just ran
that twice I shouldn't know that
shouldn't matter because we're creating
a copy when we like do mappings though
it really will screw things up
especially if you replace the current
column name so we've got to be pretty
careful about running things I sent
multiple times sometimes anyway cool um
Mississippi nan didn't we drop the names
okay I thought we dropped Nan's though
like a way up here right didn't we
or is that ray min wage no I mean wait
Han
maybe I guess because we have a maybe
because we had the unemployment rate but
not a minimum wage value for that
specific County I'm not really sure well
we're gonna keep moving on and see if we
hit any further issues probably
something to dig into though maybe but I
got so much I want to show so far so
we'll continue for now cool
is that okay anybody who disagrees speak
up if I don't hear anything we'll just
continue so so what I'm gonna do now is
we need to so our presidential dataset
has like if we do state let's say let's
say pres 16 st dot unique okay so these
are our actual states but they are the
two-letter now we know postal code right
so we need to convert this state to
being the postal code rather than the
full state name so how could we do that
well turns out up to this point we
actually already did that right we've
we've already got the postal codes and
the state we've got a dictionary we can
just map it so let's do that so what
we're gonna say is state AB it was state
abbreviation for now PT dot read CSV and
we saved it because we're smart cookies
but we did it and to be honest with you
as I was making this series I did not
plan that
that's just it just came in yeah it just
was super convenient state AB dot CSV
and we will read in
let me just print out state ab dot head
real quick and let's just get this the
way we want it so first of all we're
going to say index underscore call
equals zero greats and then we only care
about the postal code so state AB equals
state AB don't forget double brackets
there before you get screwed and it's a
series it'll be horrible okay great so
cool so now what we want to do is
convert that to a dictionary so we're
Alabama maps to Al and so on so to do
that we're just gonna say state AB Dix
equal state AB I think first we would
say dot to underscore dict
and then let me just print that real
quick let's just do state AB dict right
so then we'll just say we convert it to
dict and then we'll say postal code boom
now we've got the dictionary that we can
now map to the column so we can actually
just rename that column just right in
place so we can say County 2015 state is
equal to County 2015 state dot map and
then we can actually just map the
dictionary and it's just beautiful and
magical and it works so we map that and
this would be something you would not
want to run twice yeah
County twenty-fifth where do we define
County 20 ok here so I could run that
again right so the reason why we're
seeing that is because of County 2015 so
we could lay it we could have actually
said unemployment county dot and we
could run it copy there and that should
fix our error
I'm too afraid though I just wanna have
to run everything again but yeah that's
probably what we would need to do is
throw it let's just try that really
quick dot copy so we throw in the copy
and then let's run all of this again
cool okay so now hey man so let's just
do County 2015 diet tale okay great
so we have everything there that we need
so everything's been converted yeah so
now what we want to do is let's check so
print
Len County 2015 and now let's check the
length of pres 16 pres 16 okay so press
16 is longer so let's map that to 2015
now let's do pres 16 dot columns real
quick whoops so we have a couple of
issues here we have we want to map these
things together so we're gonna merge
them together or join base merge and
join on the back end of pandas are
actually the same code it's just the
default is different but you can you
actually do the exact same operation
with either one so yeah so you can
pretty much interchange such things but
so the first thing to note is how do we
want to merge these together well we
would merge them based on the year in
the state so or I'm sorry not the year
in the state the year is the same
because everything is only one year so
this is both of these or one point in
time so we would act we want to merge
them together on state and County those
are the two different things
so the first thing that we want to do is
make sure state and county have either
identical between these two data sets
and right now they're not one is st and
then the other is county but it's laura
kc whereas here it's capital s in full
states and then county so it fix those
two things first so first we're gonna
say and in fact let me make a little
more space so first we're gonna say pres
16.3 name and then we will rename the
following columns and we're going to do
a lowercase C County gets RIM renamed to
uppercase C County simple and then we
want to do state as well so st maps to
capital S state and then we're gonna do
this in place equals true pres 1610
great so state Suman County here is
correct it's just this is this state is
us and that's totally fine no big deal
so now to join these two things together
what I would like to do is join them and
we can join them on an index that's the
easiest way so for both of these we're
just gonna set the county and state as a
double index basically so you'd have an
unlimited number of indexes all right so
that's what we're gonna do first so
we're gonna do the same thing for both
of these so we're gonna say for DF in
and then we're just gonna list them cuz
we're gonna run this exact same code
with both anyways so County 2015 2015
and then pres 16 what do we want to do
well we're gonna say DF dots DF just
sets underscore index and we are going
to set a dual index we're going to say
County County come on ant whoops whole
it unscrew anything up state and then in
place equals true cool now what we want
to say is we still haven't filtered pres
16 by so like if we say
Prez 16 dot head again we still have all
of these so we got Donald Trump Hillary
Clinton Gary Johnson and so on and right
now I just want to grab Donald Trump
later maybe you'd want to add in all
these other ones and just see like maybe
like Gary Johnson's libertarian so his
relationship to like minimum wage and
people who vote for a Gary Johnson might
be significantly different because like
he would be probably a proponent of no
minimum wage right and so anyway maybe
you'd want all of these candidates but
for now let's focus purely on Donald
Trump and so that is probably that will
serve for the biggest divide between
people you know generally between
Hillary Clinton and Donald Trump was
where the majority of the votes went so
anyway press 16 to head so the next
thing we want to say is let's just say
pres pres
is going to be equal to pres 16 where
pres 16 cand is equal to Donald Trump
Poe nice Donald Trump and then pres and
then like the only thing we care about
at that point is we don't care about any
of these other columns except for the
percent column right so the next thing
we're going to say is pres 16 equals
pres 16 and then again don't forget to
make this a true data frame we are
curious only of the percent column then
pres 16 drop an A in place equals true
just to get rid of any mess and then
pres 16 dot head when we're done
hopefully no errors great okay so now
we're ready to merge these things
together so to do that we'll make
another data frame all together together
that's going to be equal to County 2015
dot merge and we are going to merge
Prez 16 and then we're gonna merge on
equals and we'll do County and state you
know state and then all together drop an
A in place equals true and let's just do
that real quick see where we stand all
together or not head let's drop that
year column so all together dot drop and
we are gonna drop here all axes equals 1
because we're trying to drop columns not
rows and also in placing the screw in
place equals true and then again all
together dot head I can't type together
very fast okay now let's grab the
correlation and covariance so all
together core and then all together dot
covariance okay so it appears that that
these things like let's say the
percentage vote is like with minimum
wage for example it appears the
percentage vote is negatively correlated
with minimum wage so the more people
let's say as percentage goes up that
voted for Trump minimum wage goes down
or vice versa so the higher the minimum
wage is less like we were to vote for
Trump that seems to be the relationship
which kind of makes sense like higher
minimum wages are a more liberal type
issue so that would make sense that
those things are co-related
but curiously enough they don't appear
to two very together so I'm not really
sure how to reach read into that to be
honest with you otherwise like the
unemployment rate doesn't appear to be
too covary with who we voted for and
then also
doesn't a port appear really to
correlate either so interesting
it appears that politics are way more
complicated than we thought I guess it's
not just simple hey anyway I think
that's enough for pandas we've done a
lot of stuff with pandas again there are
so many things that we can actually do
with pandas that it would take me
forever to show you guys like all of the
things that you could plausibly do like
for example we mostly just did stuff
with CS fees we don't really read from
databases or anything like that and use
other data files and there's there
really is like so many things that we
can do but for the most part I tend to
find myself reading and data kind of
reshaping that data in some way doing
basic analysis like describes and then
the next tutorial I'm going to show you
guys is the machine learning stuff so
how do you go from a data frame and feed
that through an ml model that's what I'm
gonna be showing you guys in the next
tutorial with a different data set we're
gonna be leaving politics behind and
yeah so questions comments concerns
whatever feel free to leave them below
again any of the findings that we found
here are really not statistically
significant even though really the
finding was we didn't find anything so
so anyway that's all for now yeah
questions concerns leaving below discord
GG slash Centex I will see you guys in
the next video