what is going on everybody and welcome
to yet another installment of the data
analysis and data science tutorials with
Python and pandas in this tutorial we
are going to be working with a new data
set the u.s. minimum wage by state from
1968 to 2017 so this is the least amount
of money an employer can pay an employee
II organized by state and it has high
and low value both in that x terms or
amount and then also calculate it out
using the CPI which is like the cost of
living let's say for a really quick term
and so they've got it in value in like
2018 value numbers so we'll probably be
working with that we'll work on the true
low so the actual lowest minimum wage by
state or something like that just to
simplify things so anyway this is the
data set we're gonna use so go ahead and
download that extract that and I'll have
that in the datasets directory so coming
over here let's go ahead and read it in
so import pandas as P D and D F equals P
d dot read CSV data sets slash minimum
wage data dot CSV okay so let's go ahead
and read that in and boom we get this
nasty nasty error it's a some sort of
unicode decode error and we get blah
blah blah so basically what we're seeing
here is pandas wants to by default use
utf-8 encoding but for some reason we
have an encoding issue now I have not
confirmed this but I'm gonna guess if we
look at table data and we come over here
and look at where is it yes so table
data is the scraped unclean data from
the US Department of Labor so that's
where this data is coming from so I'm
going to guess that it has some sort of
issue there which is odd because when
you scrape you're getting data in utf-8
right so I don't know anyway we have an
encoding issue so
the first thing that I would always do
is like I would by default encoding is
almost certain to be utf-8 but if you
hit an issue like this the next best
thing to try is gonna be Latin encoding
let's try that and sure enough that
works so that's what we'll go with
for now but one thing we can do is just
go ahead and save this so we don't have
to deal with that ever again and say df2
csv and let's save that as dataset min
wage csv with the encoding of you
utf-8 okay cool
so that's just one a quick example of
why we might use pandas to convert
things so next what we're gonna do is DF
equals PD reads CSV and we will just
read in this CSV so do you have let's
just do will read that in DF not head
cool okay so this is the data that we're
working with and what we're interested
in mostly is probably this low column
now basically the objective here at
least an interesting thing would be to
almost do the same thing that we did in
the last tutorial but I'd like to see
some sort of correlation so this is by
state over time a value that's changing
much like we had with avocados and in
fact usually carry on this tutorial with
the avocado data set up to this point at
least so where our objective is going to
be kind of the same thing it's gonna be
okay we've got these columns of values
or this column that contains a bunch of
values and actually we want to make that
all the top now what we did in the last
tutorial is totally fine but I'm gonna
show you guys a new way to do that the
pandas way so so let's see how that that
works so I'm gonna say GB for grouped by
equals DF group by and then
we were we're gonna group by state so
I'm just gonna say state there and then
what we'll do is we can say well we can
do a couple of things so this is going
to create this like group I object so
one thing we can say is GB dot get group
and then we can get a very specific
group so let's just say like Alabama for
the just the first state that we see
here get group Alabama dot set index and
we'll set the index to be the year and
then we'll just print out the head okay
so that's one way we can just grab a
group but the other thing is we can
actually just iterate over the groups so
for example we could do something very
similar to what we did before but do it
in less code basically so we're calling
this actual minimum wage equals PD dot
data frames so that's going to be the
same and we'll do that as well and then
we're going to say is for name group in
DF group by state we could also have
just saved the GB above but I'll just do
this for the sake of clarity of exactly
what's going on here
so this that's how you will iterate so
you'll get named because the thing we
grouped by it'll be that specific name
and then you'll get group and that will
be basically your data frame so for name
group ok so then all we have to ask at
this point is if act min wage empty
we're gonna do something pretty similar
but we're just gonna say act min wage
equals group there we go group dot sets
index and we'll set the index as year
and then again we want that to be a data
frame not a series so we're gonna say lo
dot 2018 and then we're gonna throw in a
dot rename and then this is how you can
rename columns so I'm gonna say rename
and then we're gonna say columns equals
I was hoping it would look a little
better as we went off the screen that's
okay and then that would be a dictionary
and inside the dictionary you're gonna
put the original name and then the new
name you want
have so the original name is lo dot 2018
that's becoming a challenge to see it's
unfortunate anyway hello Eco load of 20
and then what we want to change that
name to is whatever the name is so it's
just the state okay so now we've done
the rename let me make sure dot rename
columns so I need to close off our name
that's really hard this doesn't run off
the screen as well as one would hope
anyway that if I did this can we see it
yeah okay so I'll just zoom out for now
group set index year 2018 rename cool
okay so that's if it's empty otherwise
what we would want to say I'm gonna copy
that line and then we're just gonna say
else ax min wage equals ax men wage dot
join in that exact line okay hopefully
it should have another is it that it is
there okay we just still can't totally
see it but anyway let me zoom out even
further just make sure make sure you
fully closed all that off okay so now
we'll come on down here and a clips ax
min wage dot head okay now let me clean
that up actually do that now I'll zoom
back in and hopefully we don't have any
super long lines anymore
that makes things hard so okay so here
we have everything is the actual you
know the low data organized by state and
then year which is pretty cool so kind
of the same thing we did in the avocado
just using group I rather than using
Python logic again
and you can always use your Python logic
to do things but chances are in pandas
there's a built-in way because something
doing something like this is a super
common task this isn't like you're not
the first one to need to do this so it
probably exists all right so now what
are some other things that we can do
like with this data so one of the cool
things like right out of the gate that
we can do with a new data set is like
Axman wage dot describe and this just
gives us a quick kind of rundown of some
basic stats of our data frame so in this
case we could see things like count this
is just how many rows of information we
have mean this is the average right so
this is the average minimum wage over
the course of all these years in 2018
terms the this is the standard deviation
so we can see which states have varied
very greatly this is the minimum value
and then these are like your percentiles
and then this is the maximum value so
just really quickly we get a lot of
information with that describe that then
maybe later either we could use this and
maybe graph it in some way so you could
just graph the standard deviation for
all the states and then see immediately
okay who's got the biggest standard
deviation or something like that anyway
so that's one cool thing that we can do
to get a lot of information from our
data really quickly one of the other
cool things that we have is correlation
and covariance just instantly just built
in for us so we can do Ackman wage dot
core and I'm gonna do dot head just so
we don't print out like this massive
table so with correlation this would
display all the states by all the states
right so like Alaska will you know this
you know diagonally will be perfect
correlation with each other obviously we
are we do have some not a number data so
we can look into that next and figure
out okay what's going on there but
otherwise we get correlation in what
okay probably gonna finish this tutorial
shortly I'm surprised I didn't lose
power there that was really close
lightning anyway where was I
so we got the instant correlation data
here what what perfect timing for my
Franken mug we're Frankenstein's monster
mug anyways
so so the one thing that is curious to
us has immediately when we see data like
this is like what why are we getting
these Nan's why are these zeros we want
to see like did we make a mistake or is
that just something inherent about our
data so so one thing that we can do is
start to kind of look through our data
and just kind of see like what's going
on here so one thing we can say is D F
dot head and immediately we can see
Alabama has nothing it's got no values
and then even like the table data just
has this like dot dot dot with it so Rob
Lee something's not working and then
what you could do is I wonder if they
put the link yeah they do put the link
here so you can just click on the link
literally and come here and see mmm no
there's definitely dot dots it's not
like some JavaScript that we would have
to click it's just simply not there so
coming back here D F dot head so one
thing that we could say is we could just
check like okay so if we say issue DF
equals DF where
DF let's just say really a lot of these
values could work but we'll just say low
20 18 is equal to 0 so that's our issue
DF right and then issue D F dot head and
we can see okay there's quite a few of
these and then we could even find out
just how many states are problems by
doing issue DF state is it a cat yes
capital dot unique and now we can see
okay these are all the states that's for
whatever reason we just can't get data
on so that's okay like I said I don't
know why that's an ellipses here
it just is I don't know why we're not
getting that data but whatever so coming
on over here I think we'll just have to
move on from that so what we could say
is we could just get rid of that data
like that's it's no good we have no data
so this is gonna be super common in a
lot of data sets where you've got data
but then you've got a lot of missing
data for whatever reason and one thing
we know for sure is all of the Alabama
data is no good and that's the truth for
all of these states all the Florida data
Georgia Illinois so there's no reason
for us to continue with those states in
our data set it just doesn't make any
sense so what we can say instead is we
can import numpy as MP because we're
gonna use NPN Ann and you should have
number I should already be installed if
it's not if you get an error you can pip
install numpy but you you should have
gotten numpy when you install it pandas
so I don't think I need to say that but
just in case okay so now what we want to
do is act min wage and then we're gonna
replace all instances of zero with NP
dot nan so we're just gonna say not a
number I know zero is a number but
actually what's happening is we don't
really have any data at all and whoever
parsed this decided to call that zero
rather than no data I think I don't
actually see zeros being reported here
so yeah so I think that our we're better
off just replacing that with not a
number because we actually don't even
know and it definitely wasn't zero I
don't think surely those states actually
have a minimum wage who knows I could
learn something new I'm pretty sure it
like he wasn't Texas in there yeah okay
so Texas had minimum wage so anyway
moving along okay so we replace it with
not a number and then we can just say
drop an A and then we will say axis
equals 1 so if AK sees is one that will
replace that's like going to do that'll
get rid of columns so if a column
contains not a number it will get
of that if the default is ax is equals
zero which actually means rose so if any
nan is in a row it's going to get rid of
the entire row which obviously we don't
want because then we would lose all our
state data so actually we just want to
get rid of the columns that have Nan's
so axes one so then we could say dot
core dot head and now we just have the
states that actually have minimum wage
data now the next thing that we could do
is we could check for let me think your
well so we don't actually know if all of
these are like all zeros like so for
example maybe in the 1960s Texas had no
minimum wage and then later it got it so
maybe that's why I had Nan's don't
really fully know so one thing we could
say is like for problem in flips in
issue D F and then state unique and then
we're just gonna ask if problem in min
wage core oh did we even diff'ent
I think I just okay yeah so we just
printed that out so what I'm gonna do
instead is just do whoops
this and then we'll say min wage core
equals AK minimum wage okay core dot
columns so if we find that problem we'll
just say print we're missing something
here or something like that but we
definitely showing it because we should
be dropping the entire column if it had
any Nan's
so the other question you could ask is
like how many zeros are in any of those
so like one thing you could ask is like
you could count the number of zeros in
the state column or something like that
and that would probably give you a
better
idea because if it has any Nan's we're
dropping it but anyways we've dropped
them all anything that had a nan we've
dropped later maybe we should check and
see like maybe later on they got a
minimum wage I don't know
anyway cool so I'm trying to decide in
fact we really we could just show that
really quickly so let's do that
so we could say grouped issues equals
issue D F group by and we could group by
state and then we could say like grouped
issues get group ala bama and then dot
head will just print out a couple here
and we get okay so now we can see ok
footnote Nan's also these are all zeros
right because we haven't replaced them
with in p9 yet so one option we have is
because it's been replaced with a zero
is we could actually just some that
entire group right so if you said
something like this if we said this
group tissue get group Alabama and then
we said load 2018 and then we said that
some we get zero right because the the
entire column just adds up to zero so we
never get minimum wage data for Alabama
so then we could do the exact same thing
we could iterate over this and we could
say for state data in grouped issues
[Music]
what do we want to do if data low 2018
dot some does not equal zero point zero
then
we miss something okay
none of those hit so literally all of
them always had no data in them okay so
it's just a quick way that we could
actually check that so okay I think I'm
going to stop it here and in the next
tutorial we're going to talk about
visualizing this correlation in like a
big correlation graph which is gonna end
up sending us down and another rabbit
hole entirely oh but it'll be fun so
anyways that's all for now questions
comments whatever feel free to leave
them below as always thanks everybody
for your support your subscriptions
donations all that stuff and I will see
you guys in the next video