what is going on everybody welcome back
to another data analysis slash data
science tutorial with Python and pandas
we're going to be continuing our work
with the avocado data set before we
leave it for a new and exciting data set
in the next tutorial just a few more
things I want to show with the avocado
data set especially just issues you
might run into over time so let's go
ahead and jump in so the first thing
that we're gonna do is we're just gonna
basically recreate where we were import
pandas as PVDF equals PD read PD read
CSV data sets of kado dot C S the Albany
D F equals D F F equals D F where D F
region is Albany not all Pandy Albany D
F dot set index as dates in place is
equal to true and then finally Albany
ATF dot actually we're just gonna do
average price for price dots plots and I
didn't look up the command
so I'll just for this twice and there we
have it so pretty graph but there's a
few issues that we're having right out
of the gate the first issue is these
like these dates are running over each
other and when you see something like
this it probably means pandas doesn't
actually realize it's a date so the
first thing that I would do is convert
that to a date time so when we read in
the CSV if you actually have a date the
probably the big thing you'd want to do
is go ahead and just say DF date or
whatever the column name is equals P D
to date time and then DF date so this
will probably try to think we've done
done one in the first one but I don't
think so this is our first time where
we're actually reassigning the value of
that column this is no this is no
different than any other
function really like he can just map
functions to the to the column here like
that the other thing you can do is like
map and apply and stuff like that and
we'll talk a little more about that but
this is just a nice built-in pandas
function that's gonna convert this
column and then we're just reassigning
that value because this is actually just
gonna return a bunch of values for us so
pretty cool anyway now what if we go to
graph that we can see ah okay they're
nice and slanted and it appears that
pandas actually knows hey that these are
dates so awesome so that's one thing the
next thing is this graph is crazy crazy
busy looking so the first thing I would
think of to smooth it out is to use some
sort of moving average so if you don't
know what like a moving average is it's
like at any point along the way let's
say we're gonna do a 25 moving average
so we'll take every point we're gonna
say okay at this point so in this point
and the previous 24 points what's the
average so 25 point average and then we
go one point over and do the exact same
thing we just keep slowly rolling that
and what that's gonna wind up doing is
condensing these like crazy fluctuations
and it will hopefully smooth things out
a little bit so nobody the Rena D that
is Albany DF and then we'll do average
price and then dot rolling so this is
like any rolling thing then you would
pass a number for the window and then
what kind of rolling so you could say a
rolling sum or in this case we're gonna
say a rolling mean and then we're gonna
plot it cool except not cool because
this is pretty ugly something went wrong
so the first thing I would think of when
I would see in chart like this is that
possibly things are out of order so what
if we said Albany DF dots
let's do head 25 so when we look here we
can see okay we start so it's in reverse
order and then we scroll down and at
least in this case it does look like
everything is in proper order let's do
dot index
but you can see here at least right out
of the gate the index does start at late
December and then goes down and reverse
so you would think oh this is reverse
chronological but apparently it's not
because then we have 2018 which again
goes in a reverse chronological order so
something is wrong so the first thing
that I would do is like okay let's make
sure these dates are in proper order so
what I would say is let's do all Benny
DF dot sort index in place equals true
you can also sort data frames by
specific columns and that will also
adjust the index just for the record we
do get a little bit of a warning here I
will talk about that warning in a little
bit so all I want to do now is actually
just sort that and then craft that again
okay and now we get a pretty graph and I
guess the warning doesn't show I don't
know what we've already kind of sorted
the index and stuff anyway I will see
the warning soon enough so now we get a
nice smooth out graph that actually is a
little different than the other graph
there are kind of like these two humps
but the second hump is quite a bit
higher than the first whereas in that
first graph it was clearly like this
one's clearly wrong and then even this
one looks to be pretty darn wrong so
pretty cool now we've got that fixed so
the next thing that I want to do is what
if we actually want this to be a column
in our data frame because we can later
output this data frame to a CSV again
and it doesn't have to look exactly the
same as it came in it could be anything
so we decide hey this is a valuable
meaningful column so actually what I
want to do is copy this column come down
here and I'm just say Albany D F and I'm
just going to sign a new name I'm gonna
say price 25 M a is equal to that value
okay so we'll run that see that warning
again I will explain that momentarily
Albany
dot head and then we'll just do like
three here and we can see initially we
do see Nan's
that's because the rolling periods
required 25 so the first 25 are going to
be nan because there's nothing to
calculate later you can set I think it's
literally min periods or something like
that
and you can overcome that otherwise what
we could say is Alban ADF tale 3 and
boom
we've actually got values alternatively
I will almost certainly use this again
in the future but you can say Albany DF
dot
drop an a for example and then we'll
just say dot head 3 and in this case it
what droppin a is gonna do for us is
remove any rows that have any values and
a not any or any na s anyways it removes
DNA's so later if you have like gaps for
whatever reason or you just want to drop
missing data for what a n a n a n is not
a number
so anything that doesn't have values you
can just drop them and so you can do
that and you can also drop an A in place
Eagles true if you want I'm not gonna
drop them here but just note that you
can so now that we've done that mm-hmm
let's talk about this error so first of
all it's not an error I should I can't
believe I just called it an error but
it's mama part of my point is it's not
an error so a little bit of a warning
pandas is really happy to serve warnings
like you're gonna see tons of warnings
from pandas same thing with scikit-learn
we're gonna use scikit-learn a little
bit later and like part 5 probably in
these two libraries for whatever reason
just love to issue warnings about like
deprecations possibly like future
deprecation is like way far in the
future so you'll see the warnings for
like years and then anyway with pandas
what this warning is telling you is a
value is trying to be set on a copy of a
slice from a data frame and then it says
try using dot locro i don't think that's
actually useful
I don't usually warnings are pretty
useful they'll
tell you about future deprecations and
new functions that you should use or
whatever so read them but I don't really
think this is useful personally I don't
think that's the right way to do it but
anyway the warning still stands hey it's
just letting you know you're doing
something that might screw you up later
so basically what it's telling you is
like as you begin to do work like we've
actually we've kind of forked the
original data frame into this Albany
data frame and then we're changing
values and what pan is is trying to tell
us is like if you were just to go
perfectly in order and only modify a
data frame and always keep it DF you
would never have a problem but if at
some point you fork and then you start
changing values over here and here and
here well don't forget that that that
original it came from that original data
frame and you might be either one
changing values that you didn't expect
or not changing values that you did
expect so that's what that warning is
there to tell you so how can we not see
this warning because later you might be
doing things and you might be iterating
and then you'll see like hundreds of
these warnings so one way that you can
do it is by telling pandas I get it it's
a copy so how could we do that well I'm
so glad you asked Jon Albany DF equals
DF copy and now what that does is just
return to us a new copy of a data frame
because later we could modify that data
frame and it won't impact Albany DF but
panas wants us to know it won't impact
Albany DF so instead we just do DF copy
which returns a data frame so now we can
reference that you know we can treat it
just like a data frame right away so
we're gonna say what DF copy where DF
region region equals Albany then we do
all the same things that we did before
so for example we can take this
and uh I can't remember when we actually
said okay so we set the index afterwards
so let's go ahead and set the index and
we also sorted the index I believe
afterwards so set index sort index let's
just run that real quick okay cool
and now we don't have any warnings so
basically I just wanted to show you guys
how you get away with you know not show
in the mornings okay so we've got this
graph now the next thing you might think
about is like hey I want to graph all
the regions on one graph and see them
compared okay so how would we do that so
the first thing we'd need to do is
somehow iterate over all the regions but
we have to figure out all of the regions
so the cool thing about pandas is at the
end of the day it is rows and columns
and you're each individual column is
just an array so pandas at the end of
the day can be an array you know a
multi-dimensional array homicider array
of arrays like a list of lists anyways
it can be a multi-dimensional array so
and how might that happen by the way for
example because ADF top values boom and
now you can see this is actually being
converted to an array we can do the
exact same thing with a column so we
could say DF region dot values and we
see we get an array now it's just
doesn't give us the length but what we
can say is well it's gonna be the length
of the different which I think was like
18,000 or something it's not gonna be
interesting to know that so we can also
convert it to a list to list and then
you can see okay we got quite a bit of
values here too many values so how do we
get like just the unique items well one
option would be quite literally just
convert it to a set and now we can see
ok these
or all the regions and then we could
convert it to a list and then we could
iterate over that list so okay the
problem is that's messy and there's a
simpler way so a lot of times in pandas
there's just an easier way than whatever
we're doing so in this case there is it
would be DF region dot unique done so in
this way we can get all the unique
values and then we could actually
iterate over these and for that reason
to don't forget about the pandas docks
scroll through the docks just get an
idea for all the things that are
available to you before you start trying
to do it in Python and moving it back
because again if you can keep things in
pandas it's probably gonna be faster
than what you write what you write will
may end up with the exact same outcome
but pandas is likely the pain is days
using C++ you're gonna be using Python
which is slow so so yeah okay DF region
unique cool so we have the unique values
now how might we graph every single one
so long pant are on matplotlib basically
you've got this canvas in the background
and then anytime you plot something it
just kind of adds but we're on this like
Jupiter notebook so it's gonna like not
yeah well it's Jupiter lab now but
anyways probably what we want to do is
actually create a new data frame and
then just plot that data frame so really
our task here is these regions are
values in rows and instead what we want
to do is actually almost like reshape
our data frame to be a data frame where
the columns are the regions and the rows
will the column headers or the regions
and the values of those columns are the
let's say the 25 moving average and then
the indexes date so we just kind of want
to restructure it basically mmm-hmm so
the way we're going to do that is by
iterating over the regions so there's a
graph DF and basically using all the
same code that we've used up to this
point
I'm gonna make some space I always hey I
hate being like at the bottom so okay
graph DF there's lots of ways that we
can do this but this is generally the
way I kind of like to start such an
operation so PT dot dataframe now for
region in DF region dot unique now we
just want to iterate over all of the
regions so we could let's just print
regions just so we know where we are and
then we want to do basically all the
things that we've kind of done so far so
we're gonna say region DF equals D F dot
copy and we're gonna say DF copy where
DF region is equal to whatever the
region is that we're working with and in
fact I wonder if I can just I'm gonna
copy this copy come down here pasta tab
tab region DF region DF region DF and
region DF okay I think that's good so
okay so now what we want to do is let's
just say if graph D F dot empty so
basically if this data frame doesn't
have anything in it then all we're gonna
say is graph D F equals region DF and
all we want is that 25 ma column so and
in fact okay we got it we gonna fix one
thing but for now I'm just gonna say
it's price 25 MA right that's the one
that we want but a couple of things here
this will return a series we don't want
a series we want this to be a data frame
so we can continue adding new columns so
the way that we take this is a slice
this hey jerk not cool this that that's
a dataframe so the only difference is
we've got two brackets and then later if
you wanted to specify multiple columns
you could say price MA and then you
could go with the average price like for
the raw data but I don't actually want
that I just want the price MA
and then we're gonna say so if it's
empty then it's that else we're gonna
say graph D F equals graph DF join so
this is how we can bring data frames
together on an index and then we would
want to join region D F and then that
price is 25 ma column price 25 MA now
the problem is and if you want to learn
more about joint check out the docs but
basically join is just going to be used
where you want to you have two data
frames and they're indexed in the exact
same way then you can just call you can
say the first D F dot join the second DF
and then they will be joined on their
index so the one problem we're gonna
have is we've got many regions like I
don't know 50 or something and they all
have the same column name that's gonna
be a problem so instead of calling them
all the same I'm gonna use f strings and
then we're actually just gonna do
underscore and then we're going to say
whatever the region name is so this is
how we're gonna give it a unique column
name every time so copy that and I'm
gonna paste it into here and then
finally into there so with that we can
join them all so I'm gonna go ahead and
run this I already know it's not gonna
work
Oh actually before I run it I'm gonna
just say up to sixth not 26 16
okay so run that and then we can
actually see here that it's it's slowing
down we got to Grand Rapids grant Great
Lakes and then Ike I don't know if it's
done yet let me see
yeah okay now it's done but like the
first bunch came really quickly right
and then it slowly starts grinding ah so
um so yeah that's relatively interesting
so I was trying to figure out what the
heck is happening cuz this is a logic
that I've used for years and I've never
had this issue and if I watch the
processes like if I actually open up
like task manager or something
this is exploding Ram now what I don't
know is why it explodes Ram what I did
find out is why so if we look at if we
look at our data frame let's just graph
let's just do D F dot Tale 3 if we look
at our data frame everything looks
pretty hunky-dory and all set the
problem comes with right here so here we
have separate PL use right so each
unique PLU has its own column what
doesn't have its own column is type so
you have two different types so if we
said DF type dot let's just do unique
again there's actually two types
conventional and organic this is what's
causing our problem because we've got
duplicate dates so it's expected that
our index is the same but the problem is
will we end up having multiple dates all
the same so for example if I say graph D
F dot tail and run that well these are
all the same date and so when we go to
actually join pandas is looking for okay
where should we join this well has many
index
is that are identical so it's like I
don't know what to do so so again I
don't know the underlying reason why Ram
is exploding so much but I know why and
it's because of the date so instead what
we need to do is when we bring in
why we got to go all the way up to the
top huh trying to decide so DF copy I
kind of want to I guess I just want to
get it at the source to be honest with
you guys so probably what I'll do is yes
we convert to date time so I'm gonna
take this and in fact let's just let's
just take up all of this and then come
down here and just kind of redo what we
should have done so import pandas as PD
D F equals P D re in CSV that's fine and
then let's just immediately sit pick
either conventional or organic it really
doesn't matter but I'm just gonna say DF
equals DF copy otherwise it's gonna be
like ah that's exactly what it sounds
like by the way DF type equals and then
we'll just go with or gimmick okay then
we set the date and then let's do a DF
so for example do you have to so well
while I have you guys I can just say DF
dot sort values and we're gonna say DFS
sort values by equals date and then
we're gonna say ascending equals true I
believe button normally it's actually
false I think is the default and then
we'll say in place equals true so so
this is just how you would sort a data
frame by of certain values so rather
than because before we showed you sort
index but I said hey you can sort by
columns too here's how you do it and
then let's do D F dot head
[Music]
cool and now I kinda wanna just well
let's just iterate over this so now what
I want to do is come back up here
because we're actually setting index
here anyway so let me take this come up
here
copy come down here give sort values
should be good there let's just run that
real quick make sure okay I think so I
mean that was much quicker right so now
I'm gonna get rid of this thing that
limits us graph TF and then let's just
let me Wow at the end here let me just
say graph D F dot tail so run those duh
yeah it's definitely going through that
like really fast okay so now finally we
can plot this so first you know the
first thing that we might say or try to
do is graph D F dot plot oh there we go
two issues one is the legend is just too
crazy the other thing is the graph isn't
quite yet big enough so if you if you
want again pandas can always be
converted out to be an array or list or
whatever so if you know matplotlib you
can make crazy graphs just how you know
with matplotlib but with pandas like for
example dot ply that's some magic we
never even imported matplotlib so you
can either just use what they give you
you don't have to put up with this or
you can make custom at plot lib graphs
if you want to learn more about
matplotlib you told i've got like a
total super long series on us to data
analysis data vis maybe i thought i
clicked it anyway there we go and you
can go through this tutorial there's
quite a few parts and you can learn all
about custom charts but instead we can
just say a few simple things here so we
can say figs eyes and we'll
make this eight five that's by its in
inches apparently and then legend we're
gonna set that equal to well uh no idea
why I messed that up
anyway legend equals false beautiful so
this is all of them compared to each
other yes it would might be useful to
have the legend actually but it is what
it is why do we have this empty gap we
talked about it already it's because of
the n/a values so we can actually say
drop an A and then plot it and then now
you don't have that gap there anyway I
think that is a stopping point we could
definitely poke around further we could
probably do this entire series on the
avocado data set that I might get boring
so in the next tutorial we are going to
visit a new data set with new challenges
so yeah if you got questions comments
whatever leave them below also shout out
to my most recent channel members is a
baya shower Carlos Daniel a guar and
zuki Marwan thank you guys very much for
your support you guys are amazing I will
see you guys next time questions
comments below come chat with us in
discord otherwise join me in the next
video till next time