what is going on everybody welcome to
part 6 of the Python data analysis and
data science with panda's tutorial
series in this last installment we were
going to be talking about applying
machine learning to a panda's data frame
so basically how what is the typical
workflow when doing machine learning
with pandas data sets so generally for
if I'm going to do machine learning
first I do want my data pre-processing
in pandas anyways so it's just
convenient to know ok well what's the
next step like once we've got our
features how do we feed it through a
model so that's what we're doing here
we're gonna be grabbing a new data set
and we're gonna grab the Diamonds data
set so go ahead and download that put it
in your datasets directory so what's our
objective gonna be well we are going to
be predicting the price of diamonds so
basically this table contains a bunch of
values here we've got the carat cut
color clarity depth table I don't know
what that means but it's table percent
we've got the price the X Y Z which is
length within depth of that diamond so
the curiosity is can we take all of
those values except for price feed those
through a regression model and predict
the price of that diamond so that in the
future when we get diamonds and we don't
know how much to pay for our diamonds we
could just run them through a model so a
typical regression task here nothing too
fancy and the question is are these
features all descriptive enough to give
us the price of this diamond
probably we shall find out you can use
another data set if you'd like you can
this one has almost fifty four thousand
rows which is quite a few samples that
we could actually work with here which
is pretty important for traditional ML
you're gonna want probably more than ten
thousand rows I would think and then if
you want to do like deep learning more
than like a hundred thousand rows we're
just gonna use regular machine learning
models nothing too fancy we're not going
to do deep learning here but if
learn more about what we're doing you
want to learn more about machine
learning or just scikit-learn or just
basic ml models in general you can check
out this initial machine learning
tutorial series here this one basically
what we do is we go through starting
with regression which is what we're
gonna do here talking about how
regression works we do an applied
example using scikit-learn and then we
actually write a regression model
ourselves and really we do that with all
of them so we do that with regression K
nearest neighbors support vector
machines and so on so the idea is to
learn about the model learn how to apply
it with scikit-learn and then how do we
write that ourselves from scratch we do
use numpy and stuff like that but not
using some sort of machine learning
library so it's pretty cool series and
if you want to learn about deep learning
check this one out
so anyway if you want to learn more
about that kind of stuff in parameters
because there's a lot of parameters here
that we're gonna be working with and if
you want to know more like you don't
feel like like you feel like you're
still kind of in the gray area as far as
what all this stuff means you can check
that out that series out so now what we
want to do is let's say you just are a
complete amateur you can check out this
choosing the right estimator chart
basically if you just Google choosing
the right estimator you will find this
this is from scikit-learn by the way and
this is just how you can pick a specific
estimator for and that's a classifier
it's like your model how do you pick the
right one for what you're working with
now so for us we have more than 50
samples we do not want to print a
category we do want to predict a
quantity so because we're doing
regression we want to predict price
which is since a regression task as
opposed to classification where you're
trying to predict you know one out of
five classes in this case so it's not
classification because it's like any
price right we're just we're trying to
come up with some sort of calculation
for price do we have less than a hundred
thousand samples if we if no then so if
we have a lot of samples that's kind of
confusing if you have more than a
hundred thousand samples go with std
regression or otherwise you should go
either here or with support vector
regression with a linear kernel so we'll
probably do SV our linear kernel
and again if you don't know what that is
you just click on it right and this will
tell you okay here's what you need to do
so okay I understand right okay SK learn
import SVM got it
use your classifier got it do a fit got
it easy right so let's get started so
we're gonna import pandas as PD we are
going to say DF equals PD don't read CSV
date whoops not a capital D data sets
and diamonds don't CSV then we also want
to say index call equals zero because
this data set Grace's us with a luck
that's avocado with a lovely index not
only is it useless it's also in string
form fascinating okay so coming back
over here so we're gonna say index call
equals zero because that way we don't
generate duplicate indexes in this case
the index column is completely useless
but one thing you always want to keep
take note of is let's go ahead and just
DF dot head here anytime you're doing
machine learning it's really easy to
cheat even when you're trying not to
cheat you're doing your very very best
it is easy not to so looking at this
data set it's easy to cheat anyway
looking at this data set basically we
like all of the columns here are
meaningful columns and then you have pry
except for price price of oil price is
meaningful as well but price is what
we're trying to predict so when we go to
build this model we actually want to use
all of the columns and set for price but
when we get to that point I'm gonna
point something out and if I forget
someone comment below because it's
important okay so that's our data set
now one thing with machine learning is
all of the data that you pass into your
model at the end of the day basically
all machine learning is is linear
algebra okay so everything has to be
numbers we have to convert everything to
numbers and ideally they're meaningful
numbers because if they're not
meaningful they're useless
so there are ways like so we have cut
color and clarity all of these need to
be converted to numerical values in
pandas as well as probably scikit-learn
and I know it's in chaos and tensorflow
there are always like two categorical
methods that you can call so like in
pandas we can say DF so for example what
if we just said DF cut dot unique unique
okay there's not very many here but you
can imagine a scenario where there's a
lot so this presents problem we need to
convert these to numerical values well
one option you have is DF cut anyway
sorry I was trying to see if that had a
meaningful order I don't believe it does
but anyways DF cut and then we could say
dot as type and we can convert this type
to categorical actually I think it's
category E and then dot cat dot code I'm
gonna run that real quick but uh codes
there we go
okay so this will take cuts it will
figure out okay how many uniques are
there and then it just assigns RS you
know the first one it finds is zero
sorry zero the second one is one two
three it just keeps doing that until
it's done it's reached the maximum
number and so it just assigns an
arbitrary code to our cut the problem is
we're doing regression likely linear
regression and we would prefer our
values here to have meaning behind them
so because cut is this isn't arbitrary
right premium is better than fair and so
on so we want to preserve that order so
we're going to preserve that order so
just know this exists so later if you're
doing classification you just need
arbitrary classes so when you're doing
classification you could use this
totally to make your classes into codes
but in this case for features we
our features to be meaningful so we're
not gonna do that so instead what I'm
gonna do is create dictionaries for all
these things and in fact I'm gonna copy
and paste these from the text-based
tutorial because there's nothing to
clean here so copy/paste why are we let
me zoom out a little bit they're a
little far in there and then copy and
paste this one so this will just be
dictionaries that we're gonna map how
did I know this in my diamond aficionado
no I found these these from the
description here is the data set it they
order them and stuff for us so yeah so
now interesting clients I started this
one at 0 that shouldn't matter but
that's funny I kinda want to fix it
let's fix it I don't know how I just
noticed that now but anyway let me fix
that real quick or 5 6 & 7
I don't really want to pass a value of 0
to my regression model if I could avoid
it let me fix that in a text-based
version okay so now that we have these
we just want to map them and so I'm just
gonna come down here and I'm gonna say
DF cut equals DF cut da map and then we
just map cut class dict right that's it
and then we're gonna do the exact same
thing for clarity and color so I'm just
gonna copy paste copy paste do this copy
paste paste and then it's just clarity
dict and then color dict so whoops no
just colors all I want color color and
color awesome
so we map all those let's just check it
with the DF head at the very end
great we now have our dataset is
basically it's ready it's been converted
we're ready to pass this through an
actual model sort of we'll talk about
why not in a second first we need
scikit-learn so let's go to our favorite
command
prompt and do a pip install Sai kit -
learn we will grab scikit-learn and
while we're doing that I'm just gonna
come back over I think and just start
typing so cool so we're going to import
SK learn and then we're gonna go from SK
learn import SV em so dang it I want to
create space but I can't because it'll
probably installed yet oh we have cool
so what kind of it awesome okay so what
we want to do first is anytime you've
got data you want to probably shuffle
that data because the latest thing often
we're gonna train and like order and the
latest thing is gonna be more biasing
than the first thing that model saw so
it's just usually a good idea to shuffle
the model especially if they're sort if
that dataset is sorted in any way and is
this dataset sorted in any way well we
come over here and if we scroll to the
tippity-top we can see price does appear
to be it this dataset appears to be
ordered by price now so that's a problem
and then also it could become a problem
later on so first I'm gonna shuffle this
data set just to get it over with so
there's many ways to shuffle a data
frame there's way in pandas to do it by
shuffling by index but I don't want to
do that because that's ugly the way the
way pandas has you do it I just don't
like and scikit-learn is it's really
simple and we're using scikit-learn
already so I'm gonna say DF equals SK
learned you till z' dot shuffle DF done
that data frame is shuffled now we want
to assign values for x and y so in
machine learning generally capital X is
your feature set lowercase Y sometimes
you'll see people use uppercase Y it is
your are your labels so X what is X X is
the feature set so this is the
list of features that points to that
label what's the label price so the list
of features is basically everything
except for price right so this is pretty
simple in this case we just do DF drop
and we're just gonna drop that price
column on axes 1 because we're not
trying to drop rows we're trying to drop
columns and then convert that to values
and then the next thing is DF this is
just DF Price dot values now I did say
that I wanted to point something out so
let's just run that real quick and then
let's print let's just look at X really
quickly so look at how X compares to up
here for example it appears that this is
indeed carrot carrot I'm thinking Karros
anyway this is carrot right now one
thing to watch out for now imagine at
the very beginning that we did not do
index call so if we hadn't done that
this index value here this course of
probably would have been a string pandas
might have fixed that for us but imagine
it didn't and now one of your columns
was actually gonna be indexed which is
just incrementing number 1 to whatever
would that be a problem or would that
just be noise well that would be a
problem that would inform the model of
price why because this dataset is sorted
by index apparently because index just
increments by 1 and it was sorted by
price so that is one way that you could
just unbeknownst to you have cheated and
that's the kind of stuff you got to
watch out for it's always just it's
super difficult you're gonna find this
kind of stuff happens all the time it's
really possible that I'm cheating in
some way already on this dais I haven't
spent too long trying to make sure I
don't but so if I do let me know below
but anyway that's the kind of stuff you
got to watch out for because it will get
you so anyways it appears that we've
done it right one way you could always
be certain is to instead of
no blacklisting just the price column
you could whitelist just the columns
that you want to include but I don't
feel like typing all those out so that
should be fine now X Y values okay now
we want to I also kind of want to fix
kind of on a scale mmm so I think that's
what we're gonna do now
so SVM we're also going to import
pre-processing in general scaling often
helps sometimes only a little I've never
seen it hurt so generally what you want
to do is scale your data and the target
is usually between 0 & 1 but the really
the point of scaling is to simplify your
data for a model at the end of the day
these models again are just linear
algebra so the simpler we can make this
problem the motor we can like reduce
this problems complexity the better so a
scaling we're just trying to bring the
range of all these values to something a
little more digestible by our model so
I'm just gonna say now x equals
pre-processing dot scale X on your own
time you can comment this out and see
how much of an impact it makes I or go
to the text-based version of this
tutorial I show the difference I just
don't want to these models are gonna
take a little bit to Train I want to
waste time so anyway so we've scaled our
data now what we want to do is we want
to split apart training and testing data
so I'm gonna say test size equals 200
because we got like fifty four thousand
rows so we got tons to work with now
we're gonna say X train why train X test
Y test so we want to split apart some
number of our data that the model will
never see so X train and Y train these
are the things that our model is gonna
fit against so we're gonna train it on
that data and then later when we're done
training we're gonna test our model on
this data because it's never seen this
data so if you tested it on data that
you fit you're likely to perform much
better than on data that it's never seen
so the
true test is always out-of-sample data
it's never seen again this is a really
easy thing to screw up and cheat on and
somehow get a hundred percent accuracy
but not really so keep that in mind so
now X train is going to be equal to X up
to negative test size so this is up to
the last 200 in our case copy paste make
sure that is why otherwise you'll not be
happy and then this will be X for the
last test size so for the last 200 :
don't forget that either that would be
annoying okay so now we're ready now we
define our classifier so CLF is
generally the standard for classifier
equals and we've imported SVM already
and we'll do SVR and then we'll say
kernel equals linear and again if you
want to see linear versus I also run an
RBF I guess I've already oh it clicked
so probably back yeah I also run the I
do iso SG regressor I think RBF and
linear kernel I can't remember I might
only do RBF to be honest I can't
remember anyway if you want to see some
you can check that out or just run it
yourself see how you do but anyway we're
going to do a linear kernel then now we
do the training which is called fit so
see left off now it's it fit and that's
X train y train so that process is gonna
take a moment so I'm gonna go ahead and
run that and I'm just going to glance
over it make sure I did everything right
for a waste of whole bunch of time every
process price values blah blah blah blah
blah values here scale extremely trained
up to the last 200 and then this is the
last 200 ok great so now once we fit a
model we'd like to know how good did it
do so you can do that with self dot
score and we'll score based on X X test
Y test these score is going to be an
r-squared or coefficient of
determination
the sense of basically zero is bad 1.0
is great same thing so like with so
that's how you would score a regression
model in the case of classification it
will be quite literally is it right is
it wrong zero to 100% don't know so it's
a little more simple there in this case
this is just like it's more of a factor
of like how wrong are you like with
regression and in probably in theory
that's even more accurate to be honest
so she left escort okay and then the
score is fine but especially like with
regression I'll go the classification
the score is like totally obvious like
okay so we screwed up 30 percent of the
time with Russian you're almost never
gonna be you're like you're not gonna be
perfectly accurate if you are the
challenge was too easy or you cheated so
the question of like r-squared is kind
of like super vague like especially in
the sense of diamonds like does that
mean we're thousands of dollars off or
or are we like just like a few dollars
off of each timing so we're only like a
few dollars off like that's awesome
model so what does that mean well we can
just look at it so instead what I'll do
okay so we've got our answer and it says
we did a 0.87 4 which is actually super
accurate that's pretty darn impressive
to be honest cool so that's a good
looking score or r-squared
but now we must see it with our own eyes
what did it actually predict so what
we'll say is for for X comma Y in list
zip you don't have to make this list I'm
converting it to a list so we can slice
it because I don't really want to show
all of them I don't think cuz it's 200
things I guess we could show them all to
be honest let's just show them all mmm
this is the only one we're probably
gonna do it anyway so zip and then we
want to zip together X test Y test X Y
then we're gonna say print and we'll
make this an F string model and the
model predicts seal f dot predicts and
in predict you always pass a list and it
always returns a list even if you just
want the one thing
so predict X and then we'll say the
zeroeth because we're only expecting one
thing so that's what the model predicts
and then we'll just say actual and
actual is whatever Y was cool so that
should be 200 things so we'll just print
that out and we can see how quickly that
went and then let's zoom in a little bit
okay so we can see that the model is you
know close on the first two here the
model suggests that we pay somebody to
take the diamond off our hands probably
not ideal situation there okay and then
we continue on and most of the I mean
we're like in the general ballpark but I
mean some of these are like still pretty
far off but you can see that like you
know this $12,000 diamond the model knew
okay this is more expensive and more
valuable than this diamond and so on but
we still are off quite a bit on some of
these like this like negative so the
question is why is that happening
I don't know but generally the way that
we would overcome something like this
just for the record is you've got this
model and then generally what you're
gonna see is someone trains another
model so maybe I will show the RBF
kernel so get the heck over here yeah
okay so what I want to do is I'll just
take this cop cop cop cop cop me go down
here
pasta colonel let's do our B F fits
great we'll start that ball rolling and
we will come up here and we'll just run
the score and we'll do this
and then I'll print this just so we'll
see it at the tippity-top when it's done
I believe this will take a long time to
run though so I may pause it and then
come back but otherwise I think I'll
probably just we're basically done here
but I do want to see what that happens
so so generally what's gonna happen is
when when you actually use machine
learning models in practice chances are
it's not like a scenario where you've
got like one model to rule them all
that's not how it works you generally
will do something more like a voting
classifier or an ensemble of classifiers
so you'll have 5 930 391 the classifiers
right you'll have a ton of classifiers
and then they all will vote so or in
this case they will all make predictions
and you'll take the average prediction
or something like that also if one of
them like in the case of diamonds no
diamond should be in the negatives so if
one of the classifiers predicted a
negative you would just like toss that
thing out like that toss that prediction
out so in general when you combine a
bunch of classifiers
they tend that that's almost always
improves kind of pre-processing I've
never seen it actually hurt unless you
use like a horrible classifier for some
reason and I sometimes it only helps
just a tiny bit but yeah if you always
want to eke out the best performance
that's a great way to do it like you
could probably make a voting classifier
and scikit-learn has methods for doing
that you could probably make a voting
classifier and get certainly get
r-squared to be way better ok anyway a
quick shout out to the channels most
recent members zoom key VJ Trevor and
Hao yan thank you guys very much for
your support you guys helped me do what
I really love to do so you guys for that
are awesome all right we are done so 55
not it definitely did worse in terms of
R squared but if we zoom on in enhance
we we appear to not have any negatives
which
is a perfect example of why we tend to
use voting classifiers because while
this classifier is less accurate in
terms of r-squared
it's it's not doing it as negatives
nonsense
so that carries some value but yeah you
can see it's like way off on a lot of
these ok anyways you guys can play more
with that if you want see if you can
make a classifier that's even better
or a classifier a regression model
that's even better anyway that's it for
now if you guys have questions comments
concerns whatever feel free to leave
them below otherwise that's probably it
for the series I might add some more
later on but probably not I don't see
what else I need to add about pandas to
be honest just take out the pandas Docs
to learn more at this stage so I will
see you guys in another video series