Hey guys, in this video I want to go over how to do data splitting using python and
there's a very simple function from the scikit-learn python library that does that.
I have some sample code that demonstrates data splitting in python that I'm going to
go over in this video.
Before we get to the code demonstration, I first want to quickly introduce or remind
people what data splitting is.
When you're doing data analysis, you're probably going to import into python a dataframe and
a dataframe looks something like this.
It has columns and rows.
The columns are usually predictor variables or features and you have usually some number
of features.
And then you usually have one outcome variable as well.
This is usually known as your y.
And these are the columns that make up the dataframe.
And then you have a certain number of rows and the rows represent your sample.
Let's say you have observations from 100 people and that's your sample size.
You're going to then have 100 rows of data that are observations of all the features
and your outcome variable. And these columns and rows together make your dataframe.
Now when you want to do data analysis, you're going to want to fit different models to your
data and then pick the best model that fits your data.
When you're fitting models to data in machine learning, you first want to do data splitting.
With data splitting, you're going to randomly sample from your entire dataframe a subset
of this dataframe and basically split it into two pieces.
From your original dataframe or dataset, you want to split that into a test set and a train
set.
With data splitting, you're going to randomly select a subset of the rows and then all the
columns from your dataframe and put a random selection into the test set and then the other
remaining rows are going to go to your training set.
When you do this data splitting, you usually want to have about 80% of your split data
go into the training set and then you want about 20% to go to the test set.
Basically the majority should go to training and you want a chunk to go to test set.
Sometimes people do 75-25.
It kind of is up to you the split you want to do on the number of observations or the
number of rows in your sets, but usually it's about let's say 80-20.
And the reason you want to do this data splitting step before you do data analysis is you want
to train your models on the training set and then pick the best model based on a subset
of the data, and then once you have that best model picked out, then you fit it and you
test how good it actually fits to your test set.
Now when you do this test-train split of data splitting before you do data analysis, the
reason you want to do that is very important and it's basically because you want the model
that you pick that's going to be fit to your sample data to generalize well to the population.
In statistics, you never know what the true population looks like.
You only get a subsample of the population
and you can only model using your observed sample.
So when you fit a model, you want it to generalize still well enough to
new observations. Let's say we fit a model. Let's say this is like a linear regression.
We fit a model to our observations. Well, in the population you might have additional observations
that you don't observe and we still want our model to fit that data well.
What data splitting does is it helps us avoid both overfitting and underfitting our model.
The idea is that if you don't split your data, you might be overfitting your model.
You might be picking a model that works really, really well for your sample
but it might not generalize well to the unobserved population,
which is what we really want.
Or conversely, you might be avoiding underfitting a model, which is not capturing
the full complexity of what's going on at the population level.
By splitting your data at least into two parts, a train set and a test set,
you're going to help avoid this problem of overfitting and underfitting your data
to the population because you're going to be fitting your model to one set of data and then when you test
the data on a test set, the model has never seen these new observations in the test set.
So this is like new data as if it was from a population that was previously unobserved.
You're basically allowing yourself to test how well your model is going to do to unobserved observations.
This is a really common step in data analysis and machine learning. Now let's go over to how to do this in python.
Python has a really simple function for doing data splitting in the scikit-learn python library.
I'm going to do my code demo in Anaconda in a Jupyter notebook and
the environment that I'm going to pick already has scikit-learn python library installed.
When you go to your environment--whatever environment you choose to run your Jupyter notebook in--you just want to make sure you have scikit-learn
already installed and loaded in. When I check my environment, I see that I already have the scikit-learn python library installed,
which means I can use functions from that library. And then next, I'm just going to open a Jupyter notebook.
I have a python folder and then I have a data splitting folder. And this has my Jupyter notebook that I'm going to open up
and it also has data in the same folder. And I'm going to use that data that I found online
as my demo for running the code.
This is my short example of how to do data splitting in python.
I'm going to go over how to randomly split this dataset that I found on kaggle. It's just a public dataset. I'll put a link in the description box below
and it's basically Brazilian house prices. It's like the price of rent for houses in Brazil based on a couple different predictor variables or features.
We're going to go over how you can split into a test-train set that dataset. The first thing I'm going to do is I'm going to import
that dataset using pandas. I'm going to import pandas as pd. After I do that, I'm going to
load in the data as a dataframe. I'm going to load it in a variable called housing_dat
and then I'm going to read a csv file. The csv file is in the data splitting folder. It's already in my Jupyter notebook folder that I'm going to call locally.
I'm going to run that. And If I ran that correctly, it's now going to be stored in the variable.
When I run that line of code, I'm just checking that the dataset was loaded in correctly.
When I look at the dataframe that was loaded in, we have about 6,000 observations of
the prices of rent in Brazil for the different predictor variables and
we have the outcome, which is the actual price based on all these other factors. So you can think you can
try to build a model to see what predictor variables are important in predicting the price of a house in Brazil.
Once we've made sure that we've loaded in our dataframe correctly, let's next go over how we actually do the train-test split
in python. Like I said, we're going to use the sckit-learn python library and that has a single line function
that does the train-test split very easily. It's just a function called train_test_split and we're going to use that
function call. To use that function, we have to separate first from our original dataframe that we want to split,
we have to first separate all the predictor variables into one dataframe
and then the column of the outcome variable in another dataframe.
From our original dataframe, we want to separate basically all the predictors and the outcome variable.
To do that, I'm going to run these lines of code. I'm going to call all the predictor variables and all the rows capital X.
From the housing_dat dataframe, I'm going to subset columns 1 through 13.
That has the first couple of predictor variables. I'm just going to run that.
Then just by running X, I'm just double-checking that my X was correctly subset
and it looks like it was. I have split off from my original dataframe all the predictor variables.
And then next, I want to do the same thing with the single column of the outcome variable.
That outcome variable is called total. From the original dataframe, I'm just going to say
to get all the rows and just the column total.
And I'm going to save that to a variable called y.
I'm going to run that line of code and then I'm just going to check that I ran that line of code correctly.
I see when I run y that it is the column with about 6,000 observations and it's the outcome variable of the total house rent.
We have our predictor variables and our outcome variable split correctly
and now we can finally use the train-test split function. To do that, I'm going to first load in
the scikit-learn python library and in particular I'm going to import that function because it's going to be the only function that we use
from scikit-learn. To do that, I'm going to run from sklearn.model_selection.
This is where the python library scikit-learn comes in
and I say from that python library, import the train-test split function.
Then after we do that, we can finally run the function. To run the function, it's going to be train_test_split.
You pass in first the dataframe that has all the predictor variables. Then
you pass in the dataframe that just has the outcome variable and then you pass
in a parameter test_size. That's going to be the size of the percentage of the observations that you want to put into your test set.
Previously we said that when you do a train-test split, sometimes you can do 75-25, sometimes you can do 80-20.
In this particular case I say make 20% of the observations go to the test set.
By putting .2, I'm saying 20% is going to go to the test set. I'm going to run that line of code
and usually you want to save the output from running this function, but I just wanted to show what the function does first.
As you can see, you first have about 4,000 or 5,000 observations going to one dataframe
and it has all the predictor variables.
Then about 1,000 observations go to another dataframe that has all the predictor variables.
And then the next dataframe that's output has about 5,000 observations of the outcome variable.
And finally, the final output is about 1,000 observations of the outcome variable.
You can see what's happening is it's splitting the X and then it's splitting the y.
It's giving you the train set for X. Then it's giving you the test set for X.
Then it's giving you the train set for y and the test set for y.
If you want to save all that output into the appropriate variables,
you can do that using this line of code that I have here.
The first thing that it outputs is the training predictor variables. I'm going to save that as X_train.
Then it gave me the test predictor variables. I'm going to save that as X_test.
Then the same thing for the y's. If I actually run that line of code,
then instead of outputting to the screen, it's going to save the data into the correct four variables.
And then you can use these variables in your next step of doing model fitting and model selection.
And I also just wanted to say as a last bit for this video, if you wanted to
set a seed to make sure you're getting the same split every time,
you can add the optional parameter random_state and that basically is the same thing
as setting a seed. If you want to make sure you're not randomly splitting but that you're splitting the same way each time,
you would just add random_state and then put in whatever seed number you want.
And that would give you the same split each time so that you can reproduce other people's data analysis.
And that's it for this video.
*chiptune music*