>> We've seen a lot of different ways that we can manipulate data.
Now, one of the biggest tasks that we're going to have
is to get some insights from that data,
that we're going to want to be able to
try and predict certain behaviors,
try and predict certain values.
Now in order to do that,
what we'll need is to split
up our data that we'll need to be able to say,
"Hey, these are all of the things that are
going to impact some particular value."
Or to put this another way to give
you a couple of different examples
or scenarios that given X data predict y value.
So for example, given a particular customer,
I want to be able to predict
whether or not they're going to buy your product,
or given a shopping cart,
I want to be able to offer up
some other items of that customer might be interested in.
If you've ever shopped online,
you've seen this behavior where you've seen
that little spot that says other customers have bought.
This is exactly what's happening behind
the scenes is it's a little bit of machine learning.
Along those lines with our data,
since we've been looking at Flight Data,
that what we want to be able to do is say,
given a particular flight,
predict the time that it's going to be delayed,
whether or not it's going to be early, etc.
But in order to do that,
what we're going to need to do is to take
our data and split it up.
Now we're going to need to create two different DataFrames.
There we're going to want to set up the labeled DataFrame.
Basically, what we want to predict in our case,
that's going to be how late or
how early is a particular flight going to
be into a DataFrame commonly labeled as y,
and in particular a y?
I know, I've said previously,
my challenges with
single letter variable names and things like that.
This is convention, so I do follow along
with it even though it pains me inside.
So the value that we're going to predict is going to be in y.
So in our case,
the value that we want to predict is the minutes early or minutes
late that it's going to wind up being
at that particular flight would wind up being.
The data that's going to influence this,
and this is going to include things like the time of departure,
the time of arrival,
the airport that it took off from,
the airport that it was going to, etc.
All of that information is going to make up
the features or what's going to influence the label.
So if you stop and think about it,
the airport that were coming from,
the airport that we're going to when you're talking about
things like weather, air traffic control,
etc, all of those different things are going to have an impact on
whether or not a flight is early or a flight is late.
So we're going to take all of that information,
put that off somewhere else into a different DataFrame.
This DataFrame is going to be labeled as X.
So if we wanted to take a look at a little bit more
of a concrete example here.
What we're going to notice is we've got the distances,
we've got the elapsed time,
and then we've got the arrival delay.
So seven minutes, four minutes early,
five minutes, etc, the whole way down,
what we want to be able to predict or our label is going to be
that delay and everything else is going to influence it.
So if we remember from what we had before,
using that location indexer we're able to
go get all of the rows for that arrival delay column,
and then in turn we can go grab all of
the features by just grabbing everything else.
We could either list them off like we have here
or maybe if it was basically everything about that last column,
we could've done ahead and set-up a range where we indicated,
"Hey, go zero to whatever it might be."
The end result is going to wind up being,
that will have two different DataFrames, X and y,
where X is going to be all of our features and y is going to
be the label that we're going to be trying to predict.
You're also going to notice that
the index is going to come along for the ride.
That gives us the ability and also gives our
trainer the ability to align everything properly.
So that way we know which pieces of
data are going to be correlated to which eventual results here.
Now, the other thing that we're typically
going to have to do is to
split up our data into training data and testing data.
When we think about trying to train up
a machine learning model or really train anything up.
What we're going to need to do is feed it some information,
"Hey, this is what's going on."
We're going to make some assumptions based on that information,
and then we're going to want to be able to test our assumptions.
This is how we learn.
That we'll learn something and then we'll
go off and we'll test that behavior and we'll see,
"Hey, is this in fact the case?
Is this not the case?"
That's exactly what we're going to do
when we're setting up our machine learning model.
Now, we're going to say, "Hey,
this is the information that I want you to use to train yourself,
and then this is the information that I
want you to use to confirm that
all of those assumptions that you've now
generated are in fact correct."
Typically, you're only going to have one set of data.
So what you'll do is take that one set of data,
take some of the rows,
turn that into the training data,
and then some of the rows and turn that into the testing data.
So you're going to notice that we've got our DataFrame X.
What we're going to do is we're going to split that up into
some training data and into some testing data.
I want you to notice the indexes here.
You're going to notice that on the training side,
we're going to have zero and two,
whereas on the test side we're going to have one and 17.
Everything else that's there is irrelevant.
The takeaway that I want you to get out of
this is we didn't just simply say, "All right, well,
the top 70 percent that's where we're going to use to train,
and then the bottom 30 percent is what we're going to
use to validate all of
those assumptions that we're going to use to test.
That's not really going to give us
good results because information could be skewed.
It's possible depending on our data set,
where it is that we've loaded it in from,
that maybe there was a sort that took place.
So maybe for example,
everything is sorted based on the destination airport.
So now what I'm going to wind up with down at
the very bottom is everything that's landing in,
I'm trying to think of an airport code that starts like a Z.
So whatever that might be.
I'm just going to use SEA,
Sea-Tac just because S is a little later.
So let's say that I've got Seattle down to the very bottom.
So now that winds up becoming my testing data.
So all of the assumptions are
going to be based on everything about Seattle,
and then I'm going to wind up testing
those assumptions against Seattle.
You can easily see, "Hey,
that's not really a good way to do things."
So instead what we want to do is we want to randomize this.
So that way we're not going to be
biasing our training and our testing.
So when we go in and we look at our results,
we look at our accuracy, we're going to know,
"Oh, okay, we've done a good job here.
We're testing this and training this on some nice random values."
That's exactly what's happening here.
So as a general rule,
train on 70 percent of
your data and test on 30 percent of your data.
Somewhere in that ballpark.
It's a general nice starting point.
It's a nice general rule of thumb that you can follow.
Now, there is a nice little library
that's available to us called Scikit-learn.
That does an awful lot of really nice things in this space.
One of the things that it does is it offers us a train_test_split.
What's great about this, is we can actually just give
it our X DataFrame and y DataFrame.
So our features and our label,
and then it will give us back everything that we might want.
So we pull that little function in,
then we go ahead and call it.
What you're going to notice is that it will give us back that
X_train, X_test, y_train, y_test,
where X_train is going
to be all the features we're going to train on,
X-test is what we're going to use to
test all of those assumptions that are generated,
and then the y_train and y_test is going to do
the exact same thing except
for the values that we want to be able to predict.
Again, that's going to be that label.
X and y is what we're coming in with.
The test size is where we're going to
indicate how we're going to split everything up.
Now the last item that's on there is the random_state.
Now, we're setting the random_state to a value here.
By doing that, we're now seeding
how random values are going to be generated.
So at the end of the day,
when you specify a random_state,
you're actually not going to be getting random values anymore
because it's always going to be basing it on that exact same seed.
The reason that you might decide to do this is if
you're looking for some level of replayability.
So that maybe what I want to be able to do
is to test out different algorithms or otherwise.
I want to make sure there were always doing
it on the exact same training data,
and on the exact same testing data.
So that's why you might decide to set that random state.
The end result of all of this is we're going to
start with a DataFrame that looks a little bit like this,
where what we want to be able to do is to predict our delay.
So we're going to split off
our distance and our elapsed time into the X side.
We're going to set up the y side as the predicted delay.
Then you'll also notice that we're going to
have our train section,
and we're going to have our test section.
We'll be able to use those to in turn train up our model.
Let's go in and see a couple of
code examples so we can hopefully bring all of this together.