Hello!
In this video, we'll be covering model evaluation.
Clearly, when considering evaluation models, we want to choose the one that will give you
the most accurate results.
So here, we'll introduce and discuss three types of evaluation models that can be used
to help with this goal.
These models are * Train and Test on the same Dataset,
* Train/Test Split, and * Regression Evaluation Metrics.
We'll talk about what each of these are, as well as the pros and cons of using each of
these models!
The first evaluation model we'll talk about is the simplest one: Train and Test on the
Same Dataset.
Essentially, the name of this model says it all you train the model on the dataset, then
you test it using the same dataset.
In a general sense, when you test with a dataset in which you know the classification for each
data point, you're able to obtain a percentage of accurate predictions for the model.
This evaluation model would most likely have a high training accuracy and a low out-of-sample
accuracy, since the model knows all of the testing data points from the training.
We said that training and testing on the same dataset produces a high training accuracy,
but what exactly is training accuracy?
Training accuracy is the percentage of correct predictions that the model makes when using
the training dataset.
However, a high training accuracy isn't necessarily a good thing.
For instance, having a high training accuracy may result in an 'over-fit' of the data.
This means that the model is overly trained to the dataset, which may capture noise and
produce a non-generalized model.
Out of Sample Accuracy is the percentage of correct predictions that the model makes on
data that that the model has NOT been trained on.
Doing a train and test on the same dataset will most likely have low out-of-sample accuracy,
due to the likelihood of being over-fit.
It's important that our models have a high, out-of-sample accuracy, because the purpose
of any model, of course, is to make correct predictions on unknown data.
So how can we improve out-of-sample accuracy?
One way is to use another evaluation model called Train/Test Split.
1.
Train/Test Split involves splitting the dataset into training and testing sets respectively.
After which, you train with the training set and test with the testing set.
2.
This will provide a more accurate evaluation on out-of-sample accuracy because the testing
dataset is part of the dataset that could have been used to train the data.
3.
This means that we know the outcome of each data point in this dataset, making it great
to test with!
And since this data has not been used to train the model, the model has no knowledge of the
outcome of these data points.
So, in essence, it's truly an out-of-sample testing.
4.
However, please ensure that you train your model with the testing set afterwards, as
you don't want to lose potentially valuable data.
The issue with Train/Test Split is that it's highly dependent on which datasets are trained
and tested on it.
The variation of this causes Train/Test Split to have a better out-of-sample prediction
than training and testing on the same dataset, but it still has some problems due to this
dependency.
Another evaluation model, called K-Fold Cross validation, resolves most of these issues.
1.
How do you fix a high variation that results from a dependency?
Well, you average it.
K-fold Cross Validation, in its simplest form, performs multiple Train/Test Splits using
the same dataset where each split is different.
Then the result is averaged to produce a more consistent out-of-sample accuracy.
This is better than using a single Train/Test Split, where our model could test poorly due
to a bad Train/Test Split of the dataset, such as having too many outliers in the test
set.
2.
This example here illustrates the basic concept of k-fold cross validation.
The entire dataset is represented by the points in the image at the top left.
If we have k=4 folds, then one way we can split up this dataset is shown here.
The results of each fold is then averaged, keeping in mind that each fold is distinct,
where no training data in one fold is used in another.
3.
We wanted to show you an evaluation model that addressed some of the issues we've described
in previous models.
However, going in-depth with the K-fold Cross Validation model is out of the scope for this
course.
Thanks for watching!