Hello and Welcome to this video lesson on 'Model selection'.
Unfortunately it has been prooven by two gentlemen named Wolpert and Macready in their
"No Free Lunch" theorem, that there is nothing like a perfect model for all data sets.
So you will need to figure out hat models are working better for your particular problem versus others.
Picking the right model for your data set, used case and machine learning problem, is a challenging task
when you get into this the first time, but lukily we at RapidMiner are here to help!
Your first step should be to consider in which of the main task buckets you are in,
because if you know that then you can already eliminate a number of learning algorithms.
You have a 'classification' problem if you question is: "Is this A or B? Will this be A or B?"
You are looking at a 'Regression' with the question: "How much or how many? How many will happen?"
The question for 'Clustering' problem is: "How is this organized? What belongs to each other?"
Associations & Correlations are used to answer questions like:
"What happens together? What changes together?"
And finally if your question to answer is: "Is this weird?" then you are in the area of 'Anomaly Detection'.
In this video we are looking at the most common machine learning task which is 'Classification'.
Now that we know our bucket we want to identify which subset of models is relevant to us.
We have developed a handy tool for you which will give you the answer but before we show you that
let's look at the data which we are using first because we will need that information to adjust the tool.
For many models certain data-types like for example 'categorical' information
is not suitable so that will help us to further narrow down our options.
Ok so we open our '05 cross validation' process and inspect it.
The first operator retrieves customer data and the second filters it to make sure we only have examples
with a valid entry in our 'Churn' column - which is our 'label'.
In order to take a look at our data after these two steps let's set a breakpoint
to pause the process execution once we started it.
We can do this through the right mouse button and then select 'Breakpoint After'.
Later you will probably use the 'F7' key to set and remove breakpoints,
much like I will now hit 'F11' to run the process.
Now we are presented with our data and we see that we have our 'Churn', 'Gender' and 'Payment Method'-data
in form of 'categorical' or 'binary' data and 'Age' and 'Last Transaction' are numerical attributes.
It's also of note that we don't have any missing values in any of the attributes.
Since we are using an example-set to demo things we have just a few attributes and less than 1000 examples.
This is especially important together with the data types as some learning algorithms
may take very long to calculate in case you have many of either one or both.
Before we move on I also want to point you quickly to the indication
that we are now really in the middle of the process at our breakpoint.
See up here the 'Run Process' button has changed and now allowes us to resume the process execution.
However we will just stop it and then return to the 'design'-view as we have completed our data review.
So let's also remove the breakpoint.
Now we are ready to use the little helper tool.
You can find it at http://mod.rapidminer.com .
As I start filling in the information you will see how the list of applicable models changes.
Ok we have all the three column types mentioned here.
Our 'target' or as we call it at RapidMiner 'label' is 'binary'.
Rows and columns we have just very few as mentioned before so we can get by with the default settings.
Ok and now we have a selection of what our starting subset of models is in our area of interest
- which is called here 'Predictive'.
On the right hand side we also see what the RapidMiner user-community uses most.
These numbers are based on our 'wisdom of the crowds' database.
Further we see some warnings that we should be considering memory usage for
"Random forest", "K-NN" and "Rule induction".
Since we have a small example set we won't need to worry about that.
Now let's be bold and let's try to find the best of our 5 most frequently used algorithms displayed here.
Instead of going into our Cross-validation operator and swapping them one by one while writing down
the accuracies we will use the much more convenient 'Compare ROCs' operator.
However, first let's safe our process as "06 model selection".
And now we will disable the 'cross validation' and will drag and drop the 'Compare ROCs' operator
into the process pannel.
When you select it you see the option to change the number of folds of a cross validation
or the split ratio to be used to create a training and test set.
So in essence this operator loops over every algorithm it contains to determine accuracies.
And you won't need to worry about the 'training error' which we mentioned in the 'Testing a model' video
because RapidMiner will automatically perform a 'cross validation' or - if you set the cross validation
to the value "-1" - it will perform a 'split validation' .
Through the magic of video editing we will setup the 'Compare ROCs' operator
with our five preselected, machine-learning algorithms quickly.
You can see an exclamation mark on the "Random Forest" operator and when
we click on it we see that it is not ready to handle missing values.
In a different scenario we cold apply a quick fix to make sure mssing values are handled but since we know
that our data has been filtered to have no missing we can ignore the message.
Consequently we are now ready to run our model comparison.
And here is our result which is called a ROC graph.
In the ROC graph the 'true positive rate' is plotted on the y-axis and the 'false positive rate' is plotted on the x-axis.
But where do we get these curves from?
Let's take a step back and see what we have been doing before.
So we reactivate our cross validation which has a decision tree inside and we run things again.
The output of our 'cross validation' is the accuracy and a confusion matrixs,
which we explain in the "validating a model" video.
The 'true positive rate' is now the amount of positives, correctly classified
and divided by the amount of total positives.
The 'false positives rate' is accordingly the amount of negatives incorreclty classified
divided by the total amount of negatives.
The 'false positive rate' is also called 'false alarm rate' and the 'true positive rate' is also known as 'hit rate'.
Now these terms give us a good indication about the background of ROC graphs.
ROC stands for 'Receiver operator characteristics' and was established during World War II to describe
the performance of the people supervising and interpreting the signals on the Radar displays.
Now let's look at a blank ROC graph with hit rate on the y-axis and false alarm-rate on the x-axis.
Operating stuff which had an as good "hit rate" as "false alarm rate" are as useful as a coin when decided
if measures were to be taken when a signal appeared on the radar screen.
In the upper right or lower left we would see extreme examples of an operator who never triggered an alarm
and therefore had also a small false alarm rate or one how would always trigger an alarm and
therefore would never miss an actual true case but produced a lot of false alarms.
Neither of those is useful.
The best 'receiver operators' had a very high 'hit rate' and a very small 'false alarm rate'.
The same is true for our classification algorithms.
One thing which is still to be explained is: how do we get from a ROC point characteristic
which we can calculate from our confusion matrixs to a curve?
For that we have to extract the confidence for each prediction made by our classification algorithms.
The confidence is similar to the probability that the predicted class will actually be correct
and therefore tells us how sure the model is with its prediction.
In RapidMiner, we can retrieve those confidence values from the performance operators.
Make sure to connect these to a results port so we get an output.
Now as we run the process again you can see the confidences displayed.
If we rank them you can see that the prediction changes for those examples
with confidences lower or higher than 0.5.
This is our decision threshold which seems intuitively reasonable.
However, in some scenarios we want to adjust this threshold to adjust our classification model.
For example in cases where we are testing for an indicator for cancer
we maybe want any example which even has a small probability to be flagged positive.
Or in cases where we are screening our customers for upsell we only want those flagged as interesting,
where we really have a high probability to land a deal as otherwise we will overload
our sales team and waste their time.
So by varying the thresholds we are in fact influencing the decision if an example
is flagged as a positive or a negative case.
Therefore we also generate a new confusion matrix with every variation in the threshold and in turn get a new pair
of true positive and false positive rates resulting in different points on our ROC graph,
which we can connect to draw our ROC curve.
Now going back to our output of the 'Compare ROC'-operator we can say that
the models with ROC curves further to the upper left are the better ones.
However, the final model selection for you is also dependent on the preference
if you are looking at a more conservative or more liberal classifier.
In this video we only compared un-optimized models with each other but often the performance
can be drastically improved if you change the modeling parameters.
In another video we will show you how you can do such an optimization in RapidMiner.
Thank you very much for watching!