ROC curves are an important tool for assessing classification models they're
also a bit abstract so let's start by reviewing some simpler ways to assess
models let's use an example that has to do with the sounds of heart makes given
71 different features from an audio recording of a heart we try to classify
if the heart sounds normal or abnormal one of the easiest metrics to understand
is the accuracy of a model or in other words how often it is correct the
accuracy is useful because it's a single number making comparisons easy the
classifier I'm looking at right now has an accuracy of 86.3% what the accuracy
doesn't tell you is how the model was right or wrong for that there's the
confusion matrix which shows things such as the true positive rate in this case
it is 74 percent meaning the classifier correctly predicted abnormal heart
sounds 74 percent of the time we also have the false positive rate of nine
percent this is the rate at which the classifier predicted abnormal when the
heart sound was actually normal the confusion matrix gives results for a
single model but most machine learning models don't just classify things they
actually calculate probabilities the confusion matrix for this model shows
the result of classifying anything with the probability of greater than or equal
to 0.5 as abnormal and anything with the probability of less than 0.5 as normal
but that 0.5 doesn't have to be fixed and in fact we could threshold anywhere
in the range of probabilities between 0 and 1 that's where ROC curves come in
the ROC curve plots the true positive rate versus the false positive rate for
different values of this threshold let's look at this in more detail here's my
model and I'll run it on my test data to get the probability of an abnormal heart
sound now let's start by threshold abilities at 0.5 if I do that I get a
true positive rate of 74 percent and a false positive rate of 9 percent but
what if we wanted to be very conservative so even if the probability
of a heart sound being abnormal was just 10% we would classify it as abnormal if
we do that get this point what if we want it to be
really certain and only classify sounds with a 90% probability as being abnormal
then we'd get this point which has a much lower false positive rate but also
a lower true positive rate now if we were to create a bunch of values for
this threshold in between 0 & 1 say 1,000 trials evenly spaced we would get
lots of these ROC points and that's where we get the ROC curve from the ROC
curve shows us the trade-off in the true positive rate and false positive rate
for varying values of that threshold there will always be a point on the ROC
curve at 0 comma 0 in our case everything is classified as normal and
there will always be a point at 1 comma 1 where everything is classified as
abnormal the area under the curve is a metric for how good our classifier is a
perfect classifier would have an AUC of 1 in this example the AUC is 0.9 to 6 in
MATLAB you don't need to do all of this by hand like I've done here you can get
the ROC curve and the AUC from the perf curve function now that we have that
down let's look at some interesting cases for an ROC curve if a curve is all
the way up into the left you have a classifier that for some threshold
perfectly labeled every point in the test data and your AUC is 1 you either
have a really good classifier or you may want to be concerned that you don't have
enough data or that your classifier is over fit if a curve is a straight line
from the bottom left to the top right you have a classifier that does no
better than a random guess it's AUC is 0.5 you may want to try some other types
of models or go back to your training data to see if you can engineer some
better features if a curve looks kind of jagged that is sometimes due to the
behavior of different types of classifiers for example a decision tree
only has a finite number of decision nodes and each of those nodes has a
specific probability the jaggedness comes from when the threshold value we
talked about earlier crosses the probability at one of the nodes
Jakub this also commonly comes from gaps in the test
as you can see from these examples ROC curves can be a simple yet nuanced tool
for assessing classifier performance if you want to learn more about machine
learning model assessment check out the links in the description below