Hello, my name is Mohammad Namvarpour.
This is the first part of a series of two videos In which I want to introduce the main
ways by which we evaluate and compare our classification models.
Today, I want to cover confusion matrices, along with some classification metrics.
The book "Machine learning pocket reference" by Matt Harrison, published by o'reilly, was
the key source of knowledge I used to create this video.
Classification is a supervised learning mechanism for labeling a sample based on the features.
For example, Our classifier may be able to separate input
images of handwritten numbers into ten categories, from 0 to 9.
In this example, the features could be the color of each individual pixel in the input
image.
As another example, consider spam filtering.
Email services use features like the sender address and email content to classify new
emails as either spam or not spam.
Since there are only two different tags, this is called a “binary classification” problem.
Now that you know what classifiers do, assume that we've created a binary classifier that,
based on a person's physiological features, determines whether they're pregnant or not.
A binary classifier like what we have developed can have four classification results:
If the person is actually pregnant and our classifier correctly detects it, the output
is a True positive Also, if the person is not pregnant and the
classifier correctly classifies that person as not pregnant, we will have a True negative
output.
These two are correct classifications.
But our classifiers can, and will, make mistakes as well.
Consider an awkward situation: a man takes this test, and our classifier classifies him
as pregnant!
We know that, at least by the time I’m publishing this video, biologically, men are not capable
of being pregnant.
So, our model is definitely wrong.
This is called a False positive At last, A false negative is claiming that
a pregnant woman is not pregnant (when she is clearly showing)
A confusion matrix is a matrix that displays the different types of binary classifier outputs.
It helps us understand whether our model is performing well or not.
The rows of this matrix correspond to the True classes, while the predicted class is
represented by the columns.
We want our classifier to avoid errors (which are False positive and false negative outputs)
and produce more correct outputs, which are true positive and true negative ones.
For example, if a hundred people take our pregnancy test, we would like our confusion
matrix to be more like this.
Notice that values on the green tiles are much bigger than the values on red tiles,
meaning that most of the test results are actually correct.
So far, we've seen that the confusion matrix is an excellent tool for model evaluation,
as well as a creative method to illustrate our model's performance.
Now that we are familiar with confusion matrices, we can learn about several other important
metrics as well.
The metrics that I want to talk about are Accuracy, Precision and Recall.
Let’s start with accuracy.
Accuracy is the percentage of correct classification.
Consider this confusion matrix from our pregnancy test example.
As we discussed before, The number of correctly classified samples
is shown by the values on the diagonal of the matrix, which contains True negatives
and True positives.
In order to calculate accuracy, We should add the true negative and true positive
values, and divide it by the sum of all elements in the matrix.
So, in this example, the accuracy of the model would be 0.85.
This is excellent since it demonstrates that our classifier is correct the vast majority
of the time.
However, we must take precautions when using the accuracy metric.
Because accuracy isn't always a reliable measure of how powerful a model truly is.
Imagine that you live in a world that superheroes actually exist.
Superheroes have been sued because of the trouble they cause, and the government has
decided to prohibit superhero activities, similar to the Pixar film “The Incredibles."
The government has now requested you to create a classifier that can accurately identify
superheroes so that security agencies can monitor their actions.
Superheroes are quite uncommon.
There are only a few hundred of them in the world, and they all live in New York City.
So, your training dataset will look like this.
This dataset is actually “unbalanced”.
Unbalanced datasets are those in which the target variable contains more observations
in one class than the others.
You could make a dumb classifier that would label almost every person as “Normal”.
Even those Who are obviously Superheroes!
At the end of the day, the confusion matrix will look like this.
Note that only 3 samples are classified as “superhero”
Accuracy of this model is 98.7 percent!
Isn’t it great?
We can sell this dumb classifier to the government and receive billions of dollars!
Well, not so fast!
People at security agencies are not fools, and they won’t use accuracy as evaluation
metric on such unbalanced datasets, nor should you.
When the dataset is balanced, accuracy is beneficial.
If it isn't, we should look at other metrics to get a better picture of our model's performance.
One of the other metrics that we can use is recall (or sensitivity).
It measures the ratio of positive values that are correctly classified.
In our case, it is the number of superheroes that our dumb model correctly identified,
divided by the number of superheroes in the dataset.
As you can see, the high number of false negatives, which are the superheroes that are labeled
as “normal”, results in a low Recall.
Precision is another valuable metric.
It is the percent of positively classified samples that were actually correct.
In this example, we should divide the number of correctly classified superheroes by the
total number of people that our model classified as superheroes, Including poor normal people
who were labeled as superheroes.
As you can see, our model is performing poorly on this metric as well.
Now, let’s compare the accuracy, recall and precision of our model and see what each
number means.
Precision tell us how relevant our results are.
A high precision means when our model classifies a person as a superhero, we can trust it,
because most of the time that it positively labels someone as superhero, it is correct.
But as you can see, precision of our model is only 33 percent, which means, two times
out of three, it falsely accuses a person of being an unlawful superhero.
If we want to avoid causing troubles for ordinary people and catch superheroes with higher confidence,
we should consider better models with higher precision.
Recall shows how well our model “Recalls” the superheroes.
A high recall score means that our model can correctly identify most of the superheroes
in the dataset and won’t let them get away easily.
Unfortunately, our dumb model has a very bad recall score and only one superhero will get
caught out of 8.
If we want to capture as many superheroes as possible, we should focus on getting better
recall scores.
Accuracy gives us the percentage of correct classifications.
There are many cases that accuracy is a very good indication of how good our model is.
but When the dataset Is unbalanced, like our case of superhero dataset, accuracy is not
a good metric to use.
There is one last metric that I want to introduce in this video.
The final score that we cover in this video, is F1 score.
It is actually a harmonic mean of recall and precision.
We can use this metric when we don’t have a preference between better recall and better
precision and want our model to perform generally well on both metrics.
For our dumb classifier with precision of 33% and recall of 12%, the F1 score is 18.2%.
In this video, we first revisited the concept of classifiers.
Then, we talked about Confusion matrices; a clever way for figuring out how well our
model is working.
Finally, using our example of superhero detector, we became familiar with classification metrics
such as accuracy, recall, precision, and F1 score.