CHRISTOPHER POTTS: Welcome back, everyone.
This is part 2 in our series on methods and metrics.
We're going to be talking about classifier metrics.
I'm sort of assuming that the metrics I'll be discussing
are broadly familiar to us.
I think that's a chance for us to step back and be
reflective about what values these familiar metrics actually
encode.
Because that really is the name of the game here.
No matter what kind of task you're working on
or what the structure of your model
is like, it's just fundamentally true that different evaluation
metrics will encode different values, different goals
you have for your system, and different kinds of hypotheses
that you might be pursuing.
You can hear in that, that really fundamentally choosing
a metric is a crucial aspect to any kind of experimental work.
It's a fundamental step in how we operationalize
hypotheses in terms of data, and models, and model comparisons.
As a result, you should feel free, for whatever task
you're working on, to motivate new metrics or specific uses
of existing metrics, depending on what
your actual goals for your experiments actually are.
Relatedly, for established tasks,
you'll probably feel some pressure
to use specific well-established metrics.
But you should always, as a scientist,
feel empowered to push back if you feel that the accepted
metrics are not reflective of your hypothesis
or are distorting our notions of progress somehow.
Because remember, areas of research
can stagnate due to poor metrics.
And so we have to be vigilant.
We have to be on the lookout for cases
in which the metrics we've accepted
might be at odds with the actual goals we have
for the research we're doing.
Let's begin our discussion of classifier metrics
by talking about confusion matrices, a pretty fundamental
data structure for a lot of the calculations
that we'll perform.
So by convention, for my confusion matrices,
I'll have the actual labels going across the rows here.
And across the columns, I'll have the predictions
from some classifier model.
So you can see in this confusion matrix,
that there were 15 cases in which the model predicted
positive.
And the actual label was positive.
Where there-- whereas there are 10 cases where the actual label
was positive and the model predicted
negative, and so forth for the other values in this table.
I think that seems familiar.
It's something we can take for granted.
But we should remember that behind the scenes here,
a threshold was imposed in order to create
these categorical predictions.
By and large, classifier models that we use today
predict probability distributions over the labels.
And so in order to create an actual categorical prediction,
we decided, for example, that the label
with the maximum probability would be the true one.
And that was-- the result of that decision was
used to aggregate this table.
But, of course, different choices of that threshold
might give very different results.
And there might be contexts in which
we want to explore the full range
of probabilistic predictions.
That's something I'll return to at the end of the screencast.
Final note about this.
It can be helpful in the context of confusion matrices
to add a column for what's called support, which is simply
the number of actual true instances
that fall into each class.
So there are 125 positive instances
in this corpus, 35 negative, and over 1,000 that's
all in the neutral category.
And that's already illuminating about how specific metrics
might deal with that extremely imbalanced vector of support
values.
So let's start with accuracy-- by far,
the most famous and familiar of all the classifier metrics.
Accuracy is simply the number of correct predictions divided
by the total number of examples.
In terms of our confusion matrices,
that is just the sum of all the values
along the diagonal divided by the sum of all the values
that are in this table.
The bounds are 0 and 1, of course, with 0 the worst
and 1 the best.
In terms of the value encoded by accuracy,
I would say it's an attempt to answer the question,
how often is the system correct?
And that kind of feeds into the weaknesses here.
So the weaknesses are, first, there's no per class
notions of accuracy.
Not directly.
We just get a single holistic number.
And relatedly, there is just a complete failure
to control for class size.
So you can see, for example, in this confusion matrix,
that performance on the neutral class
will completely dominate the accuracy values.
And it's to the point, in this table,
where no matter how much progress
we make on the positive and negative classes
because they are so much smaller in terms of their support
for neutral, that progress is unlikely to be reflected
in our accuracy values.
And that's why if you return to the value encoded,
you can see that just at a raw fundamental level,
it is simply answering how often is the system correct?
Another thing to keep in mind is that for many classifier
models, the loss for those models
is what's called the cross-entropy loss.
It's also called the negative log-loss in Scikit-learn.
And that value is inversely proportional to accuracy.
The takeaway there is that even as we
might choose other metrics to compare models and evaluate
models, we should keep in mind that our classifiers themselves
are kind of engines for trying to maximize accuracy.
And so they are likely to inherit whatever properties,
and values, and strengths, and weaknesses
are inherent in the accuracy calculation, which
as we'll see, could be at odds with our actual values
for the system that we're developing.
And that kind of feeds nicely into precision, recall,
and F scores which are attempts to make up
for some of the weaknesses that you see in accuracy.
We'll start with precision.
This is a per class notion.
For a class k, it's the correct predictions
for k divided by the sum of all the guesses for k
that were made by your model.
So in terms of this confusion matrix,
if we focus on the positive class here,
the numerator is the number of correct predictions
for that class divided by the sum of all the values
that are in this column.
And for the negative class, we would repeat that.
The numerator would be 15, and we would sum over the column.
And finally, for neutral, the numerator would be 1,000.
And we would again sum over this column.
And that leads to this vector of precision values
that you see along the bottom here.
The bounds of precision are 0 and 1 approximately.
With 0 the worst and 1 the best.
There is an important caveat here though.
Precision is technically undefined in situations
where a model makes no predictions about a given
class.
Because in that situation, you're dividing by 0.
And that's technically undefined.
It is common practice to map those to 0,
but we should keep in mind that we
are making that extra decision.
The value encoded is a kind of conservative one.
We're going to penalize incorrect guesses
for a certain class.
So you can imagine that a failure mode there is to just
rarely guess a certain class.
That is the core weakness.
You can achieve high precision for a class k
simply by rarely guessing k.
So we'll obviously need to offset
that with some other pressure.
And by and large, the offset pressure is recall.
Recall is, again, a per class notion.
For a class k, it's going to be the correct predictions for k
divided by the sum of all the true members of k.
So now we're going to operate row--
rowwise.
We focus on the positive class.
Our numerator is 15, the number of true predictions
for the positive class, divided by the sum of all the values
along the rows.
That is all the true members of that class for positive.
That gives us a recall value of 0.12.
And we can repeat that for the other two rows.
The bounds of 0 and 1.
0 the worst and 1 the best.
The value encoded is a permissive one.
We want to penalize missed true cases.
We would like to make a lot of predictions about a class
in order to avoid leaving any out, so to speak.
And that leads into the core weakness.
We can achieve higher recall for a class k
simply by always guessing k.
Never mind the mistakes.
As long as we get all the actual cases into our predictions,
we're doing well by recall.
And you can hear in that that it's
important to offset this pressure by something else.
And that's standard precision.
And the way we offset these two pressures
is typically with F scores.
So F scores are a harmonic mean of the precision and recall
scores.
It's again a per class notion.
And it has this weighting value, beta.
If we want to evenly balance precision and recall,
then we set beta to 1.
So here's that confusion matrix again.
And along this column here, I've given the per class
F1 values here.
The bounds are 0 and 1 as before with 0
the worst and 1 the best.
And you can count on the fact that the F1 score for a class
will always fall between the precision and recall classes,
because it's a, kind of, an average.
It's the harmonic mean.
What's the value encoded?
The best way I can say this is that we're essentially
trying to answer the question, for a given class
k, how much do predictions for k align
with a true instance of k?
That is aligning with both precision and recall
as pressures.
And then we can use the beta value
to control how much weight we place
on precision versus recall.
What are the weaknesses of F scores?
Well, I can really think of two.
The first is that there's no normalization
for the size of the dataset because
of the way we use the denominators for the row
and column sums.
And relatedly, for a given class that we decide to focus on,
we actually ignore most of the data that's in the table.
Consider the fact that if we decided
to calculate the F1 score for the positive class,
we pay attention to these column values
and these are row values, but we completely
ignore these four values here.
They're just not involved in the calculation at all.
And as a result, the positive class F1 score
might give a distorted picture of what the model's predictions
are actually like in virtue of the fact
that they leave out so much of the data here as you can see.
Now because F scores are a per class notion,
I think, that's useful in the sense
that it gives us a perspective on each one of the classes
separately.
But for many kinds of model evaluations,
we need a summary number, a single number
that we can use to compare models and assess
overall progress.
So we're going to do some kind of averaging.
And I'd like to offer you three ways
that we might average these F scores.
Macro-averaging, weighted averaging, and micro-averaging.
And as you'll see, these encode quite different
values about how we want to think about the F scores.
Macro-averaging is a averaging that we've
done at various points throughout the quarter.
It is simply the arithmetic mean of all the
per category F1 scores.
So it's just the mean of the values along this column.
Its bounds are 0 and 1 with 0 the worst and 1 the best.
What value does encode?
Well, it's the same values that we get from F scores,
plus the additional and non-trivial assumption
that all of the classes are equal
regardless of size differences between them, right?
And that kind of feeds into the weaknesses here.
A classifier that does well only on the small classes
might not actually do well in the real world.
If you imagine counterfactually that for our given model here,
we had really outstanding F1 scores
for positive and negative and really low for neutral.
That might really be at odds with how this classifier would
behave in the world, assuming that most
of the examples that are streaming in
are in the neutral category.
Relatedly, a classifier that does well only on large classes
might do poorly on the small, but nonetheless vital
classes that are in our data.
And that just reflects the fact that very often in NLP, it's
the small classes that are the most precious, the ones that we
care about the most.
And we're not reflecting that kind of asymmetry in our values
by simply taking the average of all these F scores.
Weighted average F scores will give
a very different perspective on model performance.
In this case, we are again just going
to take an average of the F1 scores,
but now weighted by the amount of support
for each one of the classes.
That, again, has bound 0 to 1 with 0 the worst
and 1 the best.
The value encoded is the same as the values
that we get for the F scores, but now with the added
assumption that the size of the class as the amount of support
really does matter.
And that's going to feed into the weaknesses.
And the fundamental thing here is that large classes
will dominate.
Just as with accuracy, the larger our classes,
the more it's going to contribute
to our overall summary number.
And that can lead to the kind of problematic situation,
where the small classes are just not relevant for the evaluation
metric.
That could reflect your values because.
If what you really care about is raw rate
of correct predictions, you might
want to weight the larger classes more heavily.
But again, for many contexts in NLP
we really care about how much progress
we can make on the small, but nonetheless important classes.
And so in those contexts, weighted averaging
is probably not the right choice.
The final averaging scheme that I would like to consider
is micro-average F scores.
This will be very similar to weighted averaging of F1 scores
and is directly connected to accuracy.
The way this works is a little bit involved.
We start with this core confusion matrix.
And we're going to break it down into three smaller confusion
matrices, one per class.
So you can see this one on the left
here is for the positive class.
The yes'es are 15 and the no's are the sum of these two values
here along this row.
The no's are the 20, which is the sum of these two values.
And then there's no no categories,
for all the remaining data in this quadrant here.
We repeat that same procedure for the negative class
and for the neutral class.
And then we simply sum up those three smaller
tables into one big yes-no confusion matrix
and calculate the F1 scores per category.
That gives us two scores here.
One for yes and one for no.
The bounds on this are 0 and 1.
With 0 the worst and 1 the best.
The value encoded is really easy to state.
Macro-averaged F1 scores for the yes category
are equivalent to accuracy scores numerically.
So this is identical in terms of that metric.
And we have this additional problem now that--
well, we have the same kind of value reflected
as we have for the weighted F scores or for accuracy,
but now we have brought in an additional source
of uncertainty, which is we have a number for the yes
category and the no category.
And hence, no single summary number.
The convention in the literature is
to focus on the yes category, but that simply brings us back
to accuracy with a more involved calculation.
So that's obviously not very productive.
And I would say as a result here,
the two real choices that you want to make
are between macro-averaging and weighted averaging
of your F1 scores.
And again, that will come down to what
your fundamental values are and what
hypotheses you're pursuing.
The final point I want to make is that thus far, we
have operated in terms of the confusion matrix, which
involved imposing a threshold on probabilistic predictions
in order to create categorical values that we could then
compare with precision, and recall, and so forth.
Precision and recall curves offer a fundamentally different
perspective.
In this case, instead of imposing one threshold,
we'll take every possible value that's
predicted by our classifier to be a potential threshold,
and essentially create a bunch of confusion matrices
based on that successive series of thresholds.
And then we can plot the trade-offs between precision
here along the y-axis and recall along
the x-axis for all those different notions of threshold.
And that can be really illuminating in terms
of helping us see how our system trades precision
and recall against each other.
And help us find based on values that we
have about our problem and our goals,
what the optimal balance between precision and recall
actually is.
And then if you do need to summary
a number of this entire table, average precision,
which is implemented in Scikit-learn
is a standard way of summarizing the entire curve
with a single number without, though, imposing
that single threshold that was so much shaping
all of the previous metrics that we discussed.