CHRISTOPHER POTTS: Welcome, everyone.
This is part 4 in our series on methods and metrics.
We're going to be talking about how
we organize datasets for the purposes of conducting
evaluations in NLP.
Let's begin with the classic train/dev/test split.
This is a very common format for datasets
in our field, especially for the very large
publicly available ones.
And it's really good in the sense
that in releasing datasets with these splits predefined,
we do ensure some consistency across the different
evaluations that people run.
It does presuppose that you have a fairly large datasets
because after all, right from the get go,
you are setting aside a whole lot of examples in the dev
and test splits that you can't use
at all to train your systems.
So even though your system might benefit from those examples,
you can't use them in that context.
They can be used only for evaluations.
You're just giving up a lot of potentially useful examples.
As we've discussed many times, we're all on the honor system
when it comes to that test set.
It's distributed as part of the data set,
but it has a privileged status.
The test set can be used only once all of system development
is complete, and then you do a single evaluation on the test
set and report that number completely hands off.
This is vital for our field because it's the only way
that we can even hope to get a true picture of how
our systems are truly generalizing to new examples.
That said, the downside of having predefined
train/dev/test splits is that inevitably everyone
is using those same dev and test sets.
And what that means is that over time
as we see consistent progress on a benchmark test,
we're taking that same measurement
on that same test set.
And it can be hard to be sure whether we're
seeing true progress on the underlying
task or the result of a lot of implicit lessons
that people have learned about what works
and what doesn't for that particular test set.
And that's true even if everyone is obeying that honor code
and using the test set only for truly final evaluations.
Nonetheless, information can leak out,
and we might start to mistake true progress when
we're actually just seeing progress
on that particular test set.
And I think the only way that we can really
combat this is by continually setting
new benchmark tests for ourselves with new test sets
so that we see how systems perform
in truly unseen environments.
As you leave NLP, it's common to find datasets
that don't come with that predefined train/dev/test
split, and that poses some methodological questions
for you.
This is especially true for small public data
sets that you see out there.
And this poses a challenge for assessment.
For robust comparisons, you really
have to run all your models using your same assessment
regime.
That is, the same splits.
And that's especially important if the dataset is small
because, of course, in a small dataset,
you're probably going to get more
variance across different runs.
And this can make it really hard to compare
outside of the experimental work that you're doing.
If someone has published the results of some random 70/30
train-test split, unless you can reconstruct exactly the splits
that they used, it might be unclear
whether you're doing a true apples-to-apples comparison.
So that's something to keep in mind.
And it does mean that if you can for your own experiments,
you might impose a split right at the start of your project.
This is probably feasible if the data set is large.
And what it will mean is that you have a simplified
experimental setup, and you have to do less hyperparameter
optimization just because there are
fewer moving parts in your underlying experimental setup.
It does presuppose that you have a pretty large data set
because, as I said before, you have
to give up a whole bunch of examples to dev and test.
But it will simplify other aspects of your project
if it's feasible.
For small datasets though, imposing a split
might leave too little data, leading
to highly variable performance.
And in that context, if that's the kind of behavior
that you observe, you might want to move
into the mode of cross-validation.
So cross-validation, in this context
we take a set of examples, say our entire data set,
and we partition them into two or more train-test splits.
And you might do that repeatedly and then
average over the results of evaluations on those
splits in some way to give a holistic summary of system
performance.
And in that way, even as those numbers vary--
they might have a lot of variance--
we're still getting in the average,
we hope, a pretty reliable measure
of how the system performs in general on the available data.
And I'm going to talk about two ways
to do cross-validation, each with its own strengths
and weaknesses.
Let's begin with what I call random splits here.
So under the random splits regime, you take your dataset.
And let's say k times, you shuffle it, and you split it.
And you have t% for train then probably the rest
left out for test.
And on each one of those splits, you
conduct some kind of evaluation, get back your metrics.
And then at the end of all these k evaluations,
you probably average those metrics
in some way to give a single summary number
for system performance.
In general, but not always, when we do these splits,
we want them to be stratified in the sense
that the train and test splits should have approximately
the same distribution over the classes in the underlying data.
But I have been careful to say that this is not always true.
There could, for example, be contexts
in which you would like your test set to stress test
your system by having a very different distribution, maybe
an even distribution, or one that's heavily skewed
towards some of the smaller but more important classes.
And that will pose a challenge for train-test regimes
because the system's experiences at train time
will be different in this high level distributional sense
from what it sees at test time.
But that, of course, might be part
of what you're trying to pursue as part
of your overall hypothesis.
The trade-offs for this kind of cross-validation-- the good
is that you can create as many splits
as you want without having this impact the ratio of training
to testing examples, right, because k times-- we're just
going to do a random split.
And it can be consistent that we do independent
of k 70% train, 30% test or 50-50,
or whatever we decide we want that's
independent of the number of splits that we set.
The bad of this, of course, is that there's no guarantee
that every example will be used the same number of times
for training and for testing.
And for small data sets, this could, of course,
be a concern because you might be introducing
unwanted correlations across the splits in, for example,
never having certain hard examples be
part of your test set just as a matter of chance.
So that's something to keep in mind.
But of course, for very large data sets,
it's very unlikely that you'll be
susceptible to the bad part of this,
and then you do get a lot of the benefits
of the freedom of being able to run lots of experiments
with a fixed train-test ratio.
And of course, as usual, scikit has a lot
of tools to help you with this.
So I've just given some classic examples down
here from the model_selection package.
You might import ShuffleSplit, StratifiedShuffleSplit.
And of course, train_test_split is a useful utility
for very quickly and flexibly creating splits of your data.
And I make heavy use of these throughout my own code.
The second reason for cross-validation
that I'd like to discuss is K-folds cross-validation.
Here, the method is slightly different.
We're going to take our dataset and split it into three folds,
in this case for three-fold cross-validation.
You could, of course, pick any fold number that you wanted.
And then, given that it's three-fold cross-validation,
we're going to conduct three experiments-- one where fold 1
is used for testing and 2 and 3 are merged together
for training, the second experiment where we hold out
fold 2 for testing and the union of 1 and 3
is used for training, and then finally a third experiment
where fold 3 is used for testing and folds 1 and 2
are concatenated for the train set.
The trade-offs here are slightly different from the trade-offs
for random splits.
So the good of this is that every example
appears in a train set exactly k minus 1 times
and in a test set exactly once.
We have that guarantee in virtue of the fact
that we use a single split over here
to conduct our three experimental paradigms.
The bad of this, of course, can be really difficult.
The size of K is determining the size of the training test
split.
But just consider that for three-fold cross-validation,
we're going to use 67% of the data for training
and 33 for testing.
But if three experiments is not enough, if we want 10 folds,
the result of that will be that we
use 90% of our data for training and 10% for testing.
And the bottom line is that those
are very different experimental scenarios
from the point of view of the amount of training data
that your system has and probably the variance that you
see in testing because of the way
you're changing the size of the test set.
Whereas for the random splits that we just discussed,
we have an independence of the number of folds
and the percentage of train and test examples
that we're going to have.
And that can be very freeing, especially for large data
sets where the value up here in the good
is really less pressing.
And, again, scikit-learn has lots of tools for this.
And I've actually just given a sample of them here.
You have KFold, StratifiedKFold, and then cross_val_score
is a nice wrapper utility that will, again, give you
flexible access to lots of different ways
of conceptualizing K-folds cross-validation.