paper “How to avoid machine learning pitfalls: a guide for academic researchers”
This paper aims to describe some of the most typical errors made while employing machine
learning techniques, as well as how to prevent them. It focuses on topics that are particularly
important in academic research, such as the necessity to conduct fair comparisons and
reach valid conclusions. This guide was prepared by a professor and
focuses on lessons learned while doing ML research in academia
and mentoring machine learning research students. It's mostly intended towards machine learning
noobs, such as students and other researchers who are just getting started in the area;
like myself! This article has helpful hints for each step
of the machine learning process, which I will attempt to explain in this video.
Before you start to build models Although it may seem appealing to rush into
creating cool models with a few lines of code, But you should take time to think about the
goals of a project, comprehend the data,
and understand what has already been done in your field. Otherwise, you’ll have a
hard time for choosing the right model and publishing your results.
Do take the time to understand your data If you train your model using bad data, then
you will most likely generate a bad model: a process known as garbage in garbage out.
Always begin by making sure your data makes sense. Do some exploratory data analysis.
Look for missing or inconsistent records. Don’t look at all your data
As you look at data, it is quite likely that you will spot insights that guide your modelling.
It is important that you do not make untestable assumptions that will later feed into your
model. To ensure this is the case, you should avoid looking closely at any test data in
the initial analysis stage. Do make sure you have enough data
If you don't have enough data, then it may not be possible to train a model that generalises.
This depends on the signal to noise ratio in the data set. If the signal is strong,
then you can get away with less data; if it's weak, you need more data.
If you can't collect more data, cross-validation can help you make better use of what you have.
You can use data augmentation techniques to boost small data sets. Data augmentation is
also useful in situations where you have limited data in certain parts of your data set.
If you just have a little amount of data, you'll probably need to keep the complexity
of your machine learning models to a minimum. This is an effective method of avoiding overfitting
tiny data sets with too many parameters. Do talk to domain experts
Domain experts can help you understand which problems are useful to solve. They can also
help you choose the most appropriate feature set and ML model to use. Failing to consider
the opinion of domain experts can lead to projects which don't solve useful problems,
or which solve them inappropriately. Do survey the literature
You're probably not the first person to throw ML at a particular problem domain. It's important
to understand what has and hasn't been done previously. To ignore previous studies is
to potentially miss out on valuable information. Do a literature review before you start work,
or you'll be left scrambling. Do think about how your model will be deployed
While many academic studies are not intended to produce models that will be used in the
real world, in many other studies the eventual goal is to produce an ML model that can be
deployed in a real world situation. If you want to deploy your model in the real
world, there are some things you should think about before getting your hands dirty creating
the ML model. For example, If a model is going to be deployed
in a resource-limited environment, it may place limitations on the complexity of the
model. If there are time constraints, for example, a classification of a signal is required
within milliseconds, this also needs to be taken into account.
How to reliably build models Given that with libraries like scikit learn,
you can implement ML models in several lines of code,
You might get lost in a disorganized mess of experiments that’s hard to justify and
hard to write up. As a result, it is critical to approach model
creation in an organized manner, ensuring that data is used appropriately, and giving
enough thought to model selection. Don’t allow test data to leak into the training
process A common problem is allowing information about
data to leak into the configuration, training or selection of models. When this happens,
the data no longer provides a reliable measure of generality. This is a common reason why
published ML models often fail to generalize to real world data.
There are a number of ways that information can leak from a test set. The best thing to
prevent these issues is to partition off a subset of your data right at the start of
your project. This way, you can only use this independent test set once to measure the generality
of a single model. Do try out a range of different models
There's no such thing as a single best ML model for every problem. The No Free Lunch
theorem shows that no ML approach is any better than any other when considered over every
possible problem. Given that modern Python ML packages make
it easy to try out various models with very little code modifications, there's no reason
not to do so and see which one works best for you.
Don’t use inappropriate models Modern machine learning packages make it simple
to apply inappropriate models to your data. Don't choose a model based on its novelty:
old, well-established models frequently perform better than new ones. For example, If you
just have a little amount of data or the underlying pattern is basic, a deep neural network isn't
the best option. Do optimize your model’s hyperparameters
Many models have hyperparameters, which are numbers that affect the model's configuration.
The kernel function in an SVM and the number of trees in a random forest are examples of
these. It's preferable to utilize a hyperparameter optimization strategy, which is much easier
to justify when writing up. When you have a limited number of parameters, basic techniques
like random search and grid search are useful. If this isn't the case, AutoML approaches
can be utilized to improve the choice of model and its hyperparametres.
Do be careful where you optimize hyperparameters and select features
When carrying out hyperparameter optimization and feature selection, it is important to
treat them as part of model training. A particularly common error is to do feature selection on
the whole data set before model training begins. This will result in information leaking from
the test set into the training process. You should use the same data for training as well
as model optimization or feature selection. One way to do so is called Nested cross-validation,
which is a kind of cross-validation that includes hyperparameter optimization and feature selection
as an additional loop within the main cross-validation loop.
How to robustly evaluate models You must have valid results from which you
may draw trustworthy conclusions in order to contribute to development in your profession.
It's all too simple to evaluate machine learning models unfairly, muddying the waters of academic
progress. As a result, you should take testing and reporting very seriously.
Do use an appropriate test set Using a test set to measure the generality
of an ML model is important. The data in the test set should not overlap with the training
set and it should be representative of the wider population.
For example, if all of our data, including the test set, is collected on a sunny day,
then our model will probably not generalize on other situations.
So, our test set should contain some photos taken in other circumstances as well.
Do use a validation set It's critical not to include the test set
in the training process directly. Instead, performance should be measured using an independent
validation set. This is a collection of samples that are not directly involved in training,
but are used to guide training. Another advantage of having a validation set
is that you can conduct early stopping, which means you can stop training as soon as the
validation set's error is higher than it was the last time it was checked.
Do evaluate a model multiple times Many ML models are unstable. That is, if you
train them multiple times, or if you make small changes to the training data, then their
performance varies significantly. For this reason, it is common to carry out multiple
evaluations of a model. Cross validation is particularly popular, and comes in numerous
varieties. Do save some data to evaluate your final model
instance Evaluation of the potential of a general model
differs from that of a specific model instance. Cross-validation is beneficial in the first
case, but not so much in the second. You may believe that its test fold score is a good
indicator of its performance, but it isn't. If you have enough data, set some aside and
use it just once to produce an unbiased estimate of the final model instance.
Don’t use accuracy with imbalanced data sets
Be cautious about the metrics you choose to evaluate your machine learning models. The
most widely used measure in classification models, for example, is accuracy. As long
as your classes are balanced, this works great, but many data sets aren't. For instance, consider
a data set in which 90% of the samples belong to one class and 10% of the samples belong
to another. Despite being utterly useless, a binary classifier that always returns the
first class, regardless of its input, would have a 90% accuracy. The Cohen's Kappa Coefficient
or the Matthews Correlation Coefficient would be preferred in this case.
How to compare models fairly If you compare models unjustly, you risk leading
other researchers astray. Comparing models is the foundation of academic research, yet
getting it correctly is surprisingly challenging. Make sure you evaluate different models in
the same context and look at things from different perspectives.
Don’t assume a bigger number means a better model
Bigger doesn't always mean better in machine learning research, as it does in many other
aspects of life. There are a number of reasons why a greater number does not necessarily
suggest a better model. Small variations in performance may be due to whether or not the
models were trained or evaluated on different partitions of the same data set. Another factor
that contributes to unfair comparisons is the inability to apply the same level of hyperparameter
optimization to all models when comparing them. To be sure of a fair comparison, you
should implement all of the models you're comparing from scratch, optimize them to the
same degree, and run several tests. Do use statistical tests when comparing models
There are two categories of tests for comparing individual ML models. The first is used to
compare individual model instances, such as two trained decision trees. For example, McNemar’s
test is a fairly common choice for comparing two classifiers, and works by comparing the
classifiers' output labels for each sample in the test set.
The second category of tests are used to compare two models more generally.
Student's T test is a good choice for this kind of comparison, but it assumes that the
data is distributed normally. Mann-Whitney's U test is safer, since it does not assume
that the distributions are normal. Do correct for multiple comparisons
Multiple pairwise tests can lead to overly-optimistic interpretations of significance. Each time
you carry out a comparison between two models using a statistical test, there's a probability
that it will discover significant differences where there aren't any. This is known as the
multiplicity effect, and is an example of a broader issue in data science known as data
dredging or p-hacking. To address this problem, you can apply a correction for multiple tests
- the most common approach is the Bonferroni correction.
Don’t always believe results from community benchmarks
The use of benchmark data sets to test new machine learning models has become common.
The idea is that comparisons will be more transparent because everyone is using the
same data to train and evaluate their models. However, there are several significant disadvantages
to this technique, and it might result in too optimistic outcomes. In fact, evaluating
many models on the same test set increases the likelihood that the best model just over-fits
the test set and does not necessarily generalize any better.
Do consider combinations of models Different ML models explore different trade-offs,
and by combining them you can sometimes compensate for the weaknesses of one model by using the
strengths of another. Such composite models are known as ensembles, and the process of
generating them is known as ensemble learning. How to report your results
Academic research is not about self-promotion, but rather about making a contribution to
knowledge. You must give a comprehensive account of your effort, including both what worked
and what didn't. Trade-offs are common in machine learning, and you should try to reflect
this with a detailed and thoughtful approach. Do be transparent
Be transparent about what you've done, and share your results. It's good practice to
share your models in an accessible way. For instance, if you used a script to implement
all your experiments, then share the script when you publish the results. This makes it
easier for people to compare models, and ensure fair comparisons.
Do report performance in multiple ways It is possible to provide a more complete
overview of your model's performance by combining several data sources. Reporting multiple metrics
for each data set is a smart idea. Alternative metrics might provide you different views
on the outcomes and make your job more transparent. If you utilize accuracy, for example, it's
a good idea to include measures that are less susceptible to class imbalances.
Don’t generalize beyond the data It's important not to present invalid conclusions,
since this can lead other researchers astray. A common mistake is to make general statements
that are not supported by the data used to train and evaluate models. Don't overplay
your findings, and be aware of their limitations. Do be careful when reporting statistical significance
Some statistical tests are conservative, underestimating significance, whereas others are liberal,
overestimating significance. Some other factor to examine is if the distinction between two
models is significant. Even when the difference in performance is little, if you have enough
samples, you may always detect significant differences.
Do look at your models Many authors just report the trained model's
performance metrics without commenting on what it really learned. However, the goal
of research is to develop information and insight and then share it with others in the
field. So take a look inside your models and attempt to figure out how they make decisions.
It might also be helpful to give visualizations of your models for relatively simple models
like decision trees. To extract information from complicated models, such as deep neural
networks, explainable AI approaches could be used. They won't tell you exactly what
the model is doing, but they could provide some useful insights.
Some of the points I made in this video may be incorrect, or at the very least debatable.
Almost often, the theory of how to perform Machine learning lags behind the practice.
What we believe to be true today may not be true tomorrow. You have to approach Machine
learning with an open mind, and the humility to accept you don't know everything.