In statistics, an outlier is an observation point that is distant from other observations.
An outlier may be due to variability in the measurement or it may indicate experimental
error; the latter are sometimes excluded from the data set.
Outliers can occur by chance in any distribution, but they are often indicative either of measurement
error or that the population has a heavy-tailed distribution. In the former case one wishes
to discard them or use statistics that are robust to outliers, while in the latter case
they indicate that the distribution has high kurtosis and that one should be very cautious
in using tools or intuitions that assume a normal distribution. A frequent cause of outliers
is a mixture of two distributions, which may be two distinct sub-populations, or may indicate
'correct trial' versus 'measurement error'; this is modeled by a mixture model.
In most larger samplings of data, some data points will be further away from the sample
mean than what is deemed reasonable. This can be due to incidental systematic error
or flaws in the theory that generated an assumed family of probability distributions, or it
may be that some observations are far from the center of the data. Outlier points can
therefore indicate faulty data, erroneous procedures, or areas where a certain theory
might not be valid. However, in large samples, a small number of outliers is to be expected.
Outliers, being the most extreme observations, may include the sample maximum or sample minimum,
or both, depending on whether they are extremely high or low. However, the sample maximum and
minimum are not always outliers because they may not be unusually far from other observations.
Naive interpretation of statistics derived from data sets that include outliers may be
misleading. For example, if one is calculating the average temperature of 10 objects in a
room, and nine of them are between 20 and 25 degrees Celsius, but an oven is at 175
°C, the median of the data will be between 20 and 25 °C but the mean temperature will
be between 35.5 and 40 °C. In this case, the median better reflects the temperature
of a randomly sampled object than the mean; naively interpreting the mean as "a typical
sample", equivalent to the median, is incorrect. As illustrated in this case, outliers may
be indicative of data points that belong to a different population than the rest of the
sample set. Estimators capable of coping with outliers
are said to be robust: the median is a robust statistic, while the mean is not.
Occurrence and causes In the case of normally distributed data,
the three sigma rule means that roughly 1 in 22 observations will differ by twice the
standard deviation or more from the mean, and 1 in 370 will deviate by three times the
standard deviation for details. In a sample of 1000 observations, the presence of up to
five observations deviating from the mean by more than three times the standard deviation
is within the range of what can be expected, being less than twice the expected number
and hence within 1 standard deviation of the expected number – see Poisson distribution,
and not indicative of an anomaly. If the sample size is only 100, however, just three such
outliers are already reason for concern, being more than 11 times the expected number.
In general, if the nature of the population distribution is known a priori, it is possible
to test if the number of outliers deviate significantly from what can be expected: for
a given cutoff of a given distribution, the number of outliers will follow a binomial
distribution with parameter p, which can generally be well-approximated by the Poisson distribution
with λ = pn. Thus if one takes a normal distribution with cutoff 3 standard deviations from the
mean, p is approximately .3%, and thus for 1,000 trials one can approximate the number
of samples whose deviation exceeds 3 sigmas by a Poisson distribution with λ = 3.
Causes Outliers can have many anomalous causes. A
physical apparatus for taking measurements may have suffered a transient malfunction.
There may have been an error in data transmission or transcription. Outliers arise due to changes
in system behaviour, fraudulent behaviour, human error, instrument error or simply through
natural deviations in populations. A sample may have been contaminated with elements from
outside the population being examined. Alternatively, an outlier could be the result of a flaw in
the assumed theory, calling for further investigation by the researcher. Additionally, the pathological
appearance of outliers of a certain form appears in a variety of datasets, indicating that
the causative mechanism for the data might differ at the extreme end.
Identifying outliers There is no rigid mathematical definition
of what constitutes an outlier; determining whether or not an observation is an outlier
is ultimately a subjective exercise. There are various methods of outlier detection.
Some are graphical such as normal probability plots. Others are model-based. Box plots are
a hybrid. Model-based methods which are commonly used
for identification assume that the data are from a normal distribution, and identify observations
which are deemed "unlikely" based on mean and standard deviation:
Chauvenet's criterion Grubbs' test for outliers
Peirce's criterion
It is proposed to determine in a series of observations the limit of error, beyond which
all observations involving so great an error may be rejected, provided there are as many
as such observations. The principle upon which it is proposed to solve this problem is, that
the proposed observations should be rejected when the probability of the system of errors
obtained by retaining them is less than that of the system of errors obtained by their
rejection multiplied by the probability of making so many, and no more, abnormal observations.
from A Manual of Astronomy 2:558 by Chauvenet.)
Dixon's Q test ASTM E178 Standard Practice for Dealing With
Outlying Observations Mahalanobis distance and leverage are often
used to detect outliers, especially in the development of linear regression models.
Other methods flag observations based on measures such as the interquartile range. For example,
if and are the lower and upper quartiles respectively, then one could define an outlier to be any
observation outside the range:
for some nonnegative constant . Other approaches are distance-based and density-based,
and all of them frequently use the distance to the k-nearest neighbors to label observations
as outliers or non-outliers. Modified Thompson Tau test
The modified Thompson Tau test is a method used to determine if outlier exists in a data
set. The strength of this method lies in the fact that it takes into account a data set’s
standard deviation, average and provides a statistically determined rejection zone; thus
providing an objective method to determine if a data point is an outlier.
How it works: First, a data set's average is determined. Next the absolute deviation
between each data point and the average are determined. Thirdly, a rejection region is
determined using the formula: ; where is the critical value from the Student t distribution,
n is the sample size, and s is the sample standard deviation. To determine if a value
is an outlier: Calculate δ = |(X - mean(X)) / s|. If δ > Rejection Region, the data point
is an outlier. If δ ≤ Rejection Region, the data point is not an outlier.
The modified Thompson Tau test is used to find one outlier at a time. Meaning, if a
data point is found to be an outlier, it is removed from the data set and the test is
applied again with a new average and rejection region. This process is continued until no
outliers remain in a data set. Some work has also examined outliers for nominal
data. In the context of a set of examples in a data set, instance hardness measures
the probability that an instance will be misclassified. Ideally, instance hardness would be calculated
by summing over the set of all possible hypotheses :
Practically, this formulation is unfeasible as is potentially or infinite and calculating
is unknown for many algorithms. Thus, instance hardness can be approximated using a diverse
subset :
where is the hypothesis induced by learning algorithm trained on training set with hyperparameters
. Instance hardness provides a continuous value for determining is an instance is an
outlier instance. Working with outliers
The choice of how to deal with an outlier should depend on the cause.
Retention Even when a normal distribution model is appropriate
to the data being analyzed, outliers are expected for large sample sizes and should not automatically
be discarded if that is the case. The application should use a classification algorithm that
is robust to outliers to model data with naturally occurring outlier points.
Exclusion Deletion of outlier data is a controversial
practice frowned on by many scientists and science instructors; while mathematical criteria
provide an objective and quantitative method for data rejection, they do not make the practice
more scientifically or methodologically sound, especially in small sets or where a normal
distribution cannot be assumed. Rejection of outliers is more acceptable in areas of
practice where the underlying model of the process being measured and the usual distribution
of measurement error are confidently known. An outlier resulting from an instrument reading
error may be excluded but it is desirable that the reading is at least verified.
In regression problems, an alternative approach may be to only exclude points which exhibit
a large degree of influence on the parameters, using a measure such as Cook's distance.
If a data point is excluded from the data analysis, this should be clearly stated on
any subsequent report. Non-normal distributions
The possibility should be considered that the underlying distribution of the data is
not approximately normal, having "fat tails". For instance, when sampling from a Cauchy
distribution, the sample variance increases with the sample size, the sample mean fails
to converge as the sample size increases, and outliers are expected at far larger rates
than for a normal distribution. Set-membership uncertainties
A set membership approach considers that the uncertainty corresponding to the ith measurement
of an unknown random vector x is represented by a set Xi. If no outliers occur, x should
belong to the intersection of all Xi's. When outliers occur, this intersection could be
empty, and we should relax a small number of the sets Xi in order to avoid any inconsistency.
This can be done using the notion of q-relaxed intersection. As illustrated by the figure,
the q-relaxed intersection corresponds to the set of all x which belong to all sets
except q of them. Sets Xi that do not intersect the q-relaxed intersection could be suspected
to be outliers.
Alternative models In cases where the cause of the outliers is
known, it may be possible to incorporate this effect into the model structure, for example
by using a hierarchical Bayes model or a mixture model.
See also Anomaly detection
Anomaly time series Robust statistics
Robust regression Box plot
King effect Studentized residual
Winsorising Data transformation
Local Outlier Factor References
ISO 16269-4, Statistical interpretation of data — Part 4: Detection and treatment of
outliers Strutz, Tilo. Data Fitting and Uncertainty
- A practical introduction to weighted least squares and beyond. Vieweg+Teubner. ISBN 978-3-8348-1022-9.
External links Renze, John, "Outlier", MathWorld.
Balakrishnan, N.; Childs, A., "Outlier", in Hazewinkel, Michiel, Encyclopedia of Mathematics,
Springer, ISBN 978-1-55608-010-4 Grubbs test described by NIST manual
how to detect univariate outliers, how to detect multivariate outliers and how to deal
with outliers