There's a bit new terminology in this video - three more terms to go.
First is the concept of an outlier.
There isn't a particular way to test for an outlier or define it.
Paraphrasing the words of a US Supreme Court judge: "you know one when you see one".
The best way I can come up with to define an outlier is to state it as being "a point
that is unusual, given the context of the surrounding data".
In the first vector here, the number 4024 is nothing special.
In the second vector, that same value stick outs dramatically.
Here's a different example: on this scatter plot, we can see an outlier with a square point.
From the viewpoint of only the x-axis though, a value of 12 is right
in the middle of the distribution.
From the viewpoint of the y-axis, a value of 70 is high, but not extreme.
Jointly however, from the context of both the x- and y-axes, it is an outlier.
That's why we can't create rules to simply define an outlier.
Building on the concept of an outlier is the need and the use of the median.
We saw the median being used earlier when we considered box plots.
When Statistics Canada reports income data, they talk about "median income".
Why don't they report the mean?
You intuitively know that the few very well-paid people
in the population will bias, or cause a skew in the data.
It will make the mean value higher, much higher than the "average" person earns.
The median is a reliable indicator of the average,
it's a reliable measure of central tendency.
It is not affected by a few data points that could skew it.
In fact, the mean is a very unreliable estimator:
all it takes is one single data point to skew that mean.
Let's go back to the earlier vector.
The number 4024 is an outlier.
The mean of that vector is 440.4.
That's a very bad indication of the average; you can just see that visually from the numbers.
The median however is 56.5: it is not influenced by the outlier at all.
In fact, you would have to replace just under half the data
with outliers before the median breaks down as an estimate of location.
We say the median has a breakdown of 50%.
The mean on the other hand has a breakdown of 1/n.
Only a single outlier contaminates the mean
and can make it unreliable as a measure of centrality.
I want to challenge you: in your work, use the median,
instead of the mean, as a robust estimator.
It is tough: we are so used to using the average.
Historically, the median took a little longer to calculate,
because you had to sort the data first.
But that reason doesn't exist anymore; we have ample computing power now to calculate medians.
Related to the median is the MAD, the Median Absolute Deviation.
It is a robust estimator for the standard deviation
and it is defined exactly as the name suggests.
First calculate the deviations from a centre point, and of course,
you should use the median to estimate that centre.
Once you have those deviations, calculate the absolute values of them.
And then finally, take the median of all those absolute deviations.
The constant, c = 1.4826, is a constant
that makes the MAD consistent with the standard deviation.
Because the MAD uses the median to calculate the spread,
it is as reliable as the median is, with a breakdown of 50% as well.
The MAD is harder to use though in everyday life:
many software tools don't have it as an option.
And you have to be careful, not all software uses the 1.4826 factor,
and sometimes the software packages implement it in an inconsistent way.
So always read the help file for the function.
Lastly, lets go back and look at that vector of data again.
The standard deviation calculated for it is 1260, that seems like a very unrealistic spread.
The MAD however gives a better idea of the spread: 57 units.
It is not affected by that outlier.