in the last video I showed how to remove outliers from a data set using the IQR
method in this video I want to talk a little bit more in depth about outliers
and some other methods of outlier detection and mobile I'll go over three
methods of trimming outliers from data sets to get more accurate means the trim
mean method the standard z-score method and the modified z-score method hello
everyone welcome to the engineering tool box channel where I give you the tools
you need to solve real-world engineering problems so just like in the last video
I want to preface this with saying values determined as outliers aren't
always something that should be removed in fact pretty much the opposite this is
very important to understand anyone with a good understanding of stats that's
watching this video is probably yelling at their screen already but hopefully
I'm able to explain my case typically in statistics were told that outliers
should not be removed unless they can be proven to be erroneous data points these
are things that are either not possible like an adult weighing 12 pounds or
something like that or we can gather at least anecdotal evidence to prove that
the data point is invalid and this totally makes sense however from an
engineering perspective we're often trying to act on these statistics with
limited data time or ability to set up controlled experiments so if 9 out of 10
data points suggest one thing and the last looks to be an outlier I'm gonna
assume that's a special case and act on the 90 percent but just remember that by
removing outliers from a data sets we're making an assumption and it's not
necessarily correct as you'll see in this video there are different ways of
determining outliers that right there shows us that outliers are somewhat
loosely defined the method used to determine outliers directly affects what
is classified as one each does set and scenario is different and without
understanding the method used to determine outliers what we plan to do
with them and having an overall conceptual understanding of the data
itself we can end up making incorrect assumptions we should always ask
ourselves what the goal is and be careful not to introduce bias by picking
a certain method or just removing outliers just because it gives us the
numbers we want by the way if there's any statisticians out there I'd love to
get your input on this anyways let's dive in before we do this in Excel
it's quick run through each method first is the trimmed mean method this is not a
method for detecting outliers but it is one way of removing the potential
outlier values in order to give us a potentially more accurate mean it's very
important to note that this method could very easily remove not all
our values it works by taking a certain number of values or a certain percent
with a total count and removes them from the main equation in this example the
top three and bottom three values are being removed this method does not work
well for small data sets but it does work well on skewed or fat-tailed
distributions the assumption we're making here is that the data left after
trimming will be more normally distributed and more useful
approximation of the mean next the standard z-score method is a common way
to detect outliers in large normally distributed data sets
it works by calculating a mean and standard deviation for the entire data
set then calculating the z score from each data point then comparing those
z-scores to a certain threshold often times three two and a half or two
standard deviations from the mean can be considered outliers you can start to see
how our detection can be pretty dicey those bets will not work with small
sample size is definitely not under around 12 because the max standard
z-score for that sample size is around like three and it gets even smaller as
the sample size gets smaller last is the modified z-score which is very robust
because of the uses median values to calculate a modified z-score value for
that we use a formula where we need to find the ma D or median absolute
deviation now it's pretty hard to explain this when visually but I'll try
to explain it more when you get into Excel just for comparison let's look at
an example here the resulting means after removing the outliers each method
detected you can see that the values vary depending on the method used
you may have also noticed that essentially what's happening when we
remove outliers is the mean of the remaining values move closer to the
original median so you might ask why not just use the medium in the first place
depending on what you're trying to achieve this is certainly an option we
need to ask ourselves again what is the goal is it to approximate the normal
distribution of the data or just to understand where the central tendency is
if you're just trying to understand the central tendency where 50% of the values
are above and 50% or below then using the median is obviously the way to go
let's look at some simple examples where we might want to remove outliers to get
more representative means if we're trying to understand the distribution
and central tendency of a univariate data set or one variable we might want
to detect and remove outliers to gain a more realistic understanding of the
distribution in this example we can see that the mean is very different
depending on the inclusion of the outliers if my job were to create a
model to describe the interaction between two variables like in this
regression analysis I would more than likely exclude these outliers because
the resulting regression line does not fit the majority of the data points
again this is making a huge assumption that the data points aren't actually
representative of the interaction even though we may choose to exclude the
outliers we should probably follow up this analysis by implementing more
controls of the process and that brings me to my final very important note as
engineers we should always ask the question why are there outliers in the
first place not liar just the fact that there are outliers can tell us a lot
about the underlying process with a method of measurement we should always
try to investigate outliers and try to understand why they even exist often
time there's a ton of value in this investigation now you might find that
there was just a simple error or you might find that there is something
fundamentally wrong with the process you're measuring the cool thing about
being an engineer is usually we aren't trying to simply understand data just
for the sake of understanding it but actually understand the data so we can
change the underlying process that is driving it that's what makes outlier
detection so valuable all right I admit I was all over the place in this one
probably because the topic of outliers can be so nebulous and it's dependent on
the situation and the outcomes were looking for I felt like I was talking in
circles at times but sometimes that's just the way my mind works now that we
hopefully have a better understanding of outliers why detecting them is important
and why we might want to remove them I'll go over how we can actually do this
using Excel in the next video anyway I hope you found this video valuable and
if not at least interesting or entertaining if you did make sure to
LIKE and subscribe to get my weekly videos if you have any questions or
there's anything I can go into more depth on make sure to let me know and
maybe if you're lucky I'll make a video about it and as always thank you so much
for watching