friends in this video we will try to
understand what is
normal distribution or gaussian
distribution these two are terms for the
same thing
we'll also understand what is z-score
and we'll do some python coding to
understand
how exactly these concepts are used in
the field of data science and machine
learning so let's get started
let's say you want to do data analysis
on people's
height data set you would be wondering
what kind of analysis you can do
say you are a data scientist working for
a clothing store and
to produce the clothes of certain size
the clothing company has assigned you
this task as a data scientist you want
to do some analysis on people's heights
now when you plot these heights on a
histogram it will look like this
what is histogram histogram is a
frequency distribution for example
you have three data samples between
height
five and 5.5 so what are those samples
well
5.1 5.2 5.4
so you are just plotting the height and
the counts
of those samples on this simple chart
which is known as a histogram
and when you draw a curve that passes
through this histogram
this curve looks like a bell you know
you might have seen a bell
in a church or a temple and that's why
this curve is known as a bell curve so
this is what
normal distribution is in normal
distribution
you will have most of your data samples
around average value and then
you will have some data samples
which are far away from average on left
and right hand side
for example you have some people whose
heights are
six or maybe around seven feet you know
you'll have
very like small percentage of people
whose height will be around seven
similarly you will have small percentage
of people whose height will be let's say
less than four feet so the idea is
in nature we find many data sets
who which follow the normal distribution
for example
let's say you are looking at the prices
two bedroom prices
of apartment in Bangalore city most of
the apartments
on average they cost around 90 lakh
rupees
so you see let's say i took samples of
few thousand
property prices and you will see like
for 90 lakh rupees i have around 280
data samples
similarly you will have very few data
samples
whose high whose price will be on a
higher
higher end okay and very few data
samples whose height will be
on a lower end so most of the values
will be
centered around average and then you as
you go
far away from the average the number of
data samples reduces
you see the similar behavior when you
are examining
test scores for a given classroom let's
say you
took mathematics test you know and the
test
is giving you score out of 100 let's say
your maximum score is 85
you will find very few people who will
have
high marks and very few people who will
have very low marks
most of the people will have marks in an
average range
another example is employee performance
when they validate employees performance
majority of the employees will have
performance in the average category you
will have few best performer
and few low performers so we naturally
see
normal distribution in many data sets
and therefore
for data scientist and machine learning
engineers
knowing normal distribution is very very
important now you'll ask me okay
i understood normal distribution how can
i
use this in my real life how can i use
this in data analysis
the classical use case is during
data cleaning process you can use normal
distribution
and standard deviation for outlier
removal
unless i use the same data set but i
have added a new data point
Smith whose height is nine feet now
outlier is a data point which
which has a value that is very different
from
from your average values here you know
people's height are
usually around six feet five feet but
you don't see a person
with height nine feet okay so these kind
of outliers
can occur because of an error in data
collection process or you know in data
transmission process
or sometimes you might have
valid data points like you you can
have a person whose height
i think the person whose height was
maximum in the history
he had a height of 8.7
feet or something so you can have a
valid data point as well
but when you plot them on a histogram
you can clearly see those data points
kind of stand out
they are very far away from your normal
regular data points
during data science process you want to
treat the outlier
and by treating the outlier i mean you
either want to remove them
or you want to apply some other methods
to treat them
if you don't treat outliers it will
create problems
in your data science process in your
machine learning process you know
your machine learning model might get
skewed because of the presence of these
outliers so it's important that
you treat the outliers either by
removing them or applying some kind of
transformation
now here i have a very simple data set
when i have a data set which has less a
million data points
just by looking at the data points i
won't know which are outliers
so you need some kind of formula or you
need to apply some
math to figure out those outliers so
what formula that is
well you need to first understand what
is standard deviation
i made a very short video on that so
please
go to youtube search for code basic
standard deviation
watch that eight minute video once you
have understanding of standard deviation
now i will explain uh how statisticians
use standard deviation to remove
outliers
so here i have same histogram uh the
the bell curve again and in the middle
you will have a mean point
your average point on the right hand
side you have
see this sigma symbol is for standard
deviation so you have plus one standard
deviation
plus two standard deviation plus three
minus one standard deviation and so on
by conducting so many tests
on normal normally distributed data set
mathematician and statistician found
that
68.3 percent data points in any normal
distribution
comes in plus and minus 1
standard distribution standard deviation
similarly 95.5 percent data points
out of all total data points fall under
plus 2 minus 2 standard deviation
range similarly 99.7 percent fall under
plus minus 3 standard deviation range
now you can
use this knowledge to remove outliers
general guideline is if the data point
is
beyond three standard deviations so any
data point that is greater than
plus three standard deviation or
minus three standard deviation can be
treated as an outlier this is general
guideline okay there is no
fixed rule sometimes when data points
are small
i have seen people using two standard
deviation as well
so it's a it's a matter of you know
using sense of judgment as a data
scientist to figure out the correct
formula but what we'll do is we'll
now do some python coding and we'll use
standard deviation
to remove the outliers from our data set
i went to this particular kegel data set
i have downloaded this csv file
the file has height and weight i have
removed weight from that file
so now in this file i have people's
height you know i have 10 000 such data
points where
i have different people's height and i
have
loaded that into my jupyter notebook
i have my data frame ready and if i do
df dot height so my height is a column
and if i do describe
it tells me i have total 10 000 rows you
know
my standard deviation for this height
column is 3.84
i have my min max and so on
now we already saw that we can use
plus and minus 3 standard deviation to
figure out these
outliers now before we do that i would
like to use
c bond library to plot the bell curve
and the histogram and the way you do
that is you call
hist plot function in that you supply
your height column and you will say
kde is equal to true which means it will
plot
this kind of curve as well if
kd is false it just plots the histogram
you know
so now you can clearly see this is a
bell curve
it's a normal distribution many times as
a data scientist when you're doing data
exploration
you want to plot this kind of
histogram
to figure out whether the distribution
is
normal or not and based on that you can
make certain decision
eighty percent of time spent by data
scientist
is in data cleaning process because when
the data
comes you know it is often messy it has
errors and it has legitimate
outliers so you want to remove those
outliers
before building your model and that's
exactly what we're going to do here
so to remove the outlier first you need
to figure out a mean so let's say mean
for my height is
66 and these heights are in inches so
66.36 inch
is my mean and my standard deviation
is 3.84 now we already saw
in the diagram here where was my diagram
that i can use plus and minus 3 standard
deviation to remove the outliers
so let's see if i do
mean minus 3 standard deviation
i will get this number and if i do mean
plus 3 standard deviation i get this
number so what i'm saying is
any number between 54.82
277.91 is a valid number
anything outside that is an outlier okay
so in panda's data frame now i can do
something like okay
if my df height is
less than this number
then that's an outlier so i get
two data points like that you know
i will also say that okay it is either
this condition
or or 77.91 right so 77.91
so i get five such data points see with
who has height of 78.09
and so on i can combine this
into one condition i will say this or
that you know like df
dot height if it is less than this
or greater than this then that's my
outlier
so i found total 7 outlier out of 10
000 data points and to remove this
outlier
i can create a new data frame called df
no
outlier and i can just you know apply a
reverse condition
here to just apply a reverse condition
of this and what is the reverse
condition
well the reverse condition is this
so what i'm saying is my regular my
clean data set is something for which
the height is greater than 54.82
and less than 77.91
and when i do the shape obviously see i
find
9993 because seven data points are
outlier
so hurray as a data scientist you just
did data cleaning you remove the
outliers
and now using this particular data frame
when you build your machine learning
model it's gonna be
much better let's now talk about z-score
well i will tell you you already know
z-score if you know standard deviation
you
already know z-score it is the same
concept with a little
tweak okay what is that tweak
here again i have my bell curve in the
middle i have mean
on the right hand side i have plus one
standard deviation plus
2 standard deviation sigma is 4 is the
symbol used for standard deviation
if i have a data point here at 2
standard deviation
then the z-score for that data point is
2.
if i have a data point which is in the
middle of 2 and 3 standard deviation
let's set 2.5
then the z-score for that data point is
2.5
similarly 5 data point here between
minus 1 minus 2
then the z-score of that data point is
minus 1.5
so z score is nothing but how many
standard deviation
away your data point is
from mean so you understand right like
the z-score is for every single data
point so you can compute
z-score for every single data point and
that's what we'll do for this particular
data set see here
for this height column i took an average
which is 5.25
again i took standard deviation for this
i found it to be 0.61
now from every single data point
i can subtract average and then divide
it by standard deviation
i get my z-score so the formula for
g-score is every single data point
minus average divided by
the standard deviation okay so it's
pretty
simple concept if you know standard
deviation you already know
z-score it's the same concept there is
no rocket science here
now let's apply z-score in our notebook
and do the outlier removal
here again i have my data frame with all
the data points and
i want to calculate z-score for every
single data point
and for that obviously i need to create
a new column
when you do this it will create a new
column called
z-score and what is that column well
that column is
df.height so you take your individual
data point
you subtract mean from it so you say
df.height dot mean
you divide that by the standard
deviation so this is how you get
standard deviation
and then when you
look at your data frame i have already
added
new column called z-score and these are
my individual z-score
now i want to verify how i came up with
1.9
for whatever number well it's pretty
simple
see my mean is this okay
my standard deviation is this for that
first data point
and now what i'm going to do is i will
use the formula okay what is my formula
let's look at my formula okay
x which is a data point minus mean
divided by standard deviation so my data
point is this
minus mean okay mean is this
and what is my standard deviation
standard deviation is
this and when i do that i get 1.94 see
1.94
so it's fairly straightforward concept
now once you have
z-score column it becomes even more
easier to remove the
outlier so i will first look at
all those data points whose z-score is
greater than three
see i found five such data points whose
score is greater than three
and less than three is two data points
right so if you want to see
let's say both of these in one shot
then i can say okay less than
minus three or
greater than three and i get same
my seven data points and
you can use the same technique you know
i can say
cfdf no outlier
is equal to the reverse condition you
know the inverse condition
so i will do this i will replace
r with n and this will be like okay my z
score has to be greater than
minus 3 and less than 3 and that's my
no outlier and you find again 999
9993 so i removed my seven
outliers i got my cleaned up data frame
on which i can perform my further
data analysis and even i can build
machine learning model on top of it
now comes the most important part of
this tutorial which is an
exercise i have given a link of this
exercise page in the video description
below
you can read the description and work on
this exercise
friends working on these exercises is
very very important
so i want you to develop the solution
yourself
and then you can click on this solution
link to verify your answer with my
answer i hope
this video gave you some understanding
of
z-score and normal distribution outlier
removal was just one use case there are
many other use cases as well
as we progress forward in this
particular
tutorial series and by the way this the
link of this playlist
is given in the video description below
so you please check other videos as well
we are learning mathematics and
statistics for data science and machine
learning in a very very simple language
and
by doing a lot of practice as well so
make sure you check
other videos as well and if you have a
friend who thinks that
math and statistics is hard well try to
share
these this playlist with them so that
they can remove that bias these things
are not that hard actually
you just need to have some discipline
learn the concepts in a clear simple way
and then practice on python
okay so i hope this was useful thank you
very much
for watching this