using a very simple example we will
discuss
what is mean median mode and percentile
we will also see how these concepts are
used in the field of data science and
machine learning
and then we'll write python code to
practice some of these concepts and
most important in the end we have an
exercise for you to practice upon
so let's begin
let's say you want to open a luxurious
car showroom in Monroe township, New
Jersey
before opening the store you will go
through some data analysis process for
example you will examine the
income level of people in that area
because if the income levels are high
then people will buy
BMWs if the income level is if people
are living on a
government aid they're not probably
going to buy BMW so it doesn't make
sense to open a showroom
so you will get the statistics or the
data
on the people's income and one measure
you can use is the average income
to figure out if you want to open the
store. Here in this case
my average income for all these people
is 6250
dollar per month which is not that high
so I'll probably
not open that luxurious car showroom
but I got lucky here what if this data
set has some
outlier. Outlier means some unusual
values.
Let's say Mr Elon Musk is living in my
town
he obviously has a very high income and
if I use average now the average would
be 1.43 million
and if i use this average number to make
that decision
then it will be a bad decision because
obviously
people are not earning 1.43 million on
average
we got this number just because of some
outliers
so in this case using average is not a
good idea
so then what we should we should do here
well let's do this
let's short the values in ascending
order
and here are my values what if instead
of using average I use the middle value
so see in the data set my middle value
is this 7000
so instead of average I'm using middle
value to
make that decision and this middle value
is nothing but a median
what if my data set has even number of
data points see here I have seven total
7
so this was an easy thing it was a
middle number but i have 7 data points
what is my middle number it is this or
that well
the problem is easy to solve you take
you draw a line here
you you take both the numbers and take
an
average so in this case the median is
7500
so overall if your number of data points
are even then you take two middle values
take the average that's your median if
your data points are
R then you take the middle value and
that's your median
so this is the first use case of using
median
in a simple descriptive statistics
the other use case is handling
missing values let's say you are
building a machine learning model
which can predict predict if person's
loan should be approved or not.
Here my features are credit score and
monthly income
and some of my data points have missing
values for example I don't know what is
sophia's
monthly income. In this case in a
data science we try to estimate the
the monthly income. So here again you can
use
average maybe to estimate her monthly
income
what will be the average by the way 1.6
million
wow! So I took all these values but
because I have
mr musk living in my town he's my
neighbor
and because he has 10 million dollar
income
it is skewing up our average number so
to approximate the Sofia
might be earning 1.6 million dollar a
month is not a good idea
here again we can use medium so you
take
middle two values average it out 6500
and that's probably a better
approximation
okay so although 10 million is present
here median allowed me to
come up with a right guess so overall in
the field of data science
so far we looked at two cases one was a
descriptive analysis for opening car
showroom
the second one was a loan approval model
for data cleaning process
there are many other use cases but I
think these two use cases
will give you a solid understanding of
how median
is used in a real life. Now let's look at
the same income example once again and
here
obviously 10 million is an outlier and
the definition of outlier is basically a
data point that is very different from
rest of the data points
so now let's look at how can you
remove this outlier in order to
understand that
we need to understand the concept of a
percentile
so here again I have sorted the values
in ascending order you take the
middle value okay and that middle value
uh what it shows is fifty percent of the
data points are on the left side
fifty percent of the data point are on
the right hand side okay
so this is your median basically and
this
median is nothing but a 50th percentile
50th percentile
means okay what does it mean
15 percentile here is 7000 and it means
that 50 of my data points
have value less than seven thousand
I have total seven data points and the
three
data points are around fifty percent
right
so around though so fifty percent of
data points have value
less than seven thousand hence seven
thousand
is my fifty percentile okay now think
about it what is my hundred percentile
you need to think this way pause the
video and think
which is the value for which
a hundred percent of data points are
less than or equal to that value
well obviously that value is ten million
hundred percent of
data points are less than or equal to
that value which is 10 million so in
this case 100
percentile is 10 million what is 0th
percentile
obviously 4 000. okay what is 25th
percentile
again I want you to pause this video
right now
think about it I'm pretty sure you will
be able to come up with the right answer
all right so total values are 7
25 percent of seven values are 1.75
which is approximately two
which means I want you to draw a line
after two data points okay
one and two let's draw a line
here is the line. Okay so this line
this particular point is my 25th
percentile this point is 5500 that is my
25th percentile
now just to clarify there are different
ways of calculating percentile I just
showed you one
but another approach is to take this
5000 data point third approach is 6000
you know so percentile could be 5500
5000 or 6000 basically
either 5000 which is a lower value 6000
which is a higher value or a value in
between
and I will show you a page which will
exactly tell you the formula for this
okay what is 75th percentile again pause
the video and think you should be able
to come up with the answer
total value seven seventy five percent
of seven is five
so draw a line after five data points
okay one two
three four five here is my line okay
what is this data point
7750 so that's my 75th percentile
if you have 25th percentile and 75th
percentile
then the range, you know the range is
called
interquartile range. You will hear this
term a lot if you are studying data
science
and it's a very simple concept. It's a
range between 25th and 75th percentile
if the range is between min and max then
it's called
a range in general you know so
general range and IQR are kind of same
same but IQR is between 25th
and 75th percentile if you want to know
more about IQR I have a video
where I showed you how you can use IQR
to remove outliers the descript
link is in the description below please
check it out
all right back to our example of outlier
removal so how do you remove this
outlier well
one approach is what if I say I want to
remove
all the values which are more than 99
percentile
so here 99 percentile will be 10 million
and we'll write the code for this same
thing so you'll understand why
and easily we can remove that 10 million
outlier
using percentile okay
so just to summarize we looked at two
use cases of
percentile here one is income data set
outlier removal the other use case is
very common by the way if you have given
set or gre exam
many exams nowadays have percentile
score or
relative score rather than percentage or
absolute score
which means if you get let's say
75 percent
in the examination but
no one else got more marks than you
let's say you got the highest mark then
all the absolute percent is 75 percent
your percentile will be 100 because it's
a maximum so percentile is relative
scoring and it's
used a lot in the exam scores
okay now if you're curious about how
exactly percentile is calculated
here is the page this is a very good
website statisticshowto.com
I will put a link of this below you can
read the math
but even if you don't want to go detail
into math it's fine
as long as as you understand the concept
now what the heck is mode.
well mode is very simple you guys are
using that in everyday life
let's say you are working in a team and
team wants to go for a team lunch
your manager will tell you okay I want
to conduct a restaurant survey
so the people in the team will fill out
the survey you know Rob and Rafiq will
say i want to eat mexican, Nina italian
and so on
what will your manager do well simple he
will take the maximum right so the
maximum is called mode
so maximum votes are mexican and that
maximum votes is nothing but a mode.
So mode is the most frequently occurring
value in a data set
we use more for so many things in real
life we just don't know this fancy term
called mode
so I thought I will just give you a
quick idea let's write python code now
I have the income data in csv file and I
had
loaded that into a pandas data frame
here you can see
there are seven rows in total and what
we're going to do is use
percentile to remove the outlier that we
have here
and first function I will call is depth
of describe
this will give you a basic statistics of
a data frame
such as count, mean, standard deviation
and so on.
But look at these three important
numbers this is showing you the 25th
percentile which was 5500
so if you
remember from our presentation our 25th
percentile was 5500
our 75th percentile was 7750
see 7750 so
the describe function
gives all those awesome statistics you
can also call
df dot income which will give you of
course this column which is a pandas
series
and quantile function so quantile
okay quantile if you do 0.75
it will give you the same number 7750
you give 25, gives you this number
you can give any number by the way you
can give 45 for example
45th quantile and it will
just work as okay if you do
google search on this API remember in
the presentation I told you that
there are different ways of calculating
quantile for example let's look at 25th
quantile
okay 550 but if you look at
the
documentation if you do shift tab
you see this parameter interpolation is
linear
so there are couple of values that it
can take.
It can take values such as lower
higher nearest midpoint so let's try
lower and
higher so here if I do
interpolation is equal to lower
so when you do lower let me show you
what it will do
so when you do lower it will actually
give you this 5000 points see so if I do
lower
it gives you 5000 if I do higher it will
give you 6 000. so 5000 and 6000 or
values in between
but if you do interpolation linear it
will just take
use those two numbers okay
now if you do zeroth quantile
4000 if you do one quantile the maximum
number Mr Elon Musk
it will give you that all right now
we want to use quantile to remove the
outlier
so how about we use let's say
for example 0.99
I want to discard anything that is more
than 99th
quantile so 99 quantile is 9.4 million
okay so let's store this in a variable
so
percentile 99
and then I will
say okay in my data frame
give me all data points who are
greater than this percentile and it
gives me Elon Musk this is an outlier
and if you want to remove it and and
prepare a new data frame
which you call it df no outlier
okay then see this df no outlier so you
have to reverse this conditions you
you to say my valid values are anything
which is less than
99 percentile and I just remove the
outlier
my data frame is ready this is what data
scientists do they do lot of data
cleaning data preparation
an outlier removal is a very important
step in that process
now let's look at the second use case
which is filling a missing values
so again I have this data frame
and in that let me make sophia's income
null so that I can show you
how an a can be filled using a median
so I will do something like okay 3
that will give me this and at 3
row number 3 my income
I want to make it np dot nn
actually I need to do I location I
location will give you that location
all right so this worked
and if I do df now
all right something is not right
all right looks like some problem I'll
do it reverse way I will
take the column df income
and then at number three I will store
nan and now you see Sofia's
income is an n let's assume in real life
when you are dealing with your data
you know many times the data some of the
data points will not have a value
because of data transmission error
because the data is not available
whatever reason
but this is what happens in data
scientist life that some of the values
are missing and you need to substitute
that with the
right guess so if you make your guess
such that let's say i'm saying
my mean can be used as a substitute
so my mean for the income is this much
and it is very high because of this
outlier but if I use that to fill any
there is a function called
fill n a
and if I use that let me store this in a
new variable
it's gonna be disaster because on
average
all these guys have five thousand six
thousand four thousand values but the
so sony as Sophia's income is now 1.6
million
and it became this high because of
outlier so here is a classic case where
you should not use mean you should use
median instead and we can copy paste the
same
function here but instead of mean
you do median and when you do it now
the values look more reasonable you know
6500 dollar
as a substitute for value that is not
available
sounds far more reasonable than filling
it with 1.6 million so you can see that
whenever you have outliers the values
are not in a kind of same case range
you should use median instead of mean to
fill the any values
that's all I had for this tutorial the
most important part of this tutorial is
an exercise I have this exercise for you
the link for this will be found
in a video description below what you
have to do is
you need to use this Airbnb New York
data set
which is available on kegel so download
this csv
file by clicking on this button and
use percentile to remove the outliers
that data set
has some outliers you want to remove
them using appropriate percentile both
on the lower and right end
and there is a solution link but I trust
you you guys are
all you guys and girls are all very
sincere students and you will
click on solution only once you have
tried on your own
all right thank you very much for
watching this
and I will see you in a next video