hey everyone and welcome to my mini
course on the essentials of data science
this mini course provides a super basic
looking to data science what it is and
the three main components that make up
data science data science is a very
mainstream word like it's thrown around
a lot but its actual definition is quite
vague this mini course is designed to
help those of you who are curious about
data science develop a better and more
specific understanding of the topic
there are definitely more advanced
techniques within data science such as
machine learning but even these can be
traced back to the three essential
components that we'll cover before we
get straight into it
I thought I'd quickly introduce myself
my name is Max and I work as a data
scientist after getting my degree in
physics I find myself more and more
drawn into the world of data science so
instead of diving into the realm of
physics research I taught myself all the
tools and techniques a data scientist
needs and shortly after landed my dream
data science job I've since also started
teaching data science to others and have
been fortunate enough to teach what is
currently over 9,000 students the skills
of gathered and learned over the past
five years of my data science journey so
let's jump right into it so what is data
science well data science is you can
kind of summarize it in different ways
but the main parts of it are
transforming data into information and
this is a really big step because a lot
of people talk about you know data and
big data and all these things but data
by itself isn't really that useful until
you can turn it into information and so
if you just have a bunch of numbers
appearing somewhere and it's just you
know so much of it no one can make sense
of that and that's where you need a data
scientist to be able to transform all of
these all of this vagueness and kind of
this noise to that's going on and you
need to be able to extract information
from it
and that's what a data scientist does
now what you do with this - with this
information or how you get this
information
it's through analyzing your data so a
big part of it would be you know
cleaning things up doing some some
processes on it and then you analyze
once you've clean things up
and that is one of the ways that you can
then get information out of your data
through this analysis and you can kind
of continue on and you see trends and
patterns and all types of correlations
hopefully and all of these things again
build up into this turning data into
information component and then
ultimately you also need to
contextualize everything that you have
so your computer can't do that for you
you can Peter can kind of crunch the
numbers and stuff but it's your
responsibility also to make sense what's
in front of you and even if you see
something you just don't blindly trust
it but you need to understand you know
where am I at where am i coming from
where is this data coming from you need
to be able to contextualize these things
and then of course be able to apply as
well as understand them and so once you
have this data you know it's great but
turning it into an information into
great information that you can use and
directly apply that's where the real
power lies and that's also kind of the
role of a data scientist so that's what
the data that's what data science pretty
much is and so what is the data
scientists do well we kind of already
talked about this just a little bit but
let's go over it again any more concrete
examples um so a data scientist would
for example get and process this raw
data and then convert it into something
a little bit cleaner so you can imagine
kind of just like a data stream coming
in and it's you have this measuring
device and constantly is just measuring
all sorts of data and because like
nothing is really constant so everything
will be fluctuating up and down and so a
data scientist would be to kind of take
all of this data it'd be to kind of
clean it up a little bit you know maybe
reduce this fluctuation that you know
isn't supposed to be there that's just
kind of background stuff going on and
then put it into a format so that you
can easily plot it against some things
and then we already get to the next
point that you know once this data is
cleaner you can maybe do start doing
some calculations on them figuring out
the core statistical components you know
like what is the average values of these
what what am I really dealing with you
know getting a first look at first
understanding of what it actually is
that you're tackling and then once you
have this kind of understanding then you
can start to do some visualize they
which helped you as a data scientist
maybe see some trends or patterns
already but visualization was also
really key because they let you show it
to other people and there are great
means of communication so they help both
you as a data scientist as well as
helping others when you try to convey
this information to them all right and
then finally you have to suggest some
applications of the information so it's
not really enough to just be able to
look at it and say like yeah I see it
goes up and down and that's that's good
but what does that mean how does this
transfer into something useful and
that's also one of the key roles of a
data scientist transferring information
into knowledge and so you've got this
data into information step but you also
need to transfer this information into
knowledge and those are two really
powerful things that are worth a lot a
lot and that's pretty much what a data
scientist focuses on and then you can go
further you know and take this data and
do machine learning with it or something
if you really understand what's going on
or if you have some hypotheses of you
know what could happen so you can take
things a lot further but ultimately this
kind of turning data into information
and then into knowledge that's kind of
your role all right so let's go into the
essential techniques or the essential
components of data science so the first
essential component and we kind of
touched on this already is statistics
and basically we're gonna cover this
later on but let's just give a kind of
quick wrap down so in statistics need to
understand different data types that you
can encounter and so there are data can
come in different ways and we'll go
again into more detail with this later
but it's not just you know you get a
bunch of numbers data can come in very
many different ways depending on the
field that you're in and so you need to
be prepared and you need to kind of be
aware that data may not always just be a
direct number for you then of course you
need to understand some key statistical
terms like you know the different types
of means and also understanding
fluctuations in data and the reason that
this is important is because these key
statistical terms give you an overview
of how this data is behaving and
depending on how the data is behaving
you may want to approach it differently
so if you know that your data is
very clean there's a very little
fluctuation then if you visualize things
you can probably trust what's going on
or if you want to maybe fit some curves
to it or something but if you see
there's a lot of fluctuation in your
data visualizing it is going to be much
more difficult because you just see
jumps everywhere and you're not really
sure which of this is actually true and
which of this is caused by you know like
some interference somewhere or someone
is messed with my system and so all of
these things will kind of be hinted to
you through statistical terms so it's
probably good that you know you're kind
of comfortable with these things and
that you can be able to get some meaning
and meaning out of them all right and
then finally it be in statistics to be
able to you know split up and group or
segment data points so that when you
have this big data set you want to be
able to you know maybe split it up into
smaller things compare different regions
look more into more detail into some
things and maybe you know isolate two
components because you know hey these
things are probably going to be
important the rest I don't really care
about that much so being able to kind of
pinpoint an isolate and meddle with the
data a little bit so these are the kind
of statistical components that we're
gonna look into all right so the next
big thing and we've already talked about
this too is data visualization and we'll
see why data visualization is a really
key skill for data scientists and then
we're also be gonna be covering
different types of grass that you can
use and how you can compare different
number of variables so for example you
can have one variable grass where you
only look at one thing and you only want
to look at this and you want to see how
these how this changes you have your
typical two variable grass which you
probably know where you have this X and
a y-axis and then you can kind of see
how two variables relate to each other
or you can have three variable or even
higher variable graphs and where you
plot maybe three different things or
even more if you want as long as it
makes sense next to each other so that
you can compare multiple things at the
same time all right and now we come to
the other big thing that you're probably
going to need as a data scientist which
is going to be the ability to program
now not every data scientist can do this
but this is
really really essential in my opinion to
your role as a data scientist because
knowing how to program is going to make
your life so much easier if you know how
to program you can kind of take your
ideas and your thoughts and you can put
them into actions in the computer and
you can just automate everything you can
customize things you can explore you can
prototype you can test and you're not
reliant on some you know application you
don't have to master some application
and if it doesn't work or if one feature
isn't there you have to contact customer
support and maybe it's not even possible
and then you have to wait for an update
or maybe something is bugged
with programming there's just you're so
much more reliant on yourself and you
can really just do whatever it is you
want to do and you're not reliant on
other people or on the tools that other
people have built for you but rather you
can just pretty much go and you know
just do what you want to do without
there being major roadblocks and then
we'll also look at some essential
packages in Python so in programming you
never want to reinvent the wheel you
always want to start off where the last
person left off and so the ability to
program and be able to write simple
programs you would need to teach
yourself but you wouldn't need to write
highly complex mathematical packages or
data analysis packages those are already
out there all you need to do is be able
to download them and implement them in
your code and they're gonna work you
know they've been tested a lot there's a
huge community's working on them on
improving them and everything all of
this is for the community and so the
whole community kind of works together
to improve it no one's really directly
trying to make a lot of money off of it
so they're not going to charge you all
of these service fees and everything
everyone's just trying to improve their
package because if it improves everyone
also benefits from it and so we'll look
at some of the libraries or we'll talk
about some libraries that you can use
especially in Python and to help you
along your way with data analysis and to
become a successful data scientist in
this chapter we're going to talk about
statistical data types now we're going
to look at the three different types of
data which are summarized as numerical
categorical and ordinal types of data
now these are the types of data that we
talked about before how you can't just
expect your data to be cut be kind of
numerical and so we'll see the miracle
data but we'll also see the two other
types of data that you may be you know
encountering in your career as a data
scientist all right
so let's talk about numerical data first
though numerical data is also known as
quantitative data and it's pretty much
things that you can kind of measure it's
it's great numerical stuff that you can
do math with you can compare it you know
saying this Plus this makes sense he is
greater than B these are you know all
examples of numerical data numerical but
data can we split up into two different
segments one of them is going to be
discrete and so discrete means the
values only take on distinct numbers and
an example of this would be you know IQ
or something like that a measurement of
IQ or if you do a coin toss the number
of times that you toss heads so you can
you know you can have 15 heads you can
have 12 heads out of you know 20 coin
tosses you can have 500 heads out of a
thousand coin tosses or 500 out of 600
or all of these things but all of these
are distinct numbers and now they don't
have to be whole specifically but they
do have to be distinct so that's that's
the kind of very important part that you
know there's a kind of step size that
you're dealing with and of course you
can still say hey you know flipping
eight heads out of twenty is better than
filling seven heads out of twenty so if
you want to flip heads lettuce or
flipping eight out of 20 is worse than
flipping 7 out of 20 if you're going for
as many tails as you can so all of these
kind of comparisons that make sense so
that's the discrete part of numerical
data then we have the continuous part
and now the continuous part is really
that values can just take on any number
and they're not unlimited by decimal
place so a value that can you know can
be like one point one and then the next
value would be one point two that's not
continuous that's still discrete because
you have this step size of zero point
one continuous means literally ever
number from start to finish can be taken
on and this doesn't mean that every
possible number in the universe from
negative infinity to plus infinity and
all imaginary numbers and everything
that comes with it that doesn't that
that's not required for continuous it
could really be that just every number
between zero and one can be taken on so
for example let's say you have a bottle
of water and this bottle of water can
hold one liter now if you fill your
bottle up and it starts off empty and
you fill it all the way up to the top
the amount of water that you've had
needed to take on every single number
between zero and one because you can't
just fill up water you know in kind of
small increments of say hey I'm gonna
put in 0.2 liters every single time
because the water doesn't just you know
teleport from A to B but when you're
pouring in water it's more like we see
in the stream here and the water level
rises and Rises and Rises and so the
amount of water that we have in our cup
needs to take on every value between
zero and one so that's an example of
continuous data for but you see that you
know we can be limited to zero and to be
between zero and one we don't have to
you know start at zero and go all the
way up to infinity or something but it's
just that the range that we're looking
at every single number can can be
applied or every single number can you
know happen another good example would
be the speed of a car if you start and
you you know you're standing still and
you're studying you're standing at a
stoplight and then you want to
accelerate in the speed limit us say you
know 50 miles an hour or something to
get to 50 miles an hour from your
starting position your car has to take
on every single speed in between and of
course you won't see that you know on
your spot on the speedometer it would
say something like zero miles an hour
one mile an hour you know maybe you can
go to like it's going 0.1 0.2 0.3 or
something like that so it may look
discrete to you but that's not how your
car is going your car doesn't say it
like oh I'm gonna go in these step sizes
of speed it's gonna accelerate
gonna take on every value starting from
zero going up to 50 miles an hour and
you're gonna when you're in this
transition you're gonna take on every
single one of those speed values so
that's how continuous data looks like
and it's important to understand the
difference between this discrete and
continuous just because you may want to
approach it differently now of course if
we're dealing with computers our
computers can't deal with infinite
numbers in the decimal places we have to
cut it off somewhere and so usually
continuous data is gonna be rounded off
at some point but it's still important
for you to know that you're dealing with
continuous data here rather than
discrete so that you know hey there can
still be other stuff in between here and
or all of these things rather than you
know having specific step sizes and all
you see is just kind of a bunch of lines
at every step size but you can expect
that when you have continuous data that
everything is just kind of filled filled
up that everything can and may even well
be in between certain places so that's
that's kind of the important thing to
note between discrete and continuous
alright so the next type of data that
we'll have is categorical now
categorical data doesn't really have a
mathematical meaning and you may also
know it to be qualitative data um and
categorical data it describes
characteristics so a good example of
this would be for example gender so here
there is no real mathematical meaning to
gender of course you know if you have
good data you can say male is zero and
female is one but you can't really
compare the two numbers even though you
assign numbers to them and you may just
do this so that you can split it up
later on there your computer can
understand but it doesn't really make
any sense to compare you can't say you
know is male equal what you can say male
is not equal to female but you can't
really say is one greater than the other
or is one approximately equal to the
other those things don't really make
sense because they're not well-defined
what does that mean and you can't really
add them up either you can't say male
plus female but that doesn't it doesn't
give you a third
category or something so categories that
you can't really apply math to them but
there are nice ways to kind of split up
or group your data and they provide
these nice qualitative pieces of
information that are still important
it's just you can't really go that well
about you know like plotting them on a
line or something like that
so those are important things to note
with categorical data and then another
example would for example be yeah
ethnicity or you could also have
nationality all of these things are
examples of categorical types of data
yeah I'm so like we said you can assign
numbers to them but that's really just
for your code so that it's easy to kind
of split them up but you still can't
really compare them how are you gonna
compare nationalities there is really no
definition for you know comparing one
type of category to another alright and
so the third type of data that you can
encounter is something called ordinal
data
now ordinal data is a mixture of
numerical and categorical data and a
good example of this would be both tell
ratings so you have you know star
ratings 0 0 1 2 3 4 or 5 stars or maybe
even 6 stars or you know whatever it is
whatever the hotels go up to these days
um but it's still not as straightforward
to compare so I'm sure you've seen two
different types of three-star hotels one
of them you know had the bare minimums
the beds were okay but it wasn't really
anything special and then you had this
three-star hotels that you could have
sworn we're at least four-star and so
star ratings do make sense we can say
you know a four-star hotel is probably
better than the three tier hotel because
there have been standards there are
standards for these things they have
been checked you know if you go to a
four-star hotel you know what to kind of
expect but still it's not completely
defined so like you know coming back to
this three star example it's very hard
if you just say hey we're going to a
three-star Hotel it's very hard to know
exactly what to expect because there are
different parts of three-star hotels
there are three-star hotels that have
developed onto like have a swimming pool
maybe or something like that and then
there are those three-star hotels
that are really more like hostels or
something that I've just made it past
the to start place and so there it's
much harder to kind of define or to know
what to expect now if you take averages
of these star systems though then you do
get a much better idea of what's going
on so if you have you know consumer
reviews or something like that and you
say Oh from you know 500 reviews our
hotel has an average rating of like 3.8
then you know that the three star hotel
that you're looking at is pretty much a
four star hotel it feels like a four
star hotel even though it may not have
all of those qualifying characteristics
that's the kind of feel you get from it
whereas from another three star hotel
you may have a rating of like 2.9 or
something and there you know oh you know
this hotel is more towards the lower end
of the three star some people may not
even consider it to be three stars and
of course you know this rating may be a
little bit biased because they went to a
different three hard star hotel first
and then they went to this one and they
were expecting something completely else
from the three star hotel so they said
this can't be three stars this is two
stars but it's because of the way that
the ranking system is defined underneath
and everything and so when we have these
averages of these ordinal numbers then
they kind of start to make a little bit
more sense alright so let's go over a
small exercise and see if we can
identify what type of data we're dealing
with so the first thing we'll look at is
gonna be the survey response to
happiness now you have people filling
out a survey and then this and then one
of the questions is you know how would
you rate your happiness and it's gonna
be bad neutral good or excellent what
type of data with this be well this
would be an ordinal type of data because
it's still in a form of categories and
you're asking for the subjective opinion
but it does make sense so you can still
compare them you can say excellent is
greater than good good is greater than
neutral neutral is greater than bad but
what exactly does it mean to be good and
excellent you know where do different
people draw the line for this there's
it's still a little bit of vagueness
involved but generally it does make
sense and you can compare it and if you
have a lot of surveys
and you averaged them the values you're
gonna get are probably going to be very
well representative or at least pretty
good representative all right so if we
look at the next thing which is going to
be the height of a child what type of
data is that now we can say it's
probably numerical and well it actually
most definitely is numerical so the
height of a child is a numerical value
but let's go a little bit deeper and say
is the height of a child discrete or is
the height of a child continuous well
even though when you measure height you
get something like you know five foot
five foot three or 160 centimeters or
something like that it's not a discrete
value because to get that height you
have to have reached every single height
before and so even though at the moment
you may be measuring it you're kind of
rounding it off to how much your
measuring tape can measure so like your
measuring tape is kind of limiting the
height but if you had a super super
precise measuring instrument you could
measure not just you know five foot
three or something like that you could
really go into detail with the inches
and the decimal places and there and
everything kind of going on so the
height of a child would be a numerical
data type but it would be continuous all
right now let's take about talk about
the weight of an adult do you expect the
weight of an adult to be either discrete
or continuous so we can probably agree
that it's numerical because it's a
weight value it's it's pretty much
defined to be a number and what do you
expect it to be discrete or continuous
well the right answer here is gonna be
continuous again because to reach a
certain weight they would have had to
have reached every single weight in
between before so again the weight is
something that we can consider to be
continuous all right and so finally
let's look at the number of coins in
your wallet again we can already by the
name it says a number of coins so we can
probably agree that this is a numerical
type of data but the number of coins in
your wallet would that be discrete or
continuous well the answer would be
discrete
because it doesn't really matter what's
your anoint your coins are they could be
50 cent pieces that could be 25 cent
pieces ten or five or ones or anything
you know like a two or something like
that but they're not going to be but the
number of coins that you're gonna have
we're gonna sum up to a whole number so
you can have one coin you can have two
you can have three all of these things
but you can't have infinite fractions of
a coin you can't have say you know the
square root of two number of coins that
doesn't really make sense so you have a
defined step size you have one coin and
then if you have a second coin then you
have to you get a third coin meaning of
three you're going in step sizes of one
so for the number of coins in your
wallet we'd be having discrete numerical
data in this tutorial we're going to
talk about the different types of
averages now we're going to see the
three different types of averages which
is the mean the median and the mode
alright let's get started so we'll start
off with the mean now the mean is the
typical average that you know and really
what the mean is is you just sum all of
your values up and then you divide them
by the total number of values that you
have now the great pros of the mean is
that it's very easy to understand it
makes sense we just have everything we
have and just kind of add it all up and
then divided by what we have and that
should give us a good representation of
what is the average and it also takes
into account all of the data so since
we're adding everything up and then but
dividing by how much data we have we're
taking into consideration every single
data point now there are some problems
with this so one of the problems is that
the mean may not always be the best
description and we'll see why when we
look at examples for when we should use
the median and the mode and the mean is
also very heavily affected by outliers
so since we're taking everything into
consideration if we have big outliers
that's really gonna change how our mean
looks like so if we just have normal
values you know between like one and
five and all of a sudden we have like
10,000 in there that's really gonna
affect our mean so mean is heavily
influenced by outliers and the bigger
the outlier
more the mean is influenced by it all
right so let's see some examples of the
mean we'll go through a worked example
first and we can see our data set here
which is just a bunch of numbers and
what we're gonna do to calculate the
mean is we're just gonna take every
single one of these numbers and we're
gonna add them up and we can see the
total result that we get here and then
the next thing we're gonna do is we're
gonna take this total result we're gonna
count the amount of data points that we
have and we're gonna divide one by the
other which then gives us our mean as we
can see here so that's an example
calculation of the mean but let's see
some example applications of the means
so when would we use it well good
application would say if you look at the
time it takes you to walk to the
supermarket so sometimes you walk a
little bit faster and maybe it takes you
20 minutes to get there sometimes you
walk a little bit slower it takes you 25
but on average it takes you somewhere
like 22 or maybe 22 and a half minutes
or something like that so if you say I'm
gonna go to the supermarket you're like
it's gonna take me this much time to get
there another good example of the mean
would be exam score for a class so to
get a good understanding of how people
do in an exam or in a class you can look
at the mean exam score last year and
since our exam scores are kind of in a
smaller range a mean is gonna be good to
use it because you can get anything
between 0 and 100 but realistically
speaking no one's probably going to get
a zero so your range is even smaller and
so you're less affected by outliers and
you kind of know how hard a class is
gonna be just by being you know able to
compare their means so if you look at
one class and it's mean is higher than
the other but they have a large number
of students or something then you can
probably say hey it's easier to get a
good grade here or something like that
or maybe you know some of these it's
more simpler overviews without diving
too deep into it alright another good
example of the mean would be to say how
much chocolate do you require when you
get this kind of sweet craving and
you're not gonna say like oh you know I
require one chocolate bar two chocolate
bars
but like you're gonna say Oh on average
you know I require you know maybe
three-quarters of a chocolate bar and
sometimes I may want a little bit more
because I feel like it and when I start
eating chocolate I crave it even more
sometimes you know I have it at first
and like the tasters doesn't sit right
with me right now and so I have a little
bit less but these are kind of the
amount of things so like if you have
this craving you know either you say oh
I'm gonna try to be strong or you like
hmm well I know this feeling and I know
if I eat about you know three-quarters
of a bar of chocolate or something I'm
gonna feel good my craving is gonna be
satisfied so you kind of know what to
expect so these are some of the examples
for how we would deal with a mean well
when we would use mean all right so
let's look at the next thing which is
going to be the median now the median
represents the middle value in your debt
data says now if you have an even number
of data points you don't really have a
middle value and so in that case the
meeting is gonna be the mean of the two
values so it's going to be the two
meeting values added together and then
divided by two so the pros of using a
median value is that the median can
sometimes be more accurate than the mean
and we'll see some examples of this
the median also evenly splits your data
so you're not really you know affected
by the mean in the sense that if you
have an outlier in the mean and it drags
everything to the right it could be that
your outlier drags things so far to the
right that all of your data is to the
left of the mean and only the outliers
to the right so that would be an extreme
case but that can happen whereas the
median you know it's always located
directly in the center of your data and
the median also doesn't care about
outliers so if you have huge outliers at
the beginning and at the end it doesn't
really care because outliers by
definition aren't very common because
they're outliers and so if you have them
at the beginning or house them at the
end they're gonna be very few in number
which makes them outliers and therefore
the median doesn't really care about
outliers that much a con though is that
the median doesn't really give you much
information on the rest of the data
sure you know you know what's at the
center but you don't know how does
everything
we behave you only know where is the
center of our data so let's see some
examples well do a working example first
where we see our data set here and we
can count how many values we have is we
go from left to right then we can say
we've got 1 2 3 4 5 6 7 8 9 10 11 12 and
13 data points so we've got an odd
number and so our median value our
center value is going to be the seventh
data point because it's 6 from the
beginning and it's also 6 from the end
so it's equally spaced both from the
beginning and from the end and so that's
why we see our median value here is 26
it's located directly in the center now
what is the median useful for well the
median is often used if you look at you
know household incomes for a country
because if you were to use the moon then
these billionaires they would just
completely you know they would give you
a false description of what really an
average household income is because
normally if you have you know like an
average value and you can say oh the
average household income from this
family would be say $40,000 or something
like that or that would be the median
value but if you were to use the mean
instead then all of the billionaires and
all the millionaires in the country they
would change that household income and
then you would say oh you know the
average household income per family
would look like 60k and that's a bad
representation because that doesn't
actually give you a realistic look at
what the average household family has
and the average household family really
does it's you know centered at like 40k
and sure there are people below and
there few people be high but that's
what's in the middle whereas if you were
to use the mean instead for your average
you would kind of get this inflated
household income which wouldn't be
representative to the rest of your and
the rest of the country another good
example of the median would be the
distance that people cover to get to
work so if you look at this in terms of
you know kilometers then you can say
like oh you know some people they walk
to work and it's like you know one
kilometer at most so something like that
and then you can expect people to travel
most people travel around three
kilometers to work and sure there are
some you know that travel much further
because they want to live outside of the
city and there are some that travel very
very short distances because they have a
house right next to the office where
their house is the office or something
like that depending on where you're
working but then you can look at you
know like we're in the middle how do
people travel to work what time or what
distance do they need to cover and so
that would be another good use of the
median and a meeting another good
meeting value is what do you usually
spend when you buy a new item of
clothing and so sure you know sometimes
may go to that expensive clothing store
and you could get a jacket that costs I
don't know north of a couple hundred
euros or dollars whatever system you
want to use and sometimes you can go to
a secondhand store and get it for very
cheap but usually if you go into stores
a jacket I don't know maybe cost you
like a hundred dollars or something like
that and so you know if you go out you
can expect to pay about $100 no not
really you know taking that much
accountant - what story going in - so
most of the stores that you're gonna
visit are gonna have that price for the
jacket so that would be another good use
for the median all right let's look at
the third type of average that we can do
which is the mode now the mode looks at
the most common value in your data and
it's not really defined if there are
several most common values but if
there's only one most occurring value
then that's what your mode would be and
so we'll see an example of this in a
second to the pros of using the mode is
that it's not only applicable to
numerical data so if you look at
categories for example then you can say
hey we've got five people from the US
you know and two from Canada and one
from France and you know that the mode
is gonna be the US because there are
five people from the US so mode is the
great average that's not only applicable
to numerical data in this sense but you
can technically also apply it to
categories or to ordinal numbers if you
wanted so that you can say the most
common country that we have were the the
average kind of country that we would
expect
tear is the US and sure there are other
countries but the average or the most
common one is gonna be the u.s. in this
case so yeah and then of course and the
other Pro is that we allow to see what's
most common what pops up the most so
that's a great use of the mode if there
are cases when you know recurring values
happen a lot which is the case for
discrete numbers for example so in
discrete numbers values recur often and
so it's good to use the mode icon of the
mode it's gonna be that it doesn't
really again give you good understanding
the rest of the data similar to what we
had for the median but also it's not
really applicable if you just have a
bunch of different types of data then
there isn't really gonna be a mode if
there's not enough of each data it's not
really good to use the mode you don't
want to you know have thousands of data
points and the most reoccurring value it
reoccurs like three times that's not
good you want to use the mode for
situations where data re occurs often so
like we saw the country example but
let's actually see a worked example but
also some other examples for the mode so
the worked example here would be again
we take our data set and we can count
how many times different numbers appear
and so if we go through the numbers
we'll see that twenty six occurs the
most and so that's gonna be our mode
here so we've got 22 and 25 that both
occurred twice but 26 occurs three times
and so 26 is gonna be our Millah it's
gonna be our most occurring value now
the mode is gonna be useful for things
like the peak of a histogram so if you
draw this histogram and if you don't
know what a histogram is don't worry
we'll cover that in a later lecture to
let me go into data visualization but
the peak of a histogram that's gonna
show you the mode of the data the most
occurring data a good another use of the
mode will be if you look at employee
income at a company because at a company
you know you can again have the boss
which takes off the mean and you can
have you know higher level employees to
which we kind of shift the median but if
one third of your employees earn minimum
wage
that's gonna be the best average or say
40% of your employees earn minimum wage
you're probably not your employees
because that wouldn't be a very good
system to have but a 40% of the
employees at the company that you're
looking at earning a minimum wage that's
not a really good thing to have and if
you look at the mode you'll easily see
that the average in this case would be
to earn minimum wage because that's what
most people earn and sure you know the
boss he or the CEO or something you know
he may shift the mean up heavily and
then the fact that you have higher ups
if you look at the median value you may
even well be too far you know too far to
the right that you really don't consider
these employees that all are in the same
amount but you really want to get that
description which is what you get here
from the mode and then also the out kind
of an of an election is where you use
the mode for and sure sometimes you may
only have two values sometimes you may
have three but if you have different
candidates and say you have five
different candidates then the person
with the most votes is gonna win the
election because they have the most and
so they are again you'll use the mode in
this lecture we're gonna look at a
spread of data and we're going to start
off with looking at the terms range and
domain then we're gonna move on to
understanding what variance and standard
deviation means and then finally we'll
look at covariance as well as
correlation all right so let's start off
with a range and domain now let's start
off with the range though so the range
is basically the difference between the
maximum and the minimum value in our
data set so that's that's kind of simple
to think about so let's just kind of go
through this with a work example let's
set up a company in the town and this is
the only company in the town and the
owner of the company earns a salary of
200k a year and then the employees you
know they all have different salaries
but the lowest employees or maybe the
part-time workers they earn something
like 50k a year so we've got data kind
of ranging from 15k up to 200 K and so
our range is the difference between the
maximum and the minimum value in our DNA
so we take 200 K and we subtract 15 K
from it
and we've got a range of 185 K in salary
so that's how big our salary can change
so it can if we start at 15 K it can go
all the way up to 200 K so that's a
hundred and eighty five K range of
salary that people in this company can
have all right and the domain is going
to be the values that our data points
can take on or the region that our data
points lie in so if we look at this
example again our domain is gonna start
at 15 K and go up to 200 K so what the
domain defines is it defines kind of
starting and ending points or it defines
a section in our data and so in this
case the domain would define you know we
would start at 15k and it would end at
200 K and what the domain tells us is
that everything or all salaries within
you know between 15k and 200 K that they
are possible but within this domain or
within this company it's not possible to
have salaries outside of the glist
domain so if our domain again is 15k to
200 K then we can't have a salary of 14
K because that's outside of our domain
and we also can't have a salary of 205 K
because again that's outside of our
domain so pretty much all salaries
within 15 to 200 K are possible anything
outside of the domain is not possible
because that's no longer in our domain
all right so let's move on and look at
the variance and standard deviation and
we'll talk about the variance first and
what the variance tells us it pretty
much tells us how much our data differs
from the mean value and it looks at each
mean value and it looks at how different
each value is from the mean value and
then it gives us the variance it does
some calculation and we don't really
need to know the formula it's more
important right now just to understand
the concept of variance and so what it
variance really tells us is it tells us
how much our data can fluctuate so if we
have a high variance that means a lot of
our values differ greatly from the mean
value and that will make our very
it's bigger if we have a low variance
that means a lot of our values are very
close to the mean value and so that will
make our variance lower and now if we
turn to the standard deviation the
standard deviation is literally just the
square root of the variance so if you
understand one then you also understand
the other and now we can combine this if
we know the range of our data to kind of
get a better feel for our data and so
let's use an example where we have two
different countries just countries a and
B and they have the same mean height for
women which in this case will say is 165
centimeters or 5 feet 4 and we'll say
that the range of heights for them could
be identical so let's say they can range
you know the range let's say it could be
like 30 centimeters or something you can
go anywhere from say 150 all the way up
to 80 or we can even increase that and
say like anywhere from as low as 140 up
to like two meters or something like
that but let's just keep the range for
these the same and they both have the
mean height now if country a has a
standard deviation of five centimeters
which is approximately two inches and
country B has a standard deviation of 10
centimeters which is approximately four
interests then what you can expect
knowing these values is that if you go
into country a the people that you're
gonna see are gonna be much more similar
in height so our standard deviation is
lower that means our values differ lower
from the mean and so that means a lot of
the women that you're gonna see are
going to be very close to 165
centimeters or 5 feet 4 plus minus 2
inches so it's very what you can expect
when you go to this company that when
you go to this country is that everyone
is gonna be or every a lot of the women
are gonna be about that height whereas
if you go to country B they have a much
larger standard deviation and so you
can't really expect everyone to be about
5'4 because it fluctuates a lot more and
so if you go to that country you can
expect to see a lot more women of
different heights both taller and
shorter than
for all right and so that's how we can
kind of use the variants in the standard
deviation or the standard deviation to
give us a little bit more perspective on
our data and kind of allow us to infer
some stuff about our data all right so
let's talk about covariance and
correlation and so covariance will or
already has the name very incident but
covariance is measured between two
different variables and it pretty much
measures if you have two variables so
let's say we've got you know me drinking
coffee in the morning and my general
tiredness so if I use these two values
and you know get data point so this is
how much coffee I drank in the morning
and this is how tired I feel this
morning or something like that and so
what the covariance does is it looks at
how much one of these values differs or
changes when I change the other one so
what does that mean for example well if
I drink more coffee what the covariance
would look at is how much does my
tiredness change so that's what you do
with covariance you see you say I change
one how much does that affect the other
thing that I look at and our correlation
is very similar to covariance so we kind
of normalize the covariance by dividing
by the standard deviation of each
variable so what that means is we get
the covariance for my drinking coffee
versus feeling tired and then we would
just divide by the standard deviation of
me drinking coffee and a standard
deviation of me feeling tired and so
really what we're doing with the
correlation is we're just kind of
bringing it down to relative terms that
would fit our data better so that's kind
of the abstract idea the important thing
to just keep in mind is that we're
looking at one and we're seeing how much
that changes and we're seeing how much
that change affects the other one all
right so there are different types of
correlation values that we can have and
they can range anywhere between negative
1 and 1 or so their domain is between
negative 1 and 1 and a correlation of 1
means a perfect positive correlation so
that
when one variable goes up the other goes
up so for my coffee example that would
be if I have coffee in the morning then
I also feel more happy so the more
coffee I have the more happy I feel and
of course there's going to be a limit
but let's say I only drink up to two
cups of coffee or something like that
and I can drink anything in between and
the more I have the more happy I am
about it so that would be a positive
correlation the more I have of coffee
the more I have of happiness and so they
would kind of go up together
and then when we get closer to zero the
zero point is gonna mean no correlation
to us so anything between zero and one
is going to be a kind of slightly
positive correlation it's not going to
be a super strong and we'll actually see
some examples on the next slide but yeah
so anything between zero and one is
going to be a kind of slight positive
correlation not super strong and the
closer you get to zero the more it means
no correlation so an example for the
zero case would be that it doesn't
matter how much coffee I drink in the
morning it's not gonna affect the
whether they're unrelated one does not
affect the other so I could drink you
know one cup of coffee during a sunny
day and one cup of coffee during the
rainy day and it's not gonna change the
weather it's not gonna affect the
weather so they're pretty much
uncorrelated and then we can also go
down into the negative range and so the
closer we get to negative one or if we
reach exactly negative one that
correlation of negative one means a
perfectly negative correlation and so
here we can take our example of coffee
versus tiredness and so the more coffee
I have the less tired I'm gonna be so
coffee goes up and tiredness goes down
so that's how we can kind of understand
this correlation and it comes from the
covariance so it was important to
understand the covariance we usually use
the correlation because the correlation
because we divided by the standard
deviation of each is much better fit to
our data now there is one thing that's
very important to remember and that's
that correlation does not imply
causation so just because two things are
correlated that does not mean that
one causes the other so a good example
of this would be if I live in a climate
where it's usually cloudy in the morning
and I know it to be sunny in the
afternoon but every morning when it's
cloudy I drink coffee and then it
becomes sunny in the afternoon that's
not even though they may be correlated
me drinking coffee and it becoming sunny
it me drinking coffee does not cause it
to be sunny that's just you know by
chance it's just because it happens
every day and by chance there's this
kind of correlation that appears but
that does not mean that me drinking
coffee you know
results in the weather getting better a
causation would be me drinking coffee
and me feeling less tired or me drinking
coffee and me feeling happy about it
because I like the taste those would be
causations so that's an important thing
to keep in mind just because things are
correlated does not mean that one causes
the other all right so let's see these
things on a graph and so here we have
the examples again that we've talked
about but we can kind of see how the
data would look like for different types
of correlations and so we can see a
perfectly the perfect correlation of one
so one goes up the other goes up we can
see on the left side and we pretty much
get this really nice straight line so
one value goes up the other value goes
up with it and then the closer we reach
zero the less related or the less
correlation there is between them and
then the more kind of variance we have
in data so we'll notice for the case of
perfect correlation which is the one or
the case of perfect anti-correlation
which is the minus one which again we
had the example of more coffee less
tired and in those cases you know we
have a very nice thin line and our data
doesn't jump around a lot but the closer
we get to zero the less we can see you
know one causing the other and the more
we can see our data kind of spread out
and so that's what correlation would
look like in terms of graphics in this
tutorial we're gonna go through
quantiles and percentiles all right so
let's get started
so what our quantiles
quantiles allow us to split our data
into certain regions that if we're
dealing with probability they all have
the same probability of occurring or if
we're just dealing with you know sizes
of data we want to split our data into
equal regions so that's what we can do
with quantiles is just splitting
everything up so that every time we
split it you know we have equal amounts
of data all right and so an example of a
quantile would be something known as a
quartile and so that's when we split our
data into four equal regions hence the
name quartile so a quantile is the
general name for doing this
splitting procedure and then if we say
quartile that means we're doing
quantiles but for four equal regions and
so this is something that you'd probably
often see unlike university admissions
pages or something like that and they
say the top 25 percent of our applicants
have at least a test score of like 90
percent or something you know and then
they would say the bottom 25 percent for
applicants or our admission or admitted
students or something like that have a
test score that is I don't know 70
percent or 75 percent or something like
that and then the median test score is
85 percent so that's how you would go
about quartiles is that you would have
you know the lower 25% the middle 25 to
50 then you've got the 50 to 75 and then
you've got the top 25 percent so the 75%
to 100 and so you've got these four
equal regions which also include your
minimum value at the very bottom your
maximum at the very top and in the
middle you've got your median values so
that's the value directly in the middle
it's because you're splitting it up into
four equal regions and so the value that
separates the second quantile which
would be the 25 to 50 from the third
quartile which would be from 50 to 75
that value there would be the median
value all right and so if we go into
percentiles so percentiles that may have
been a name that you you've probably
heard before percentiles again an
example of a quantile but instead of
saying you know like a quartile we do it
for for a percentile
I mean splitting it into 100 equal
segments
hence the percentiles of the perks name
at the beginning though that's that's
kind of where or the percent you may
have noticed percent means out of 100 or
so that's if you are familiar with
percent and that's also the same kind of
reasoning where this comes from and so
we've got percentiles which means
splitting into you 100 equal segments
and so on an example of this is often
used in test scores so if you've ever
taken something like the SATs or
something like that then you get a test
score but you also get a percentile and
the reason I've done that is it's to
judge not you versus the test but you
versus everyone else and so if it's a
difficult test than something like
getting a test score of 60% but you're
in the 95th percentile means your score
is actually a lot better and so what you
can say with percentiles for example is
that every percentile that you're in
means you're better than you know that's
many other people so for example if you
reach the 99th percentile that means
you're better than 99% of the people
that took the test the 95th percentile
would be 90 you're better than 95% of
the people that take the test or
something like that and so that's why
percentiles are often used for tests and
they're often used for normalization
because they allow you to take into
consideration you know these factors of
like is it a difficult test is an easier
test maybe more people are scoring
higher so they don't really judge you
directly versus the test but they
normalize you against everyone else that
took the test so you take the test you
get a score and then um the percentile
checks where that score lies relative to
everyone else and so these percentiles
they allow you to give a good
normalization and they allow you to do
great comparisons because they allow you
to kind of eliminate some of these
factors of test difficulty and of course
you know there can always be luck
involved and stuff and that may not get
filtered out on and
visual basis but if you do this for a
lot of students and that's also why it's
done in these kind of big standardized
tests is that you get a percentile along
with your score so that you understand
if you know maybe if your score is lower
but the test was really hard you can
still see you know I I did really well
because people found this test really
hard and it was even harder for them
than it was for me in this tutorial
we're going to talk about the importance
of data visualization all right so what
we're going to talk about is first we're
gonna look at the role that the computer
plays kind of for us and what role the
computer is actually made for then we're
gonna look at what role the human should
play in terms of data science then we're
going to look at presenting data and
finally we'll talk about interpreting
data alright so let's get started and
talk about the role that the computer
plays no computer is much much faster at
calculating than human because that's
what it's made for it's made for
crunching numbers it's made for doing
fast calculations you know if you think
about how faster computers are there in
the gigahertz range so Giga means
billions so they just do billions of
things every second and so they're
really good for doing repetitive things
because they can do them so fast and
then we can give them these logical
tasks in terms of programming and we
give them a structure and they just do
it and they can do it over and over and
over again they're not gonna mess up I
can just repeat the same thing they
won't get tired of it and they're really
good and they're really fast at doing
these things so that's the role that the
computer should play for you it should
be kind of a means to get these hard
number crunching and all of these things
done so there's there's really no need
for you to kind of work out all this
complicated math because your computer
can do it much better and much faster
than you and it's also less prone to
error if you code it correctly so that's
kind of the only part where you come in
and it's only gonna mess up if you mess
up but generally our computer does
exactly what we tell it to do and it's
really good and it's
really fast at it now what role should a
human play in terms of data science well
humans have naturally developed to
identify patterns and we've done this
first for survival so that if we're
walking around somewhere and we see a I
don't know a big predator or hiding that
we can identify that pattern of the
predator and we can kind of pick it out
even though it's trying to camouflage
itself so humans by nature have become
very very good at identifying patterns
and you can also see this if you look at
the clouds and you see things where you
see animal shapes and the clouds or
other things so those patterns aren't
actually there but humans have become so
good at identifying patterns we can see
things in many many places and so that's
what humans are really really good at
we're able to look at things and we're
able to pick out patterns now another
thing that's really good for humans is
we are very creative and through their
creativity we can also use memory and
bring it outside knowledge and we can
also use a general understanding so
these are all things that computers
can't do so computers are kind of a
means of getting stuff to us but once
it's actually there it's our job to use
our pattern recognition abilities and of
course you can train machine learning
algorithms for specific patterns later
on or specific cases and make them
really good at that but generally if you
don't know exactly what's gonna come
then our or your first step as a data
scientist would be to try to identify
these patterns use your creativity use
your memory you know bring in all of
these different things use all of these
different things that make you human and
use all of that on the data all of these
things that a computer just doesn't have
any access to okay
so usually you know you're considering
all this the best way to do all this
would be in terms of data visualization
so you can't just show spreadsheets with
a bunch of numbers that won't really
help you because looking at numbers it's
really hard to pick out patterns the
best way to do it would just be to plot
values and then if we have these visuals
in front of us then you know we can
really identify
we can see things go up and down and you
know we can see them fluctuating and we
can see them make very thin lines we can
just look at a graph and we can just see
things and of course you know we need a
little bit of practice to understand
what that graph is trying to tell us
but once we understand the graph and in
general then you know we can look at new
grass and we can just see things so we
can start to see patterns and they may
not always be true but that doesn't mean
you know and we can't pick them out and
then that's later on you would also do
some testing trying to see if those
patterns are true if they make sense but
generally data visualization is very
good for this because it allows you to
invoke all of your human characteristics
the things that are really good that you
know make us human the things that we
talked about and the last slide all the
things like the computer can't do and
sometimes you know you if you deal just
with just these numbers it's data
visualization is for you and one sense
so that you can see these things and try
to pick them out I use them later on but
also if you're trying to show these
things to other people so maybe you have
to do a presentation in a kind of
summary then you want to make sure that
your data visualizations are good
because the people that are going to be
looking at it are much much less trained
looking at data and analyzing data than
you are and so if you try to convey them
a message and just show them a big
spreadsheet with numbers and just point
out like here look look look these
numbers you know they pop up and they're
gonna be like what are you talking about
so that's why it's really important to
have really good data visualization
skills one of them is to enable you to
do your job but the other part of it is
to show it to other people and to kind
of help you convey information to them
you know and of course we talked about
statistical values and statistical
values are very important and they can
give us a kind of good idea about the
data and what's going on inside of the
data but visualizing data is just taking
it to the next level and statistical
values aren't enough there they can give
us you know they can help us they can
support us they can give us ideas but if
we really want to understand what's
going on sometimes we just have to take
a look at what's going on
and of course they are it's also
important to make sure you choose the
right visualizations and everything
because other times you know may just
look extremely weird but just this skill
of being able to present data both for
yourself as well as for other people is
very very important for a data
scientists and then we go over to
interpreting data and we've kind of
touched on this in the last section
already but really with data
visualization it just allows you to see
this data and it allows you to apply
some reasoning to the system and so you
can if you look at data either you see
something which is great you know that
means you can try to test something see
if it's actually there or you don't see
anything and that also tells you
something that you aren't really able to
pick out a pattern so that there isn't
there isn't anything obvious that's
going on there may be something
underlying that's more complicated but
obvious to the user you know it's not
there and so all of these things allow
you to you know kind of easily or much
more easily analyze your data and kind
of prepare what are you gonna do after
that so this data visualization it
really gives you a deep deep
understanding of what's going on with
your data and then when we interpret
this data and we look at these
visualizations you know maybe you see
dips and you know maybe you see some
hills somewhere we can try to understand
all of this by bringing in our outside
knowledge so again what the human is
really good at we can you know bring in
the context of things you know maybe
people are going out to lunch here and
so that's why activity decreases or
maybe everyone is coming to work in the
morning and so that's why activity
increases compared to you know 6 a.m. so
all of these things we can bring in all
of this context we can bring in all of
this understanding to try to interpret
the data try to better understand what's
going on and then of course we're gonna
see hopefully some trends or patterns of
course like I said these may not always
be there so we're actually so good at
pattern recognition that we can see
sometimes patterns that aren't really
there and so a good example again of
this would be just looking at the clouds
in the sky and you can see animal
patterns maybe but that's
really not there that's just our minds
you know identifying all of these
patterns and so yeah that's that's
pretty much why data visualization is so
important to a data scientist it's
because this whole you human aspect is
it's just key in data science it's key
and data analytics to be able to
understand what's in front of you to be
able to understand these this outside
knowledge to be able to contextualize
this creativity that's really key to a
good data scientist and a computer can
help you with all of this the computer
can help you you know do the number
crunching a computer can help you set up
the visualizations and it can plot
whatever you want for it but ultimately
it's up to you to choose the right
visualizations to do to look at the data
to be able to communicate the
visualization as well all of those
things are up to you and so that's why
the human is so so important in data
science in this tutorial we're going to
look at one variable graph so we're
actually going to see some of the types
of graphs that we can do you know that
we talked about in our last tutorial
where we just looked at the importance
of data visualization so now we're gonna
go into data visualization and look at
the types of grass that you may want to
use or that you may want to choose from
all right and so the graphs that we're
gonna look out in terms of one variable
graphs are gonna be histograms bar plots
and pie charts so let's get started with
histograms now we can see an example of
a histogram on the right but what's
really cool about histograms is that it
shows us the distribution of the data
and it shows us the distribution across
all the values in our data and so it
shows us what happens the least and it
also shows us what happens the most and
histograms it they let us see where our
data is concentrated and they also let
us see how its distributed and so the
through this it kind of shows a general
behavior and so really what histogram is
is it looks at each value and it just
looks at how often the value has
occurred and so what we see here for
example is that around 0 you know we
have the most occurrence of whatever
value we're looking at and as we move to
the left and as we move to the right
these values start to
drop off so they start to become less
frequent and so that's what histogram
shows us they the istagram shows us a
kind of frequency how often these things
occur and so there are different types
of histograms that you can encounter or
I mean generally a histogram is just
this plotting of frequency versus your
value and so there are different ways
that this histogram can look like one of
them is the one that we've just seen
which is a normal distribution or it's
called Gaussian like histogram because
it follows this Gaussian distribution or
this normal distribution that you know
but we can also have like an
exponentially decaying value so we start
off very high and the further we get
away from that initial value the quicker
it's then it gonna decrease and you can
actually compare that to the Gaussian
like or to the normal distribution so
the normal distribution looks more of
like a bell it kind of goes up and then
curves down slowly whereas the
exponential it cuts off very fast and
then kind of slows down later on so they
do have different behaviors and then of
course you know we can also get not just
one peak like we see in this first case
and the Gaussian like distribution but
we can also get things like two peaks or
we can even get three peaks or more we
can have very large extended peaks and
so our histograms there are means of
showing us how this data is distributed
there are means of showing us you know
what things occur most frequently where
is our data concentrated but that don't
that doesn't mean that they're gonna
have to have a specific value and so
they're or your specific shape so there
are many different shapes that our
histograms can take on and depending on
what shape that you get that also tells
us something very different about our
data all right so the next one variable
plot that we'll look at is going to be
bar plots and so what bar plots do is
they may look a little bit similar to
histograms at first but they are very
different in some sense because bar
plots allow us to compare across
different groups and so that's what we
see on the x-axis down there is we look
at different groups and so we use the
same
em but we can compare that variable over
different groups and so if we look at
that in example so what we see on the
right here is we look at different
countries and what we show is we show
the average income tax and so we see
that country B for example has the
highest average income tax whereas
country D has the lowest income tax so
through this you know we're still only
looking at the income tax variable but
we are able to compare us over different
groups over different categories if you
will so other examples would be if you
look at control groups and test groups
or if you're doing some like medical
study or maybe some psychology study or
something like that you always want to
have your control group and then you can
have different types of test groups and
then you can plot each of these groups
as a bar plot and you can look at the
same variable but you can look how that
changes over the different groups
another example would be something like
comparing male versus female heights so
you've got one group that's male the
other group that's female and you can
just plot their average height and then
the tax the income tax of different
countries which is what we seen on the
right over here all right and so the
last one variable graph that we're gonna
look at is gonna be pie charts and what
pie charts are allowed to do is they
allow us to section up our data on the
and then we can kind of split it into
percentiles and because of this we can
see what our data is made up of so the
whole pie corresponds to a hundred
percent and then we kind of cut it down
at different slices and through that
slicing and then hopefully also color
coding like we've done here and maybe
even labeling or most definitely
labeling so that you know what slice
corresponds to what value we're able to
see what categories you know or what
what categories our data is made up of
and so we can see what is most prominent
but we can also see what is at least
prominent in all of these things and
then again here we can see also
distributions not as well as in the
histogram but we can still see
distributions in terms of dominance in
terms of how many groups there are is
the data spread evenly is it you know
heavily concentrated in one part of the
pie all of these things allow you know
is that's what we're able to do with pie
charts we get this nice kind of group
overview of one variable so examples of
this would be you can look at NASA D
distribution in a university and so you
can have a pie chart and just each slice
of pie which is to represent a different
ethnicity and depending on you know how
much of our percentage they make up the
total University profile that's how big
the slice of pie would be and so you can
see dominance of some ethnicities as
well as you know minorities but you can
also see just by how many slices that
are you can see how many different
ethnicity groups there are and another
example would be you can split up star
reviews for a product so rather than you
know looking at the average mm star
review you can also just use a pie chart
and you can see how many of my reviews
are 5 stars how many of them were 4
stars 3 2 and 1
and so there you can again also get this
nice different overview of how the
review system would work now we're going
to talk about two variable graphs so the
graphs that we're gonna look at are
gonna be scatter plots line graphs 2d
histograms or two-dimensional instagrams
and box and whisker plots alright so
let's start off with scatter plots now
for a scatter plot what we're doing is
we're really scattering all of our data
points onto a graph and so pretty much
every data point that we have we kind of
put a little dot onto it on the graph
and scatter plots are great because they
allow us to see spread of data between
two variables so we're always plotting
one variable on the x-axis and another
variable on the y-axis and it just
pretty much allows us to see how the
data is distributed for these two
variables and then through that we can
also see more dense areas we can also
see some sparse areas and we can also
look at correlations so maybe you
remember in the lecture we talked about
correlations we were able to see through
scatter plots where those correlations
where or
weren't any correlation so all of these
things that's what scatter plots are
really really nice for in scatter plots
of course we can also use them to have
like we see here little clusters so not
everything needs to be connected by a
line or a curve maybe something is more
like a circle and so that's what scatter
plots can show us too they can kind of
show us these groupings and we see one
cluster here but maybe you know you have
bigger plots and then there would be
smaller you know like ten little
different groupings for different things
so it's got our plots are really great
for that because they just show us where
the data points are located for these
two variables and then we can you
ourself see you know like how how do
these look like do does one variable
affect the other is there may be certain
groupings that we can see where our
dense areas where it's sparse where are
things concentrated you know is
everything spread up all over the place
is it very very narrow and only in
specific region scatter plots allow us
to see all of these things very easily
and so some examples where we could use
scatter plots would be if we see if we
look at the graph on the right we can
look at something like a car price
versus the number of cars sold so each
of these data points pretty much
represents a car that's been sold and
then the x-axis tells us the price that
the car has been sold at and the y-axis
tells us the number of cars that that
have been sold at this price and so what
we see here for example very easily is
that the more the car is priced the less
it gets sold and then maybe you can
think of that in terms of well the more
its price maybe people don't want to buy
such an expensive car maybe they found a
cheaper version of it so maybe it's just
a branding thing which is why it's more
expensive
maybe there's something just as good
quality that's cheaper maybe people just
don't have enough money so that's
probably a big factor to that people
just don't have enough money to buy
these expensive cars and so that's why
they drop off and so it may look a
little bit different in terms of profits
but the higher the car is priced the
less we see it being sold so that's one
example of a scatterplot then something
else that we can look at
is maybe the income versus years of
education so only we would look at on
the x-axis how many years someone has
been educated and then we would look at
that current income and that would just
be a point on the on the graph and we
can do that for many many different
people and then we can see how different
education for different people how that
affects their current income so that's
another thing where we can do a scatter
plot for we can also go back to one of
the earlier examples that we used very
early on where we talked about people
traveling to work and we can just plot
the distance traveled versus the time it
takes and travel to work and then we can
see you know maybe some people travel
faster it could be that some people
travel the same distance but one takes
longer than the other because one goes
by car the other one goes by bike the
other one takes public transport all of
these things so all of that we can see
in these scatter plots and just kind of
take into account these different
situations and see how that all looks
for the more for the general population
of our data or just generally for our
data so scatter plots are really really
great as a kind of first go-to just also
identifying trends identifying regions
and just giving a good overview of your
data now the next thing that we'll look
at is going to be line plots and line
plots in some sense are kind of similar
to scatter plots so we have the same
bases of the X and the y axis but the
points are connected and now it's very
important to know when to choose line
plots and want to choose scatter plots
so line plots can carry a lot of
advantages with them because this
connectedness it makes it very easy for
us to see trends because we can see
where are these lines go not just trying
to connect the points in our head you
know like kind of connect the dots but
that's exactly what I'm a line plot does
is it connects the dots for us and so we
can see these lines it's great if we
want to see an evolution of something so
maybe we want to see an evolution over
time maybe you want to see an evolution
over space and evolution with people
something like that just if our data
points are connected it's great to use a
line plot so if we know that whatever
happened before is connected to what
happens and
it's great to use line plots because
line plots show us how things evolve
because they're all connected as a line
but if we're to do scatter plots and we
just kind of plot points randomly and
just because if we go back to or our
kind of car sold car price example just
because someone bought an expensive car
or if we look at the expensive car and
it's been bought say like five times and
we look at a cheaper car it's and bought
a hundred times there isn't really a
logical connection to make between the
two and so if we were to use line plots
where we should use scatter plots really
what we'd see is just a bunch of lines
all over the place and so that's why
it's important to kind of know when to
use lime stalks and want to use scatter
plots because it can be very very
helpful if you use a scatter plot
instead of a line plot it's gonna be a
bit more confusing because you have to
try to connect the dots yourself in your
head but if you use a line plot instead
of a scatter plot it's gonna look really
weird because there's just lines all
over the place and you can't really see
anything so an example where we could
use line plots is you have the typical
distance versus time so you can look at
you know how far someone or what time it
is and then you know how far someone has
traveled just a general curve of
distance versus time that's very very
common and you can look at the profit of
a company versus the number of employees
so the more employees they imply
employee how does that change their
profits so of course they have to pay
the employees more but maybe the
employees can also do more work and
hopefully you know that kind of cancels
out what you pay them and then increase
this company profits and then what we
can see on the right here is we can look
at your creativity and how that changes
with stress so you can see that the more
stressed out you are the less creative
you are and here it's also good to use a
line plot because you kind of gradually
advanced and stress and so each point
and stress is kind of related and the
higher you go up and stress the lower
you go down in creativity and so there's
this kind of relation where we can see
this evolution so the more you get
stressed out the less creative you
become and so line plots are really nice
here because there's not this chaotic
movement everywhere but it's very nice
and it's very easy
to see this line it's very easy to
follow okay so the next graph that we
can talk about is 2-dimensional
histograms now we've seen
one-dimensional histograms in the last
tutorial where we looked at the spread
of data and we looked at the peaks and
how you know things were distributed to
the right and to the left but we can
also do a 2-dimensional histogram and so
what a 2-dimensional Instagram is it's a
one-dimensional histogram but it's a
pretty much a histogram for every single
point of the other variable that we're
looking at so really what these things
allow us to see is they allow us to see
how the different distributions of the
two variable is relative to to another
so we can see here for example in the
red region that for those specific
values them they happen a lot so that
combination of values happens a lot and
so we're able to kind of pinpoint these
frequency occurrences again and we're
also able to look at drop-offs but we're
able to pinpoint that to two specific
values now rather than just 1 which is
what we did to the 2d histogram and
these things are much harder to see in
scatter plots because in scatter plots
if we have a value occurring a hundred
times it would just be the same dot and
the dot wouldn't get bigger now of
course you can make the dot bigger
yourself if you wanted to or you could
change the color or something like that
but really if you do a scatter plot and
the same thing happens a hundred times
it's just gonna look like one dot
whereas for two-dimensional histograms
we can see that it's not just you know
it's not just happening once but we can
actually see the frequency of those
variables or those those two variables
together so an example of a
two-dimensional histogram would be if we
look at ticket prices versus tickets
sold and so if you look at the lower
left corner and we can kind of see this
red peak so that's cheaper ticket prices
but the tickets are also sold often so
we know that tickets at that price are
sold quite often and these could be you
know like new rising brand bands these
could be like you know you kind of
standard bands that maybe you want to
take someone on a day-to but you don't
want to spend
much money on a ticket but still a
concert is a nice idea and so that's a
good ticket price that sells a lot of
tickets because it gives you the
pleasure of the event without making it
too expensive and then if you move more
towards higher ticket prices and then if
you move more towards more tickets sold
then you can see that for high tickets
high ticket prices which would be you
know like these big bands then we can
again see how many tickets we've sold so
we can see that for you know a higher
price and if we go up and ticket sold so
if you want to see lots of tickets sold
for a high price then the red Peaks are
gonna give us all of these more famous
artists so that's you know one kind of
application but of course there are many
many better ones it's just these things
you know if you're in the moment and you
you can kind of then you would realize
oh this is when a two-dimensional
histogram would be a great thing for me
to use so a lot of these graphs they're
great to know and once you're in the
moment then it's much easier for you to
pick out which graph would be best
representative finally the last graph
that we're gonna look at is gonna be a
box-and-whisker plot and I want box and
whisker plots allow us to do is they
allow us to see the spread within our
data so it's not just like a bar plot
which just shows us one value but we can
actually see the statistical spread so
we can see median values which is what
we see here we can see quartiles the
little dots on the outside actually show
us outliers and so what box and whisker
plots allow us to do is they allow us to
see the statistical information but they
allow us to see it visually and that
makes comparing across different groups
which is what we're doing here much
easier and so a good example of that
would be if we look at ticket prices for
football games for different teams so we
have different teams and different teams
of course use different stadiums and
there they have different popularities
and some teams may be much more
expensive or the ticket prices may be
much more expensive than other ones and
so we can compare these ticket prices
using box and whisker plots and then we
can see you know what is the higher end
of these
so those are gonna be the more luxurious
seats and then we go to the bottom and
those are going to be the less luxurious
seats probably the ones where you stand
and then you have middle values
depending on you know the standard seats
and where you are in the stadium if
you're close to the field if you're
further away from the field but you're
still sitting all of these things we can
kind of see here and that's what gives
us a spread we can compare that across
different teams and we can see the
spread across difference teams we can
also see which teams are more expensive
you know where do the prices vary the
most for specific teams and maybe some
teams have a super launch and then they
have your you know standing places that
are just much cheaper and so you would
see a lot larger spread or maybe some
teams just have you know only seats and
see you'd see a much lower spread and so
all of these things were able to compare
using box-and-whisker plots over
different groups in this tutorial we're
going to talk about 3 and higher
variable graphs so the graphs that we're
gonna look at are is gonna be heat maps
and then we'll also look at multi
variable bar plots as well as how we can
add more variables to some of the lower
dimensional graphs that we've talked
about earlier all right so let's start
with heat maps now what heat maps allow
us to do is they allow us to plot two
variables against each other and the X
and the y and they laws to show an
intensity or a size or something like
that in the Z direction or towards us so
an example of this which is kind of what
I've tried to illustrate on the right is
a customer moving through a storm and so
we can track the path of the customer in
the X and y direction of the store so
you can kind of get this bird's eye view
and see where they move to and the
darker spots actually tell us the
positions where they spend more time at
so we can see that they spend a little
bit of time you know at the beginning
they moved in and then they stopped
lunch was what we see with dark spot
being maybe they found like the candy
aisle or something there was a specific
piece of candy that they wanted and then
they moved on and then they started to
go towards the corner around the corner
a little bit and maybe they reached the
fruits in the vegetable section there
and picked out several things and then
they started to head towards the
checkout counter which happens at the
very end and they were moving at a more
constant pays sometimes they stop to
look a little bit but they just kind of
continued moving on and so the three
variables that we've shown here as we've
shown their exposition in the store
we've shown they're by position in the
store and to their color we've also
shown the time that they spend at each
position so that's what we can use heat
maps for and then another example the
heat map would for example be if you
take a flashlight and you move it over
the screen and really what you're
showing is the amount of time that
you've shown the flashlight onto a
specific region so that's kind of
another example the heat map but usually
heat map as the name implies it allows
you to track positions and so it's very
often used for things like tracking
customers through stores or just
tracking general people location where
they like to spend their time and the
intensity that you see in terms of the
color is usually the amount of time that
they spent there all right so we can
also do multi variable bar plus a multi
varied bar plot and so it is very
similar to a single bar plot where we
just plotted one value over different
groups but rather than just plotting one
we kind of cramped them together and we
plot several and so an example of this
would be that we plot you know goal
scores
I'm goal scored 14 the shots taken on
goal as well as the shots on target and
so we can see maybe there are teams that
shoot less on goal without score less
but that's because they also shoot less
and therefore they also shoot less on
target or maybe there are some teams
that do score a lot and that's because
they shoot a bunch but they just don't
hit the target that often or maybe there
are really good teams that score a lot
and they also shoot a lot on target and
so all of these things were able to then
compare over different groups and so
that's what we can use multi variable
bar plots for if there are several
variables that would give us a better
understanding of the system than just
looking at the variables in one at a
time but it also be really cool if we
could compare all of them then we could
use multi variable bar plots for that
and just plot them on the same bar plot
and then we can see how they change you
know within a group we can also see how
they change over
different groups okay and something that
we can do is we can also just add extra
dimensions to lower dimensional graphs
that we've had and so we're kind of
limited to three dimensions because
that's the amount of space dimensions
that we live in but if we take a scatter
plot for example where we started off
with just the X and the y axis and
points located what we can do is we can
actually add a third axis so we can take
the X and the y and then we can add a Z
and that gives us an extra depth
dimension which is exactly what we see
here so rather than just plotting unlike
a two dimensional field unlike a plane
we even actually plot it in a volume and
so we can see this kind of scattered
ball that we've done kind of kind of all
that we've done here which is kind of
located at the center of our plot and so
this can be really cool because it
allows us to see depth to the problem
with this is that we have snapshots
every time and so really we're looking
at two-dimensional snapshots and so to
get the best understanding of this we
need to rotate our scatter plots or our
plots as we do them so that we can also
add in our depth perception because
right now for looking at it it may look
three-dimensional but really it's just a
two-dimensional snapshot and to get the
best understanding if our scatter plot
is located more towards us and more
towards the left or something like that
or maybe it's just really high and close
to us or maybe it's really low and far
away to understand all of these things
we need to be able to rotate our scatter
plot so that we can see it from
different angles which then gives us
this depth perception and we can do the
same thing with 3d line graphs so here
we see an example of maybe the position
of a skier as they're skiing down a hill
and then we can kind of trace that
through time and we see that they're
kind of they're going down the hill in
this nice exact motion as you should and
we can just track their position over
time so here we've added this extra
dimension to the 3d line graph rather
than just taking maybe a time and a
position at a time or something like
that we've added a second positions or
actually even a third position so we've
got the X to the one there's that
position and then we just trace it
and over time and so that gives us this
whole line here and so that's how we can
take these lower dimensional plots that
we've looked at before and we can just
add extra dimensions to them if we want
as long as it's still easy to see as
long as it makes sense what we're
looking at yeah we're really just able
to maybe just slap on another direction
there and you know compare another
variable in this tutorial we're going to
touch on the third major section that is
really great for data scientists or that
should be an essential of data
scientists which is the ability to
program okay and so why do we program
well there are different reasons why we
want to be able to program the first one
is going to be the ease of automation
the second one will be the ability to
customize and finally it's because there
are many great external libraries for us
to use that it would just make our job
so much easier alright but so let's get
started let's talk about the ease of
automation for us what do I mean with
that well being able to program it
really allows you to prototype really
fast allows us to automate things and it
also gives us the extra benefit if if we
have something in our mind we can just
take that and kind of put it into the
computer by programming it and so we're
able to automate everything very fast
and we don't have to do these repetitive
tasks
you know maybe copy pasting stuff into
or from Excel or all these things and if
we just want to repeat something or we
want to quickly change something up and
just change a small thing we don't have
to do a lot of stuff we can just change
that in our code and then click play and
let the computer take care of all of
that for us rather than us having to do
everything manually so it's very easy
for us to automate things um and also
for doing reports it's very easy to
automatically create these reports you
know all you have to do is set up your
program to deal with the data that
you're going to give it and then I can
automatically create reports every week
and the reports can be different because
you give a different data and it should
still look the same but the data the
values can be different and so that will
just automatically create all these
reports for you and you don't have to do
that all yourself the program does it
for you
but you've built the program and you're
giving it this different data so you're
still doing all of the analysis it's
just you get to skip the part of copy
pasting and like looking across and
taking over the values and doing all the
formatting of just doing the same report
over and over and over again all of that
is taken care of for you and all you
have to do is just put in the right data
you know write out everything that you
want to do and then click play and let
the computer handle all that for you
because remember that's what the
computer is doing and good at doing
doing these repetitive tasks okay we
also want to be able to program because
it really allows us to customize it's
very easy once we go into data analysis
and when we see things that we get these
ideas that we want to expand or
different directions that we want to
progress our analysis into and being
able to program it really just allows us
to take all that and put it as a code
and just choose that direction and we
can very easily dive much deeper into
our analysis and discover things fast
because it's up to us to where we want
to go and so this ability to customize
with programming it's it's very very
important because we're not reliant on
anything else we're not reliant on you
know some software and maybe it breaks
down or maybe we don't know how to
perfectly use it and we have to read the
manual and read a like a Help section
and know we know how to program and we
just type down exactly what we want to
do exactly where we want to take it
exactly what we want to see and we can
customize very very fast with that we
can also prototype very very fast with
that and maybe if a visualization is not
working to turn a scatterplot into a
line plot it's very easy you just change
one word so all of these things are very
very easy to do with programming because
we have all that power at our fingertips
and we can just you know change
everything that we're looking at
everything that's being calculated maybe
we want to calculate an extra thing and
take out something else because it's
irrelevant all of these things were able
to customize and all of that we can do
because we're able to program so really
what we're doing is we're making the
data ours so we're taking full control
of the data we're taking full control of
where we want to go with our analysis
what we want to see and what we want to
show all right
so let's talk about first libraries but
also give you two great Python libraries
that you should you know maybe feel
comfortable with or that you should
maybe consider using for data analysis
so first of all what are libraries will
libraries are pieces of code have been
pre-written by others that you can just
take in and use and so a very good
example of this is something known as a
math library and so that has all the
square root functions taking to the
power you know taking the exponential
the sine the cosine all of these things
that you know and you want to use but
you don't want to program yourself so
like it pretty much avoids that middle
step of you having to program the
equation to calculate a sine because all
of these things those are things that we
don't want to do we don't want to get
distracted from our target we want to be
able to do exactly what we want to do
without having the program completely
other stuff and so that's what libraries
are great for they're developed for by
the community for everyone to use you
know everyone is helping each other and
these libraries they just bring a lot of
power with it and so one of these
libraries is called pandas and pandas is
pretty much like Excel but it allows us
to do or we can do programming with it
which just makes it so much better
because we can do things so fast with it
we can do all this customization we can
do all this automation whereas you know
like Excel if you give it too much stuff
too much to run it would just start to
crash because it has to handle all of
this other things all these other visual
things you know the UI and there's a lot
more it's a lot it's not a structure as
well whereas in programming the program
you know your computer just goes through
everything step-by-step it doesn't have
to take care of all of these
visualization things it just does the
calculations down below but we can still
do all sorts of data management with
them so we can shift our data around we
can drop columns we can split things up
you know we can split things up our row
we can pick out certain rows we can even
do statistical calculations on our data
so we can say you know hey calculate the
mean for this we don't even have to you
know make our own formula for
calculating the mean or for calculating
the standard deviation or for
calculating correlation between
different columns all of that can be
done with pandas with just a you know
a couple of key words and so it's really
easy to do data analysis with it because
all of the functions that are there and
we know exactly what we want to do but
we don't have to write the code for all
of it so if you wanted to look at
correlations we just say hey pandas do
correlations rather than having to you
know code all the correlations for
ourselves and doing you know coding that
whole algorithm and that makes it really
easy and really fast to get results and
to get to where you're heading because
you don't have to go into any of these
mineral places you can pretty much just
skip the middleman of having to you know
write all of those algorithm to yourself
and you can just use them so that you
have your start you have your idea you
know exactly what you want to do and you
can do exactly that to get to your goal
the other library that is very cool
would be matplotlib which is what I use
a lot for data visualization it allows
me to create graphs allows me to
visualize my data allows a bunch of
customization so I can really just move
everything around in it I can move my
spines I can turn things on and off you
know all of these things are very easy
to do with MATLAB there's a lot of great
customization that I'm able to do with
it so these are like kind of two basic
Python libraries that you should
probably maybe get to know em or you can
look at some of my other courses and one
of them pandas would deal with the data
analysis part and matplotlib would help
you deal with the data visualization
part of it so that's it that's a super
basic breakdown of the three main
components of the otherwise vague term
data science if any of this has piqued
your interest then you may have a data
science future ahead of you and I
encourage you to continue to pursue your
interest if you want to learn more from
me I've got a blog on my website coding
with max comm that dies more into
different topics related to data science
you can also get access to some of the
resources such as cheat sheets and
workbooks that I've compiled for you
there if you're serious about learning
data science you can also check out my
courses on data science which are
designed to teach you all you need to
know about data science even if you have
no prior experience of course if you
have any questions that aren't answered
at my website you can always feel free
to reach out
me personally