If you want a quick introduction to the world of statistics, this is the perfect video for
you.
Today, we’ll explain several key concepts: the various types of data, measures of central
tendency, dispersion, statistical distributions, hypothesis testing, and confidence intervals.
A solid understanding of statistics is crucial in data science and fields like medicine,
economics, psychology, social sciences, business, sports, and others.
Moreover, this is valuable knowledge needed in a professional environment.
We hope you stick around until the end of the video because an engaging surprise awaits
you.
To illustrate statistical concepts understandably, let’s provide an example.
Suppose a cooking class teacher asks all 36 students to complete a questionnaire at the
beginning of the year.
For various reasons, only 31 students submit their questionnaires.
The teacher realized she would not have data on the entire class.
The available data is limited; we say it’s a sample.
Had the teacher been able to collect data from all students in her class, then the data
would be from the entire population.
A population includes every group member of a group we’re interested in, while a sample
is a smaller group taken from that population.
Even though sometimes—as in this case—we cannot collect data for the entire population,
sample data can be beneficial.
They can allow us to make informed inferences about the whole population.
(We’ll see how to do that later in the video.)
In the questionnaire, students were asked to provide two data types: numerical and categorical.
Numerical data are expressed in numbers, which can be measured—for example, students’
age, height, and weight.
We can count, measure, and add years and kilograms because this is numerical information.
On the other hand, categorical data describe groups.
A student’s preferred sport, gender, hair color, or eyes are categorical data.
We can also consider Yes / No questions, such as “Does your family live within 1 kilometer
from school” categorical data.
Adding or subtracting categorical data doesn’t make sense.
We’re ready to introduce the first statistical metrics in our crash course: the measures
of central tendency.
Such metrics give us an idea about the center of distribution or the typical value in that
dataset—which might sound more complex than it is.
Consider that the 31 students who submitted the questionnaire were asked to fill in their
heights (numerical data that can be added).
These are the student heights.
The measures of central tendency are mean, median, and mode.
Let’s compute the mean (more frequently called arithmetic average), which equals the
sum of all heights divided by 31 (the number of students in our sample) and the average
height of an individual.
We obtain 171.2 centimeters.
Then we have the median (the middle number in an ordered data set).
So here, we have 31 heights in ascending order—so the 16th number (172) is the median, the middle
value.
The mode is the value that occurs most often for numerical and categorical data.
Only one height is observed more than two times: 172.
Therefore, this is the mode of our sample.
The average height is 171.2.
But we also have students who are 191 and even 200 centimeters tall.
We need to introduce dispersion measures to account for how observations spread out in
our sample.
The range is an intuitive metric that measures the difference between the highest and the
lowest value.
So, in our case, the range is given by subtracting the shortest individual’s height from the
tallest individual’s height (200 – 151).
The range of this sample then is 49 centimeters.
A more popular and frequently used metric of dispersion is variance.
Variance better represents the overall dispersion in a dataset because it considers an observation’s
distance from the mean.
Note the formula for sample variance.
It asks us to sum the squared difference between each individual’s height and the mean.
This sum is divided by the number of observations in the sample minus 1.
We subtract 1 because we’re dealing with sample data, and this requires an adjustment
since the sample formulas are the unbiased estimators of the population formulas.
I know this sounds odd, but we must bear it in mind when working with sample data.
(Please remember that depending on whether we work with population or sample data, we’ll
need slightly different formulas.)
Here’s the variance formula when working with population data.s^2=\sum_{i=1}^{n}\frac{{(X_i-\bar{X})}^2}{n-1}
\sigma^2=\sum_{i=1}^{n}\frac{{(X_i-\mu)}^2}{n}
Sample variance allows us to understand how data is spread out with respect to the sample
mean.
Once we have calculated the sample variance, we can quickly obtain the sample standard
deviation, given by the square root of the variance.
Think of standard deviation as the average dispersion from the mean.
If the standard deviation is high, we deal with a dataset with highly spread-out observations.
In contrast, a low standard deviation indicates the dataset observations are concentrated
around the mean.
In our case, the average dispersion from the mean is 12.1 centimeters.
When we study how the observations in a dataset are spread out, it makes sense to talk about
distributions.
These inform us about the overall shape of the data and show how frequently a value occurs
in a dataset.
Let’s create a histogram—one of the most popular data visualizations in statistics—using
our students’ heights.
Our software organizes the dataset in bins and intervals (heights in our case).
If we use 5 intervals, Excel will count how many tall people there are between 151 and
160.8 centimeters, then how many are between 160.8 to 170.6, etc.
Note the graph showing that the student heights are almost symmetrically distributed.
Shaped like a bell curve, the best-known and most widely used distribution is the normal
distribution.
Most of the data falls near the mean, with fewer values at the extremes.
The mean is the highest point because it coincides with the mode.
Observations are symmetrical on both ends.
The normal distribution is beneficial because many real-life data follow this pattern.
This is almost what we have in our example, but we have too few observations to depict
a proper normal distribution.
But this gives us an idea that our graph could have approximated a normal distribution much
better if we had more sample data.
Sometimes, if a distribution we work with is asymmetrical, we say it’s skewed.
Positive skewness means a longer tail on the right side—that is, if we have had more
students who are 190 or taller.
Negative skewness means more observations (asymmetry) on the left side.
In our case, the skewness is 0.23, which means a slightly longer right tail—largely thanks
to the significant outliers of 191 and 200 centimeters.
Kurtosis is a metric that tells us how fat the tails are of our distribution.
Once the teacher has calculated the mean height of the students in her class, she can form
an expectation regarding how tall, on average, the students in her class are.
The mean is a single number—hence a point estimator.
But often, decision-makers feel uncomfortable working with a single number, which doesn’t
give them information about dispersion.
This is where confidence intervals enter the stage.
A confidence interval provides a range of possible realizations of the true population
value—an interval with which we are confident the population parameter will fall in 90,
95, or 99% of the cases.
In our case, the teacher wants to obtain a range that depicts the average height in 95%
of the subjects rather than relying solely on the mean sample height received.
Note the formula that allows us to calculate a confidence interval for our sample.
The first component of the formula is the sample average we calculated earlier.
The second component is given by the product of t and the sample standard deviation divided
by the square root of the number of observations.
We call this term the Standard Error.
The sample standard deviation and the number of observations are already known.
Let’s understand what t is.
Because the normal distribution is perfectly symmetrical, statisticians have been able
to create two statistical tables with values.
One table (the z-table) is for when the population variance is known, and another (the t-table)
is used when the population variance is unknown when working with a sample.
Both tables calculate the area under the curve considering the number of observations and
the degree of confidence we want to have.
The degree of confidence will vary if we opt for 90,
95, or 99% certainty.
Let’s see how this works in practice.
In our case, we must use a t-distribution table because we deal with sample data.
OK.
Let’s opt for 95% probability, meaning we’ll leave a 2.5% probability of estimation error
on each side.
Considering that we have 31 observations—30 degrees of freedom because we have n-1—we
obtain the t-value equal to 2.042, which allows us to compute the second component of the
formula called the margin of error.
We add and subtract the margin of error to obtain our confidence interval.
With a 95% degree of certainty, the average height of students falls in the range of 166.1
to 176.2 centimeters.
In this case, our population variance and standard deviation are unknown.
We used the t-statistic.
But for sufficiently large samples—even if the population variance is unknown—we
can use the z-table; the result after 30 observations would be very close.
The rule of thumb is always to use the t-table with less than 30 observations.
If your observations are more than 30, you can use either the t- or z-table.
Another fundamental task in statistics is hypothesis testing, which uses the same calculation
mechanics and logic as confidence intervals but reframes the problem slightly differently.
So considering our sample, what if the teacher’s intuition was that the average height of all
her students was 170 centimeters?
We want to test if this estimate is correct.
Two hypotheses are made: the null hypothesis (denoted H zero) and the alternative hypothesis
(marked with H1).
The null hypothesis is the one to be tested, and the alternative is everything else.
In our example, the null hypothesis indicates that the mean student height equals 170 centimeters,
while the alternative shows that the mean student height is not 170 centimeters
The teacher will check if 170 centimeters is close to the sample’s true mean.
If it is, the teacher will fail to reject the null hypothesis.
Otherwise, the teacher would be able to reject the null hypothesis.
The concept of the null hypothesis is similar to “innocent until proven guilty.”
We assume the mean height is 170 centimeters and try to prove otherwise.
Because we’re using the equal sign, we say this is a two-sided test.
We can have a height lower than 170, but alternatively, it can be higher than 170.
We consider both tales, hence a two-sided test.
Note the formula we need to use.
Looks familiar to what we had before, doesn’t it?
This time we solve for t by subtracting 170 from the sample mean and dividing by the standard
deviation divided by the square root of the number of observations.
We obtain the t statistic, which is equal to 0.5.
The next step is to compare the t statistic with a value from the t-distribution table
determined based on a pre-selected (95%) confidence level, remembering that we have 31 minus 1
equal to 30 observations.
Using a t-table, the critical t-value for a two-tailed test with 30 degrees of freedom
and a 95% confidence level is 2.042.
Since the absolute value of our t statistic (0.5) is less than the critical t-value (2.042),
we can conclude that our result is not statistically significant at the 95% confidence level.
In other words, we fail to reject the null hypothesis that the average student height
is not 170 centimeters.
Brilliant!
This was our short intro into the world of statistics.
Thanks for staying with us until the end of the video.
Now we have an incredible surprise for you.
But before revealing this, please like the video and subscribe to our channel with a
notification bell.
We’ll post great content for you.
So—the surprise.
At 365 data science, we love statistics and want to help you learn the fundamentals.
To facilitate that, we created a series of calculators that allow you to make statistical
computations and show the step-by-step solution with symbols and math calculations.
Below each calculator page, we have detailed articles teaching you the ins and outs of
these concepts.
Check out these pages as a follow-up to this crash course.
Using the data in our example, we can input it into these calculators and see how the
respective mean, median, and mode are computed.
Of course, we can do the same in the variance and standard deviation calculators.
We built these excellent (free) tools especially for you.
Please share them with anyone who might find them helpful.
You can find and download an Excel file with all calculations shown in the video in the
description.
OK.
This will do for now.
Thanks for watching, and good luck on your statistics journey.