So in this screencast I hope to provide a little bit of insight as to what it means
when you conduct a two sample t-test, and to begin with we have to remember what it
means to have a normal distribution. Everything that we'll talk about in this video assumes
that the measurements we collect are normally distributed. When we look at this bell shaped curve it
assumes that we'll have more samples gathered at the center of it and there might be a small
chance that there'll be a sample off to the left, but by and large most of them will be
concentrated near the center of the normal distribution, and this could be something
like the height of all adults in the world, it could be the concentration of a chemical,
something like that, high concentration, low concentration, where the bulk of them are
concentrated around the middle of it. It could also mean something like breaking strength
of different materials for example, or maybe the speed for different vehicles along the
length of a track. It's important to distinguish the difference between what's known as a sample
and what's known as the number of measurements. In this case I've collected one sample. We
could call it temperature if you wanted to, it doesn't matter what we call it, but I took
ten measurements in that sample. Unfortunately what we never get to know is the true distribution
of the population. The only way you could know that is if we could measure everything
in the whole population. For example in an exam, if we had 120 students in the class
after I grade all the exams I compute the mean of that, I do know the population mean
of all the exams in the class, but if it were something like temperature we could take an
infinite number of temperature measurements of something. If we were breaking parts we
could break all of the parts. Let's say I had 100 parts and I am trying to figure out
the mean of those parts, well I could break all 100 of them but unfortunately I don't
have any parts left, so we don't ever get to know the population mean in the wide majority
of cases. So I am going to get rid of this bell shaped curve because we don't know it,
and all we have are these measurements, and what I want to do is make some axis, and I'll
grab the measurements and we'll plot them on top of the axis. So I've taken these measurements
and plotted the vertically with the low value the bottom of the axis, and the highest value,
measurements number ten, at the very top of it, so I'll label this axis "values," and
again we don't get to ever know the true population distribution so I'll get rid of that, and
I'll also get rid of the extra points, and let's call this sample 1. And if we want to
pretend these temperatures, or these values are temperatures or something like that, let's
pretend that we're testing a new refrigeration unit, and this sample 1 came from the old
refrigeration unit, or the unit that we're familiar with, and the new one we'll call sample 2,
and when we test our new refrigeration unit we get temperatures that are somewhere right
in this range. So these are my, I took, let's see, one, two, three, four, five, six different
measurements from the new refrigeration unit, and these are temperatures of high temperatures
at the top, we would look at this and boy I would be convinced that the new refrigeration
unit, the mean temperature from the new refrigeration unit is indeed colder than the temperature
from the original refrigeration unit, but what's interesting, here I've overlayed the
distributions again, and these distributions give me the population means, here's mu(1)
and mu(2), but remember we don't get to know these, and when we look at it, it may look
obvious that mu(1) and mu(2) are equivalent, and if they are equivalent it means that the
new refrigeration unit did nothing what so ever, there was no improvement over the original
refrigeration unit, but when we look at the data alone it sure looks like the new refrigeration
unit does a better job than the old one. In the language of statistics we would say the
null hypothesis is that the old refrigeration unit, the average temperature from the old
refrigeration unit is equal to the average temperature of the new refrigeration unit.
The alternate hypothesis is that their means are different, and when we show the distributions
it seems pretty obvious to us that we cannot reject the null hypothesis. So we look at
this and the null hypothesis is true if we had access to the true populations. But when
I get rid of those population distributions it sure looks like we could refute the null
hypothesis and go with the alternate, it says that the new refrigeration unit is colder
than the old one. But you and I know, because we've seen the population distributions, that
the null hypothesis is true, that the old and the new refrigeration units are exactly
the same. So what we happen to do, when we look at this data, you and I look at it without
the distributions, we're apt to reject the null hypothesis and say offhand that the new
refrigeration unit is better than the old one. So what we've made is known as a type
I error. In other words we've rejected the null hypothesis when the null hypothesis is
actually true. What I'll show you next are some simulations that I wrote in some computer
code that picks sample values randomly across a normal distribution. For example, here's
a case in which I gathered five data points from two different samples, and I sample these
from the same population distribution. Both of these samples were gathered from the same
normal distribution. Even without the normal distributions presented we're not likely to
reject the null hypothesis, I look at this and I say well these values look awfully close
to the other one, I don't see any evidence to reject the null hypothesis, it looks to
me, even without the distributions, that their true population means are equivalents. And
here's another run of that simulation, again they look fairly equivalent, a little bit more
spread when we can actually see the distributions it's obvious to us that there is no difference,
but again we never get to know what these distributions look like. And here's a simulation
I ran with fifteen measurements from the two different samples, and again they came from
the same population mean in both cases. Here's another one, let's ask the question "is mu(1)
equal to mu(2)?" Certainly I'd say that the sample means are different, the sample mean
for sample 2 is much greater than the mean for sample 1. Again I am inclined to reject
the null hypothesis and say the population mean for sample 2 is indeed greater than the
population mean for sample 1. But as it turns out they aren't any different, I used the exact
same distributions to generate these numbers with my program, and they both came from the
same underlying normal distributions. What I've done again is incorrectly reject the
null hypothesis and made a type I error. But in reality we don't ever get to know those
distributions, all we know is to ask the question "is mu(1) equivalent to mu(2)." I an extreme
example we would certainly reject the null hypothesis for something like this, the data
points between the two samples don't even overlap, it's clear that sample 2 has a lower
mean than sample 1. But they aren't, these again came from the same underlying normal
distribution. In this case again we rejected the null hypothesis when we shouldn't have
and made a type I error. However, in this situation the odds of this happening are something
like 0.02 percent, something like 1 in 5,000 chance of us doing this just randomly. There's
a very low probability, if we looked at the raw data, that without the distributions there's
a very low probability that these actually came from the same population mean. When I
said that the odds of these two samples actually coming from the same population mean were
about 1 in 5,000, it's about 1 in 6,000, and try it yourself if you like, do a t-test
between sample 1 and sample 2, and you'll find a p-value, it'll return a p-value equal
to 0.00017. This p-value, what it gives us is the likelyhood that these two samples actually
came from the same mean, so we could rewrite this even as 0.02 percent, or we could say
this is, I get about 1 in 5,880. Typically we'd say that if there's less than 1 in 20
chance, or there's only a 5 percent chance or less that these two came from the same population
means then we would reject the null hypothesis and we would say that the population means
are different, and that 5 percent, or that 1 in 20 chance is known as alpha, and the
p-value represents the probability that we incorrectly assumed that they came from different
population means, or we made a type I error. We rejected the null hypothesis when we shouldn't
have.