Voiceover:Karim has developed a new method
for measuring cholesterol level that doesn't
involve any invasive procedure, like drawing blood.
Well, that would be very nice.
Regulations demand that the test should be correct
99% of the time.
Test should be correct 99% of the time.
So he had to verify that his method meets
the standards.
He got a random sample of 100 people, tested
their cholesterol using his new method,
compared the results to blood tests
performed on the same people, and found
they were the same 95% of the time.
Since blood tests are always correct, he concluded
that his method was correct 95% of the time.
That seems reasonable, at least for his sample
right over here.
In light of the results, Karim had to test
the hypothesis that the test is, in fact,
correct 99% of the time.
So he wanted to test the hypothesis that
the test is, in fact, correct 99% of the time.
So he performed 100 computer generated
simulations of random samples of 100 tests.
So there's a lot of 100's here.
So he did 100 simulations and each of those
samples essentially simulated the test results
of 100 people.
So it's like using a computer that assumes
this assumption.
That assumes that the test is, in fact, correct
99% of the time, and to simulate 100
of these samples.
100 samples, each of them having 100
people in them.
Supposing the true percentage of correct
result is 99%, to find how likely it is
that a sample would have 95% correct results.
The results of the simulations are plotted below.
How do the results of the simulations affect
the likelihood of the hypothesis that Karim's
test is correct 99% of the time?
So I know this can be a little bit confusing
because we have all of these ...
We're testing and 100's and all of this.
But what he's doing, he said, "Okay."
He got his test, on this one sample, his test
was right 95% of the time.
But what he wants to really figure out is,
he wants to get a sense of what's the probability
of that happening, assuming that his test
is actually correct 99% of the time?
So he has the hypothesis.
So if we assumed the hypothesis ...
If you assumed, if you assumed his test,
his test, his test correct, correct 99% of the time.
So that's the hypothesis right over here.
Then if we assume this, he's trying to figure out
what's the probability that he gets these results?
What is probability, probability of this
right over here?
Because if this probability is really low,
if the probability of getting this is really low,
then he can say, "Well then I can't make the ...
"Then I really don't feel good about
"assuming this hypothesis. That maybe we can't
"go with the hypothesis if the probability of this ..."
If the probability of him seeing what he saw
is very low, given this assumption.
So let's look at that.
So let's interpret what's happening
in this chart here.
So remember, this computer is running 100 ...
It's simulating 100 samples, each of those
samples have 100 people.
So for each sample ...
So let me make it clear.
So this is sample one.
So sample, sample number one and then there'd
be sample number two and there's going to be,
there's going to be 100 of these samples.
Each of these samples, the computer is saying,
"Okay, there's 100 people in each sample.
"One, two, three ..."
And it goes all the way to the hundredth,
all the way to the hundredth person.
For each of these the computer is saying,
"Okay, I'm assuming there's a 99% chance
"that it's correct."
So the computer uses its random number
generators, essentially kind of trying
to roll the dice.
And 99 out of 100 times, it'll say
the sample is ...
The new test matches up to the old test.
But then, every now and then, there's
that one percent chance that it says,
"Okay, it doesn't match up."
So let's say that this one had 99 checks
and then one X.
Then this would fall into this category, right
over here.
Then this sample, this sample right over here,
would correspond ...
Let me do this in a color.
So this sample, right over ...
I mean a new color.
So this sample right over here would
correlate to, say, this dot.
And then if they did the test again and
maybe it got 99 checks and one X,
then it would be ...
Say it's number two, say it would be there.
Let's say in one of the samples ...
Let's say in one of the samples of 100, I guess,
virtual people right over here, it got ...
You know, there's a 99% chance of getting
a check here, so it got all 100 people got a check.
Then that would be ...
Then it would stack it up right over here.
And so we see using the assumption that there's
a 99% chance of the test matching up,
or there's a 99% chance that the test is correct,
it got all of these results.
And we see that out of these 100 samples it took,
none of them, none of them, gave what Karim
observed, right over here.
So it seems like, if you make this assumption,
the probability of observing what he observed
right over here seems very, very, very, very low.
So this, to me or I guess to any reasonable
statistician or reasonable person, these results,
the results he got, the results that he both observed
and the results of the simulation tells us
that the likelihood of the hypothesis being
correct is low.
So this is a ...
The likelihood of the hypothesis that Karim's test
is correct 99% of the time, the likelihood
of this hypothesis is very low.
Very, very, very low.
Given this observation and the results
of the simulation.
And so you'd be likely to reject the hypothesis.