Whenever you make a decision in life, you could also make a mistake. You could be
wrong the same is true in hypothesis testing. When you make a decision about
whether to reject the null hypothesis or not, you could make a mistake. We call
those mistakes type 1 and type 2 errors. When you make a decision to reject a
null hypothesis or accept a null hypothesis (which is really failing to
reject the null), you could make a mistake. The null could be true, but you rejected it,
or the null could be false, but you accept it. A type 1 error is rejecting a
null hypothesis that is actually true, i.e., a false positive. A type 2 error is
accepting a null hypothesis that's actually false, aka false negative. Now I
could explain this with a 2 by 2 matrix in which we consider the true state of
the world against the outcome from our sample, and determine if we've made a
correct decision or an error. But I find that at the beginning, this is a little
too complex, so I'd like to start with a more accessible example. To illustrate
errors and decision making, let's consider the example of pregnancy
testing. At any given time, most women in the world are not pregnant, so that is
going to be our null hypothesis. If we select a random sample of women and
administer pregnancy tests, there are four possibilities: possibility number
one - the test correctly identifies a non-pregnant woman as not pregnant.
You're not pregnant, the test says you're not pregnant - correct result. Possibility
two: the test correctly identifies a pregnant woman as being pregnant. You are
pregnant, the test says that you're pregnant, correct result. Possibility
three: the test incorrectly identifies a non-
pregnant woman as being pregnant; this is a false positive, or a type one error.
Possibility four: the test incorrectly identifies a pregnant woman as "not
pregnant." This is a false negative, or a type two error. So which of these errors
is most serious? Well if your test says that a woman is pregnant when she's not,
this could cause a lot of unnecessary alarm, and a lot of your customers would
tweet about this on faceplant and pretty soon people would quit buying
your worthless lying pregnancy test and maybe you get sued and that's all bad. On
the other hand, if the pregnancy test tells a woman that she's not pregnant,
when in fact she is, she's more likely to just blame herself. Maybe she took the
test too early, and now a week later, the second test is saying that she is
pregnant after all, and none of the other twits on your DigDug account wall will
have a conniption fit, and they'll all think you took the test too soon. So it's
quite possible that the pregnancy test company would never find out about this
type 2 error. So remember that and we're going to come back to that in a minute. A
type 1 error occurs when we say that there is a genuine effect in the
population, when in fact there is not. This false positive type 1 error is
defined as rejecting a true null hypothesis. This means that you were
saying that a treatment has an effect when it does not. Type 1 errors are
serious, because you are falsely reporting a treatment effect that does
not exist. Research journals are greatly displeased with type 1 errors, because it
looks a lot like lying, and you can get yourself in a lot of trouble for lying
in journals. It can affect your career. That is bad news! So to remember what is
a type 1 error, think about Pinocchio. What happens when Pinocchio tells a lie?
His nose grows, and his long nose reminds you of a number
one; so lying saying that effect exists when it does not is a type 1 error. Peer
review of submitted journal articles and retesting hypotheses, especially
anomalous findings, are ways to guard against the publication of type 1 errors.
The probability of making a type 1 error is the Alpha level, typically p equals
.05, or five percent. To decrease the chance of a type 1 error, you could
change the Alpha level from .05 to .01. However, decreasing alpha also
increases the possibility of making a type 2 error. A type 2 error says that
there is no effect in the population, when in reality there is. A type 2 error
is defined as failing to reject a null hypothesis that is false. This means you
fail to detect an effect that actually exists. It's a false negative, but that's
not a big deal right? Wrong. Type 2 errors can be just as bad if not worse than
type 1 errors. Type 2 errors can be especially unfortunate in medical
testing. You are testing a cancer drug, and unknown to you, this drug is actually
effective on a very rare but deadly form of cancer. You test this drug on more
common cancers, but because your sample size is too limited you never pick up on
the small number of patients who are cured by this drug, so the results are
not statistically significant, and the drug never gets marketed, and then you or
somebody that you love gets this rare form of cancer. The drug that could cure
you is sitting right over there on the shelf, but you don't know it, because you
made a type 2 error. You missed it. So to remember a type 2 error think of the Old
English parochial school dunce cap. This is the type of school that Pink Floyd
sang about in "Another Brick in the Wall Part 2" . . . "We don't need no education, we
don't need no thought control. . ." that one. If you're making a lot of errors missing
questions, getting answers wrong, you sat in the corner with your dunce cap. So
think of the shape of the dunce cap, it's a cone, two lines, like a Roman numeral
two. You made a type two error you big dummy! You missed the effect so hopefully
Pinocchio and the dunce cap will help you remember type 1 and type 2 errors.