We are going to discuss a little bit about the idea of multiple comparisons
Corrections or multiple testing Corrections. So we're going to do this in
the context of one-way analysis of variance although the idea of multiple
testing correction applies to more than just one-way analysis of variance (ANOVA); so the
general idea of this correction is that when we do more than one test or more
than one comparison, our type 1 error rate starts to increase so for example
you'll recall that the Alpha we use for our test is our probability of making a
type 1 error, our false positive; if we do one test here with an alpha of 5%
there's a 5% chance of making a type 1 error if we do a second test with an
alpha of 5% that test also has a 5% chance of making a type 1 error. Combined
over these two tests there's going to be a greater than 5% chance of making at
least one type 1 error. So this is the concept that we want to correct for. The
more tests that we do simultaneously the greater chance there is for making a
type 1 error so we're going to learn a bit about how to control for that and
we'll get to that in a moment; so you'll recall we worked this example
comparing weight loss on four different diets comparing the mean weight loss
over diet A,B,C and D with an alternative hypothesis at least one of the means
differs: at least 1 diet has a mean weight loss that's different than the
rest. To do so we ran through a one-way analysis of variance (ANOVA), we calculated this
F statistic which was a ratio of variability and weight loss explained by
diet to variability and weight loss that's not explained by diet or the
variability that's happening between groups, variability within a group; and we
had a test statistic of 6.1 and resulting p-value point 0.0011 one and
then this led us to reject our null hypothesis or conclude we have evidence
to believe at least one mean differs from the rest; so now we need to decide
which one or which ones may differ so in order to do this what we're going to do
is we're going to do all pairwise
comparisons. Okay, and what I mean by that is we're going to compare diets A and
B, diets A and C, diets A and D, B and C, B and D and C and D; okay so we're to
compare all possible pairs of means. Mathematically we can think of this we
have four different groups and we're going to choose two of them to compare
how many different combinations of two groups can we pick from these four. You
might recall this ends up being 4 factorial over 2 factorial times 2
factorial which equals 6 okay 6 possible pairwise comparisons and
in order to do this what we're going to do is we're going to use our independent
two sample t-test type approach so we can do a t-test or a confidence interval
comparing each of the two pairs AB, AC, AD and so on. So we can either look at the
difference between Group one and group two plus or minus a t-value and again
this has some degrees of freedom, some confidence level times the standard
error for the difference in means or we can do the hypothesis test and
calculate a test statistic that's going to help tell us how far is the
difference in means we observed from the hypothesized value in terms of a
standard error. And as noted so here we're going to do six different pairwise
comparisons of the 6 tests and we've noted that as the number of tests we do
increases the chance of making a type 1 error increases as well! so we're going
to start to work on the idea of how often will a type 1 error happen and how
can we try and reduce that rate. So to work our way through this we're going to
assume that each of these pairwise comparisons is independent of the others;
that may not necessarily be a fully realistic assumption but:
first it's going to simplify some of the calculations so we can focus on
the concepts and it's also a little bit more conservative. so let's start by
thinking about what happens with each test: for each test and by that I mean
each of these individual comparisons that we're going to do if we use say an
alpha of 0.05 or 5% then essentially what we're saying there is
the probability of making a type 1 error on each test is 5% or we can think of it
for each of these comparisons each confidence interval we're going to use
confidence of 95% and so again if each of these tests has a 5% chance of making
a type 1 error the probability of not making a type 1 error on each of the
tests is 95%! all right 5% chance of a type 1 error
95% chance of not making a type 1 error now if we think over all of the tests so
over all the tests or all the comparisons the probability of making at
least one type 1 error
right again probably making a type 1 error here or here or here or here or
here or here is the probability of making at least one type 1 error we can write as 1
minus the probability of making no type 1 errors
and again the probability of making no type 1 errors we can write this as 1
minus the probability that we do not make a type 1 on the test comparing A
and B, times the probability that we don't make
a type 1 error on the test comparing A and C
all the way up to the last one, probability of No type one error on that final
test comparing C and D. But again we can multiply the individually here because
we assume each of these comparisons are independent alright so it simplified the
calculation for us a little bit now
it's a probability of not making a type 1 error on any of the given tests is 95
percent so 95 percent times 95 percent times 95 percent and you're going to see
we have that appearing six times so if you work this out it's going to
come out to be 0.265 or 26.5% so again remember what we worked out there the
probability of making at least one type one error over all six of these tests is
about 26.5 percent. Okay so you can see how our type one error
rate has inflated a lot with doing six comparisons: multiple
comparisons! so sometimes this gets called the familywize error rate or
other names like that so what we'd like to do is learn how to control this type
one error rate so it doesn't inflate to an extremely large value. there's lots of
different possible Corrections that we can do: what we're going to talk about is
one called Bonferroni's multiple testing correction and we're gonna do
this for a few reasons: the first reason is that it's the
simplest to teach and understand. All the other possible Corrections are the exact
same in concept with slight changes in the mechanics; okay so once we understand
Bonferroni's approach we can understand that the other ones are pretty similar
with some minor changes in there:
so Bonferroni's approach is to
use an adjusted alpha star again here we want to use the overall type 1 error
rate that we want divided by the number of comparisons;
here 0.05 and we're
new six different comparisons; so it's use an adjusted alpha of 0.00833
okay for each of the individual tests or we're looking at confidence right use
99.167 percent confidence for each of the individual confidence intervals
Let's see what that's going to do to the overall type 1 error rate
so again if we use this adjusted alpha that tells us the probability of a type
1 error on each of the individual tests is 0.00833 or 0.8%
or we're going to use 99.16% confidence over all tests.
it's again a 99.167%
chance of not making a type 1 error on each of the individual
tests so overall the tests it's a probability
of making at least one type one error because one minus the probability of
making no type 1 errors and here we're using 0.99167
and if you work that out you're gonna find that it comes out to roughly 0.049
or 4.9 percent. okay so Bonferroni's correction suggest use an alpha of
0.00833 for each of the individual tests or use 99.167%
confidence for each of the individual conference
intervals to have an overall type one error rate of about five percent; so the
probability of making at least one type one error over all six of these
comparisons is roughly five percent. So if we were to run through and do all the
possible pairwise comparisons building confidence intervals, we can see the common
rules for all six groups here comparing Group A to B, A to C, A to D and so on
So we don't want to focus our effort on how to plug into this formula to run through
these calculations we can easily have a piece of software do them for us what we
want to do is focus on the concepts and the interpretations. So taking a look at
these confidence intervals we can see that there's only two of them that do
not contain zero: we can see comparing Group A to Group C, the confidence interval
does not contain zero which gives us an indication there's a statistically
significant difference between Group A and C or we're not willing to accept
the difference is zero; we can also see group B and C are significantly
different. Other than that no other significant differences show up! I'm
gonna draw a little diagram here that helps us think about the conclusions
we're reaching here: so here we're saying we're confident that group C is
significantly greater than groups A or B: C and A are significantly different and
we can see that the mean for A is larger than the mean for A; C and B are
significantly different and again the mean for C is significantly larger than
it is for B, and I'm going to draw D here in the middle signifying we're not
convinced that C and D are different, we're not convinced that D and A are
different! so I want to spend a moment here now
just talking about some difficulty some people may be having with this
conclusion here I know when I first learned this material I struggled a
little bit with this concept; so some of you your brain may be going in the
direction if C is the same as D and D is the same as A, isn't C the same as A? ok
and that's true mathematically right if C equals D, D equals A, C must equal A; ok
but we're not saying that here what we're saying is we're not convinced C
and D are different! that's not the same as saying that they're the same
we're saying we're not sure they're different
again we're not convinced A and D are different! we are convinced C and A are
different. Now to get at this idea I also like to use an example that I think is a
an intuitive way to get at this: so suppose we all go out and do a
10 kilometre run, so we decide as a class let's go and get some exercise, run 10
kilometres, time ourselves and see who wins: suppose that the person who finished
first did it in 39 minutes and our second-place runner did it in 40 minutes
now without knowing too much about variability in the race times, just trying to
think intuitively you know I'm not convinced that the person who finished
in 39 minutes really is faster than the person who finished in 40. Right? if
they ran the race on a different day under different conditions
maybe they'd switch places only one minute separated them maybe that was a
chance variation or chance difference. Now let's suppose that the
third-place runner did it in 42 minutes okay again I'm not really convinced the
person that did in 40 minutes and the person did in 42 are significantly different
right, again they run on different days maybe they'd switch places only two
minutes separated them maybe that was a chance difference
let's continue on suppose the next person did it 43 minutes and again we're
not really convinced these two are different let's suppose the next one is
45 minutes then 49 minutes 51 minutes and so on now I'll tell you that I am
convinced that the person who did it in 51 minutes is significantly slower than
the person who did in 39 minutes okay and that gap is big enough that I think
that difference is real; any two days these to run I believe
this person will always be faster than that one! okay so that's what we're
getting at here that the distance between each of these might not be large
enough to be convinced that it's real; the gap between these two is big enough
that we think is not due to chance and that's similar to the conclusion we're
reaching here: okay the difference we saw between C and D was not big enough for
us to be convinced that statistically C really is larger than D the distance we
saw between the mean of D and the mean of A wasn't large enough to be convinced
that we think they're really significantly different but the
difference that we saw between Group C and Group A was large enough to believe that
we think statistically these are significantly different from one another.
Now I just want to end on some important reminders here: the first is the idea of
statistical significance: statistical significance versus clinical or
scientific significance; okay so we've talked a bit about this through different
videos but just a reminder that because something is statistically significant
doesn't necessarily mean that it's meaningful in a real world right so the
difference in mean weight loss between these dies while statistically
significant deciding if it's you know scientifically meaningful there's a
question that requires context looking at the actual effect size and the
numeric difference between the two not just if it's statistically significant;
another reminder is the trade-off between the type 1 error rate and the
type 2 error rate: when we increase one we decrease the other so just to
remind you here that in controlling this type 1 error rate and using a
lower alpha to get a lower familywize type 1 error rate we're increasing the
type 2 error rate. Or we're trying to make fewer false
positives at the expense of we're gonna make more false negatives! so just a
reminder that there is that trade-off there; another important reminder this
stuff generally we're going to do through the calculations using software
so we don't worry we want to focus our attention on the formulas or how do we
plug things in we show the formulas that we can understand the concepts and what
it is a piece of software is doing for us but a reminder we don't want to put
our focus on you know what exactly is the t value we should be using and how
do we find that in a table. right we can do all that with software that's not the
important skill to get through this stuff and the final reminder there's
many different methods of doing multiple testing Corrections we talked about
Bonferroni's; these other approaches are all the exact same: similar in concept
some of the mechanics may change slightly!
thanks for watching and make sure to subscribe to marinstatslectures youtube channel!