In this video we're going to discuss a little bit about permutation approaches
to hypothesis testing, we're also going to talk a little bit about when or why
we might prefer to use an approach like this! To do so we're going to work with
this simplified example here this is looking at the weights of chicks on one
of the two different diets and the weight after six weeks: so here we have just a
simplified dataset of only nine observations and we have the full dataset
that we'll look at later as well as we'll look at implementing all the
concepts we discussed in this video using the statistical software R! so I've
kept the example here simple so that we can see everything on the screen and
easily work through the concepts. So you'll recall that the basic elements of
a hypothesis test are always the same regardless of the approach: whether it's
parametric, nonparametric, resampling, or reshuffling type approaches the general
elements are always the same and so the elements we are looking at, the first
thing we want to do is specify a null and an alternative hypothesis, and an
important reminder that we work with assuming the null hypothesis to be true!
the second important element is that we need to choose a test statistic and
And again these in some way compare what do we observe in our sample of data to what
would we expect if the null hypothesis was true! in our example here
we're going to look at comparing two different test statistics; the third
element is we determined the distribution of the test statistic
again under the null: if the null hypothesis is true what's the
distribution of all the test statistics we could end up with and then the fourth
is that we convert our test statistic into a p-value;
again which generally
tells us what's the probability of observing what we did in our sample or
something even more extreme if the null hypothesis were true. Okay so here the
question we want to get it is do these two diets differ? is the weight gain
under one different than the other? okay and there's lots of ways we can try and
get at this and we need to be a bit more specific about what do we mean by differ
do we mean are we comparing the mean weights under the two diets or the
median weights or maybe we want to just compare the lower tail of the
distribution, say the 10th percentile, so at the lower end the really light
chickens, is the 10th percentile much higher for one feed type than the
other? right so maybe we want to just make sure are we pulling up that lower
tail of the distribution for one diet compared to the other.
and so comparing
the mean under these two diets we can use large sample approaches like the two
sample t-test which has been explored, we can do nonparametric approaches like the
wilcoxon rank-sum also known as theMann–Whitney U test and this compares the
medians under the two by using other approaches like a permutation approach
or bootstrap approach which is a related but different approach. These
allow us to test things more specific other than just comparing means or
medians. So let's talk a little bit about why we might choose to use a permutation approach:
if our sample size is small and we can't use these large sample
approaches, that might be one reason for trying a permutation test, if the
assumptions are not met as noted if we want to test something a bit more
specific other than comparing means or medians so
if our test statistics say is comparing the the 10th percentile of these two feed-types okay
or maybe want to compare the range of weights for these two feed types so
these other approaches allow us to be a bit more specific in the hypothesis we
want to test other reasons if the test statistic that we're going to work with
is difficult to estimate the standard error for, using approaches like
this are going to help us work around that! So in our example we're going to
compare the weight gain under these two different feed types. So our null
hypothesis stating it a bit generally is going to be that's the
weight gain is the same under both diets
both diets or both feed types. For test statistics we're going to consider two
different ones just so we can look at comparing absolute difference in means
as well as comparing absolute difference in medians so the first test statistic
we're looking at and I'm just going to abbreviate it TS so that we can save
ourselves some writing room the screen doesn't get too crowded so test
statistic number one we're going to look at the absolute difference in the mean
weight under casein minus the mean weight under meat meal and the second
test statistic will also consider I can abbreviate TS number two this is going
to be the absolute difference in the median under casein minus the median
under meat meal and again I'm throwing a hat on top here to represent that they are
sample estimates. If we would work this out for our observed data here we'd find
that test statistic number one it's going to be the absolute difference in
means: the mean for a casein:the mean of these four here it's going to come
out to be 349.25 and the mean for meat meal comes out to be 316, the absolute
difference is thirty 33.25 and our second test statistic the
median for the first group is 373.5 and the median for the second is 315
the absolute difference 58.5! okay so if our null hypothesis is true if the
weight under these two feed types is the same we'd expect these test statistics
to come out to be roughly zero right the mean should be the same the medians
should be about the same. we know they're not going to be exactly the same right
it's a sample they're gonna drift somewhere hopefully close to zero if the
null is true! so we want to know how much evidence do these estimates provide
against that null or in other words what's the probability of getting test
statistics this large or larger if the null is true and we'd expect these to
come out to be roughly zero so we're going to build up to getting to this
p-value using a permutation testing type approach. okay so to do this remember we
start by assuming our null hypothesis is true okay and so essentially what we're
assuming is that these nine observations here are equally likely to belong to the
meet meal or the casein group right that weight is independent of the feed type
or that we can state it this way that these labels are in some sense
irrelevant okay this observation of 260 is just as likely to belong to casein as
it is to meat meal. Right the weights and the feed types are
independent so the way we're going to try and get at building up the
distribution of test statistics and getting to our p-value is by considering
all possible permutations of this data okay what we mean by that is as stated
if these observations are equally likely to belong to either feed type right if
they're independent of feed type what we can do is consider all the
possible orderings of these observations so what are all the different orderings
of these observations each of those will be a unique permutation data set for
each of those orderings we can work out the test statistic and then we'll have
what are all the possible test statistics we could end up with if our
null hypothesis was true okay if weight gain is independent of feed type.
Now in reality there's going to be way too many
permutations to actually work out so in just this simple example with nine
observations if we want to consider all the possible permutations the one that
we put in the first slot can be any of the nine observations okay then the next
observation can be any of the remaining eight then the next can be any of the
remaining seven and so on all the way down to one, Okay, nine factorial and
that's going to come out to be 362,880 okay in other words there is about 360,000
different possible orderings of these nine observations, now you can
imagine we get to a dataset with 50 observations or a hundred or a thousand
okay considering all possible permutations isn't going to be possible
even with high computing power these things explode really quickly so what
we're going to do is we're going to take a simulation okay or a sample of all the
possible permutations so in other words we're going to take these observations
we're going to shuffle them up okay to a random order that's gonna be our
permutation dataset, calculate our test statistic do that again shuffle these
around in a different order and repeat that over and over again to get a bunch
of different possible permutations
we can do that by either shuffling these
observations here or you can shuffle the labels around okay I'm going to do it
shuffling these nine observations as I personally find it a bit clearer to look
or think that way but either way is going to get the same result it's going
to be the exact same conceptually; so let's get to creating our first permutation dataset
so I'm gonna label that permutation
number one here our first permutation data set so we can think of: out of these
nine we're going to randomly select one to be our first observation suppose it
comes out to be 303, now we're sampling without replacement
meaning that observation does not go back in the pool right we want to look
at all the possible orderings of these the second observation I randomly select
suppose it comes out to be 379 then select the third 260 the next is 380
325, 390, 368, 315 and 257 right so this is one of the possible reorderings of these
nine observations now for this permutation data set let's go through
and calculate the test statistic number one and test statistic number two so
first I'll put down T.S1 and I'm just going to put a subscript P1
here to indicate this is from our first permutation data set. If we
calculate the mean of the casein group it's going to come up to 332.5 and the
mean of the meat meal 329.4 and the absolute difference 3.1; if we
calculate this second test statistic and again I'm going to put a like a
superscript here P 1 to indicate it's from the first permutation the median
for the first group is 341.5 the median for the second group is 325 and
the absolute difference is 16.5; right so again these are giving us
an idea if our null hypothesis is true if these observations are independent of
feed type or equally likely to belong to either feed type what type of test
statistics can show up, we'd expect it to be 0 and these
give us some indication of how far they might drift from zero if our null
hypothesis is true; now let's go through and just create a second permutation
data set so I'm going to label our second one permutation number two okay
and again let's consider a possible reordering of these suppose our first
one comes out to 390 second one is 325 then 303, 380, 260, 379, 257, 315 and 360 all
right so again this is another possible reordering of these nine observations; if
we go through and calculate the test statistics for these and again I'm going
to call it test statistic number one but I'm going to put p2 here to indicate
it's come from our second permutation data set we calculate the mean for the
first group its 329.75 and the mean for the second group
331.6 and the absolute difference 1.85; we look
at test statistic number two the median for the first group comes out to
341.5 the median for the second group comes up to 325 and the
and the absolute difference is 16.5 okay so again this gives us a
second idea of what our possible values the test statistics could take on if the
null hypothesis were true so what we're going to do is we're going to go through
and we're going to repeat this P times P permutations so permutation number
P in reality there's no limit to the number of times or number of
permutations you can take; the suggestion or the rough guideline is at least
10,000 or more the only real limitation is computing power and how long it takes
to run through these things an important note is that increasing
the number of permutations you run through does not increase the amount of
information in the data okay we only have nine observations having more
permutations doesn't create more information, then what we are going to do
is determine the distribution of the test statistic that's going to be looking at
the distribution of all these test statistics, permutation test statistics
we have here and if we looked at it say histogram of all the test statistic
number one from our permutation sets and this is going to give us our estimate of
the sampling distribution right under the null; then what we go through is work
our way to a p-value and what the p-value is we're going to look at the
number of times the permutation test statistic was greater than the observed
test statistic, all that divided by P: the number of permutations we've run through
okay so in this kind of just simple demonstration here I'm running through
two permutations if we look at test statistic number one we observed a test
statistic of 33.25 in our permutation test statistics we
ended up with 3.1 and 1.85 okay so we actually
didn't end up with any permutation test statistics larger than our observed test
statistic okay so in this case of 2 our p-value here would come out to be
zero okay we're gonna actually run through more than two permutations but
this is the concept of how we get at that p-value right. so a reminder a
p-value is trying to get the idea of what's the probability of getting what
we got in our sample right what we observe in our sample or something even
more extreme if the null is true, if the null is true and we'd expect something
like this it's worth noting one kind of drawback or one limitation of a
permutation type approach is that we actually cannot build a confidence
interval through this approach, it only allows us to test hypotheses so what
we're going to get into is looking at the full data set and implementing this
concept running through a permutation test for test statistic one and
statistic number two using the statistical software R!
For more videos subscribe to marinstatslectures!