Hello, my name is Arnaud Delorme, and this is a series of videos on robust statistics
applied to EEG data. This is part 4 which is about the difference between bootstrap
and permutation.
So we have already talked about bootstrap. When comparing 2 paired conditions, under
the null hypothesis, we assume that the difference is close to 0. So we can put all the pairwise
differences in a bucket and randomly draw from that bucket to build the difference distribution.
If 0 lies in the tail, then we can conclude that the two conditions are significantly
different.
What about permutation? For permutation, under the null hypothesis, again there should be
no difference between conditions A and B so we can randomly permute some of the values.
So here is one random permutation for example. For each pair, there is 50% chance of switching
condition. Then, we take the average of the pairwise difference. Note that this is equivalent
to randomly inverting the sign of some of the pairwise difference before taking the
mean. Then we can build our null distribution and we assess if the original un-permuted
mean difference lies in the tail of this null distribution or not. If it does, then we can
conclude that the two conditions are significantly different.
Let's quickly review that with a concrete example where we just want to determine if
husbands are older than their wifes. For boottrap, we put the age differences in a big bucket
and randomly draw 8 values then calculate the mean repetitively to create the mean difference
distribution. If 0 lies in the tail, then the difference is significant.
For permutation, again we put the difference values in a big bucket, but for each iteration,
we randomly permute the sign of some of the differences. We take the average of these
values, to build the null distribution. Then we conclude that there is a significant difference
if the original unpermuted difference lies in the tail of that null distribution.
So this what about unpaired statistics? The interesting thing with this example is that
we can perform either paired or unpaired statistical comparison. We have done the paired comparison
so now let's do the unpaired one. In this case, there is no pairing, and we just have
a group of men and a group of women. We want to assess if the men as a group are older
than women. To do so using bootstrap, we are going to
have one bucket for men and one bucket for women. Then we bootstrap the bucket for men
and get an bootstrap age for men, and we do the same for women. We subtract the two mean
age and we build our difference distribution. As in the unpaired case, if 0 lies in the
tail, we conclude that the two groups are significantly different.
What about permutation? For permutation, as for bootstrap, we have a bucket for men and
a bucket for women. We randomly permute the values between these two buckets. So we take
any value in the men bucket and swap it with a value in the women bucket. The difference
with the paired case is that the age of any man can be permuted with the age of any woman,
and we do not permute only within couple. We then take the mean from each bucket after
permutation and build our null distribution. If the original un-permuted mean difference
lies in the tail of this null distribution, we can conclude that there is a significant
difference.
So which one is better. Anyone want to bet? Mr Delorme, Mr Delorme
Yes, me from the past. This is an easy one. Permutation is better
because it is more intuitive. To test if A and B are different, you can just swap values
between them. Easy! Me, from the past. I am not surprised. You
are wrong.... again. Permutation might sound more intuitive, but bootstrap is more statistically
sound. In fact, in his book on robust statistics, Wilcox only has a few pages on permutation
while the rest of the book is about bootstrap.
So why is bootstrap more statistically sound. The assumption for using surrogate statistics
is that the current sample is the best estimate of our population. When you are drawing subjects
from your population, you want each draw to be independent. This is the case with bootstrap,
because you are drawing values with replacement. Each value we draw, we put it back in the
bucket so we might draw it again, and each draw is independent. This is not the case
for permutation, where we draw each value only once. So when using permutation, each
draw depends on previous ones. Imagine you are drawing subjects from the general population.
Picking a subject should not depend on the subjects you have picked before. And this
is one of the reasons bootstrap is more statistically sound than permutation.
In practice, on actual data, we cannot really see the difference, so it is more a conceptual
preference for bootstrap than a numerical one.
So this is the end of this presentation. I want to thank you for your attention, and
I hope to see you in one of my next videos.