Hello, my name is Arnaud Delorme, and this is a series of videos on robust statistics
applied to EEG data. This is part 5, which is about correction for multiple comparisons.
What is the problem of multiple comparisons? Well, when you calculate time-frequency decompositions
in EEG, you have a time by frequencies array of values, and you need to run
statistics for every single value or pixel in the time-frequency image as shown here.
So, the standard approach is to calculate statistics for each pixel one by one
as if the pixels were independent. Then you obtain a significance mask,
which you may use to mask the value which are not significant. The problem
with this approach is that with a standard 0.05 statistical threshold,
by chance, 5 percent of the pixel will end up being significant, even though there might be
no difference between the conditions you are comparing. How can we solve that problem?
To solve this problem, we need to correct for multiple comparisons.
I am going to show you several methods commonly used in EEG research. First,
we have the well-known Bonferroni correction, which consists of dividing the p-value threshold
by the number of comparisons. So if you are performing 100 comparisons and have a p-value
threshold at 0.05, you end up with a corrected p-value threshold of 0.0005.
Then you have the Holm-Bonferroni correction, False Discovery Rate,
the Max method, and two clusters methods, and I am going to detail how each of these works.
Mr Delorme, Mr Delorme
Yes, me from the past. Let's use Bonferroni. It is so simple.
Why are we even bothering with other methods? Can we use Bonferroni and end the class now?
Me from the past, I find your lack of interest in learning disturbing.
The main problem is that Bonferroni is overly conservative,
so often, we observe no significant effect while they might be present. Let's pursue.
First, when we talk about performing multiple comparisons, it is essential
to speak of the family-wise error rate. The Family-wise error rate
is the probability of making at least ONE error when performing all of these statistical tests.
So when doing a single test with a p-value threshold at 0.05, the family-wise error rate
is still 5%. However, when performing two tests, you have 5% of making an error for
the first test and 5% for the second test, leading to a compounded error of about 10%.
When performing 20 tests, then the likelihood of observing at least one false positive is 64%.
Why is not linear, with 10 tests leading to a 100% likelihood of observing a false positive? This is
because the probability of observing one error is one minus the probability of observing none. I put
the formula below if you wish to ponder on that. Bonferroni is one way to control the family-wise
error rate. At a statistical threshold of 0.05,
when using Bonferroni, we always have less than 5% chance of observing one false positive.
So let's look at other tests. Let's say we have ten p-values and that we want to correct for
multiple comparisons. Holm-Bonferroni is almost the same as Bonferroni,
but once we have corrected the first p-value with Bonferroni, we can correct the second p-value with
a little less aggressive threshold. First, I will order p-values from lowest to highest,
so most significant to least significant, as shown here. So if we have a p-value threshold at 0.05,
without doing any corrections, these would be the significant values.
Now, let's add a second column containing a gradient calculated using an inverse function
from the Bonferroni corrected value to the uncorrected value.
We will see in a second why we use an inverse function. So it goes from 0.005 to 0.05.
Then in the third column, we subtract column 1 from column 2. We scan p-values from lowest
to highest. Whenever the value is negative, we consider it significant according to the
Holm-Bonferroni correction. The first p-value is corrected against Bonferroni.
The second value is corrected against Bonferroni as if there was one less value, etc... so that
explains why we use the inverse function. So in this case, we have two significant values.
We can also obtain a corrected p-value threshold. We can look for the last row for which the value
in the 3rd column is negative.The corrected threshold is 0.05 divided by 10 for Bonferroni
and 0.05 divided by 9 for Holm-Bonferroni which is a little less conservative.
Both Holm-Bonferroni and Bonferroni correction accurately control the family-wise error rate.
Now, what about False Discovery Rate. False Discovery Rate is a different concept,
where we control the number of false positives. When we set the false discovery rate to 0.05,
we know we can expect about 0.05% of false positives,
which correspond 40% chance of observing at least one false positive. So we are not controlling the
family-wise error rate anymore. Instead, we are controlling the percentage of false positives.
And we can set our FDR threshold to an arbitrary value. We just need to
remember that this correspond to the number of significant values that might be false positives.
So how does FDR work? We calculate FDR in a similar way as Holm-Bonferroni. However,
while we used an inverse gradient of statistical thresholds for Holm,
for FDR we use a linear gradient between the Bonferroni corrected and the uncorrected p-value.
Then we subtract column 1 from column 2 and we look for negative values. As for Holm-Bonferroni
and Bonferroni, we can also obtain a corrected threshold, which is the p-value corresponding
to the last negative value in column 3, so in this case 0.01. Any p-value less than 0.01 may
be considered significant after FDR correction. And you may notice in column 3 that the difference
is negative then goes positive and then negative again. After the difference becomes positive even
once, we just ignore the following differences - and this is the same in Holm-Bonferroni.
Again, it is essential to remember that at FDR 0.05, we may end up with 0.05% of false positives.
Is there any other methods? Well, let's look at a method called the MaxT
method or sometimes called the single-step maxT method, as opposed to multiple steps maxT methods
we will see in the following slides. T stands for statistics so we are looking at the maximum
statistics. Instead of building one distribution for each pixel, we build a single one for all
pixels. We take the maximum of the statistics across all pixels at each iteration and add it
to our null distribution. For example, it could be the maximum difference or maximum
t-value. To make it two-tailed, we calculate the absolute value before taking the maximum.
Then, to assess significance, we test where each original difference
lies in this distribution. For statistics that have similar distributions of values across pixels
or time-frequency decomposition, as this is often the case in EEG, this type of method controls
family-wise error rates pretty well. It means that if some values are significant,
you can expect them not to be false positive. Note that another related measure, called minP,
consists of computing the parametric p-value for each pixel and taking the min across all pixels.
It is more computational expensive because you have to calculate the p-value,
and it has slightly different properties. However, using both of these methods, there are
issues with EEG. For example, if the distribution of values differs across time and frequencies,
this method might ignore some effects at some frequencies.
We can fix this using the methods we will discuss next.
The next method is the cluster method which is a multiple step maxT method. In this method,
we use parametric statistics at a given p-value threshold, for example, 0.05.
We bootstrap the conditions, and we look at all the significant regions. We count the number of
contiguous pixels in each region, use the largest region, and add it to the null distribution.
To assess significance, we evaluate if significant regions' areas
lie in the bootstrap null distribution's upper tail. In this case, this is only true for the
red area. It is that simple. Note that we are using the region's size, but we could use
other summary statistics, such as the sum of the T-value in that region. Note that it is
the same as the maxT method, except we compute a different statistics across all pixels. In
the previous slide, we take the maximum statistics across all pixels. Here, instead we calculate the
maximum area of significance at a given p-value threshold, which is another type of statistics.
And, since you can use any statistics with the maxT method, there are several variants,
and we will see another at the end. Because we end up building a single
distribution, this method controls well for family-wise error rate.
Here is another example where we have 100 noisy trials and are interested in identifying
significant regions in the mean. If we do not correct for multiple comparisons, a large number
of pixels are significant. When we apply the cluster method, only two regions are significant.
Also, an advantage of the cluster method when applied to EEG,
is that you can not only use it across times and frequencies but also across channels. To correct
across channels, you need to define a neighboring channel matrix - which channels are close to each
other. Then one may search for clusters in space defined by the channel neighboring matrix. In the
channel by channel matrix shown here, I have ones shown as white when two channels are close to each
other and 0 shown as black otherwise. So when we compute clusters, it does not have to be in one,
two or three dimensional standard space, what we call Euclidean space. Note that it
is also possible to find clusters in a mixture of Euclidean space and space defined by the channel
neighboring matrix, for example when assessing significance across time-frequency and channels.
And of course, you can also do that in one dimension: when performing cluster correction,
you look for the largest region of adjacent p-values. Here is an example, where we compare the
uncorrected, the Bonferroni corrected, and the cluster corrected solution. The cluster corrected
region of significance is similar to Bonferroni in this case, except larger. Often though, there
are no regions of significance with Bonferroni, while there might be with the cluster method.
One issue with this method is that sometimes, we may have large clusters
below our statistical threshold, and they are not considered significant. For example,
here we have two clusters, a positive cluster in blue and a negative cluster in green.
The blue cluster is above the t-value threshold and ends up being significant
but not the green one. So is there a method not constrained by the arbitrary threshold?
So let’s look at the threshold-free cluster enhancement method,
which is yet another type of MaxT method. Here is a one-dimensional example. In this method, we will
not use the raw statistics as in the maxT method but instead use the values enhanced
by the level of significance and the extent of the region they belong to. At a given time,
here represented by the vertical red line, we scan different statistical thresholds,
here represented in different colors, and we calculate regions of significance for that
threshold. Then we calculate the product of the statistical threshold and extent
of the region of significance. We do that for multiple thresholds and we sum these
products across all thresholds. So let’s look at an animation at two different times
At the first latency, we have all these regions which are significant,
and we sum across them and obtain a TFCE value of 9. For this second latency,
we have one more region of significance, the purple one, so the TFCE value is larger at 10.4.
In this slide, we have the original T-values and the cluster enhanced T-values.
Then, we can use the maxT method on these values. Instead of using the raw statistics
as in the standard maxT approach I showed you earlier, we used the cluster enhanced statistics.
So which statistic is better. Here is a summary slide. Also, I am not including
Bonferroni because Holm-Bonferroni is superior without loss of generalization.
Bonferroni-Holms is fast, but it too strong of a correction,
especially when samples are not independent. FDR is fast to calculate, and the interpretation
is clear: at FDR 0.05, you should expect 0.05% of false positives. However, it does
not control for family-wise error rate. The cluster method uses prior knowledge,
such as the size of the cluster. It also appropriately controls for family-wise error
rate. However, one must be careful interpreting results, as one may only draw conclusions about
the clusters and not the original values. It is also computationally expensive.
Finally, the threshold-free cluster enhancement technique
has similar advantages to the cluster method without having to choose a statistical threshold.
It takes even longer to calculate, though. If I had to choose one, I would choose this last
technique, and I illustrate how to use it on EEG in one of the videos using the LIMO plugin.
In this slide, I include a list of references if you want to dig deeper.
So this is the end of this presentation. Thank you
for your attention, and I hope to see you in one of my next videos.