Hello, my name is Arnaud Delorme, and this is a series of videos on robust statistics
applied to EEG data.
This is part 1 which is an introduction to inferential statistics.
In this presentation, there are a lot of statistics obviously.
I want to mention though that I am not a statistician.
My approach here is very practical, and focuses on inferential statistics – which about
figuring out if an effect is robust and significant.
Why do we do statistics by the way?
There are 2 key aspects to statistics.
First, once you find that great feature in your EEG data, you want to know if it is reliable
and reproducible.
Statistics can tell you the likelihood that this effect occurs by chance.
If you want to test if dogs can talk and you see a dog talking, then you do not need statistics.
However, statistics is especially important for EEG which is a noisy signal, and effects
can appear due to random fluctuation in that signal.
The second important part is modeling your data.
If you have an hypothesis that a specific treatment affects the EEG in a specific way,
you can test that hypothesis with statistics, and either accept or reject your model.
You also determines how well you want your model fits the data.
Also once you have a model, you can make prediction on data that has not been acquired yet.
This is the outline of this lecture.
First, we will talk about parametric and non-parametric statistics, which means basic statistics,
t-test, ANOVA, etc…
Then I will talk about what we can do to improve on that.
I will then describe the bootstrap and the permutation methods, which consist in building
surrogate distribution of values to perform inferential statistics.
Also, in EEG, we perform large numbers of comparisons – usually one for each time
sample, or each time/frequency point -- so it is important to correct for all these statistical
tests, something we call corrections for multiple comparisons.
I will talk about different methods for correcting for multiple comparisons commonly used for
EEG.
Finally, we will talk about group-level statistical analysis of EEG data in EEGLAB.
Before we start, I want to mention a few important take-home messages.
First, look at your data, and show your data.
Looking at the raw data will give you a general sense of the quality of the data.
Looking at your data from different angles, and using different methods, will ensure that
whatever great discovery you have made is not a fluke.
Second, a perfect universal statistical recipe does not exist.
Depending on the question you are interested in, you might want to use different methods.
There are workhorses of course that you will use often, but sometimes you need to use special
methods – for example, using circular statistics to look at differences in phases of the EEG
signal in the time-frequency domain.
Third, keep exploring: there are many great options, most of them available in free software
and toolboxes.
How do we do inferential statistics and draw conclusions about our data.
For example, say you want to compare the EEG of your participants from group A to participants
of group B. Mr Delorme, Mr Delorme
Yes, me from the past.
This is super simple.
You just put the values in a statistics software and see if the p-value is below 0.05 and you
are done.
No need to waste your time on statistics.
Humm, me from the past.
If I did not know you better, I would probably recommend that you give up doing science for
the sake of humankind.
This view is shared by a lot of people and one of the main reasons why we have reproducibility
problems in science.
If I rephrase what you say, your null hypothesis can be that there is no difference in the
EEG of your participants A and B. You perform EEG measurements and compare the actual difference
for the EEG between A and B with your null distribution – in general based on parametric
statistics.
If the measurement lies in the tail of the null distribution, so the upper and lower
area representing 5% of the total distribution area, then you can decide that it is significant.
And Yai!!
You are done.
So let's see why this simplistic approach is incomplete at best.
First, we also need to consider the alternate hypothesis.
The alternate hypothesis is that there is a difference between the two conditions.
There is also an error associated with incorrectly accepting the null hypothesis.
So if we look at the table below me.
If we reject the null hypothesis and it is true, this is a type 1 error, so it is the
standard alpha value we are all familiar with.
And if we fail to reject the null hypothesis and it is false – so in other words, if
we conclude there is no difference between group A and B but in fact there is one, this
is a type 2 error.
One may argue that this error is not as important as the one that concludes that there is a
difference when there is none, but if this error is too high, you will never be able
to conclude there is an effect while in fact, it is present.
Most importantly, you should not conclude that there is no effect unless your type II
error is low.
You might have head the quote “Absence of evidence is not evidence of absence”.
A derived concept is statistical power which is one minus the type II error.
The higher the statistical power, the less likely we are to accept the null hypothesis
when it is false.
Finally, standard practice recommends reporting the exact p-value and not only if it is above
or below 0.05 which is a threshold which was arbitrarily set in the 1920s.
To compute the exact p-value, we can simply assess the relative area in tail of the distribution
using our experimental value as threshold.
Also, one must report the effect size, which is an indication of how different the EEG
between A and B is.
In case, of two groups, it is simply the difference of the mean for A and B divided by pooled
standard deviation.
Understanding effect size is important.
These are standard effect sizes, with 0.2 representing a small effect size, 0.5 representing
a medium effect size and 0.8 representing a large effect size.
Why does it matter?
You can imagine that if you have a large number of participants, say 10,000s, you might be
able to observe significant effects for minute changes in the EEG.
These changes might be so small that they cannot be used for any practical purposes,
and are not even interesting.
This is why it is irritating to see in the news sometimes: Vitamin X taken daily increases
longevity, and if you were to look at fine print, you might be able to see that this
study was done on 100,000 people and that longevity statistically only increases by
3 days.
Really, who cares about a such a small effect, which is anyway possibly confounded by other
variables.
From my perspective, this is close to fake news.
This is why it is very important to report the amplitude of the effect.
The other advantage of considering effect size and statistical power is that it can
help you calculate your sample size.
These days it is difficult to get a grant accepted without at least a basic power calculation
to justify your sample size.
Once you know your desired alpha value, your desired statistical power, and expected effect
size, you can use standard tools to calculate sample size.
So here for example, I just found this website, where you enter the group 1 mean and standard
deviation.
So this could be your ERP amplitude for example.
Then based on the literature, you try to infer the minimum change you will want to be able
to observe, in this case let’s say a 10% increase for your treatment group.
You also enter your maximum alpha value and minimum statistical power, and the software
will return the minimum sample size.
There are also functions in Matlab and R to do that which I indicate here.
This solution is not perfect, because it uses parametric statistics assumption for your
data, so the measures can be biased, but it is a good start.
Note that for group analysis in neuroimaging studies, sample sizes usually range from 16
to 32.
If you have less participants, your study might be underpowered.
So we have done our due diligence, performed power calculation, and acquired some data.
Now we are ready to analyze it.
Why not use simple parametric statistics.
Based only on the mean and standard deviation of the data, we can calculate t-values.
Then we can compare our calculated t-value to the distribution of t-values under the
null hypothesis and assessed if our t-value lies in the tail and is therefore significant.
There are formulas for unpaired data – such as two groups of participants randomly sampled
in the population.
We also have a different formula for paired data, such as different conditions for the
same participants.
We can also calculate ANOVA based on the same principles when we have more than 2 groups
of participants or more than 3 conditions, except we will now deal with the F-distribution
instead of the t-distribution.
It is important to note that these statistics assume that the data can be reasonably well
fit to a Gaussian distribution.
This is implicit when we use the formulas here which assume that the mean and standard
deviation are accurate descriptors of the data.
Of course, if you use these statistics, you should also test if this assumption is valid
and your statistical software will usually do that automatically.
And there are other types of statistics based on the type of data and hypotheses you may
have.
There are statistics to compare discrete values, for example when you count the number of participants
on which you observe an effect or when you consider yes/no answers to questionnaires.
There are statistics for continuous data which are typically the ones we use for EEG.
There are also statistics for ranked data.
These are usually used to compared ordered data but can also be used on continuous values,
such as reaction time or even EEG.
For each type of data, you have different recipes if you want to compare groups of participants,
conditions, or perform regression analysis, etc....
So, this is just a summary slide, so you can pick the best standard statistics for your
usage model.
Another approach is to use surrogate statistics and we will talk about that in one of the
future lectures.
I also want to mention that for continuous measurements, the general linear model framework
can encompass all study designs.
I have made a video on this topic and put a link in the description.
So I want to thank you for your attention, and I will see you in the next video.