Hello. This is part 2 of a series of video on independent competent analysis applied to EEG time series.
This is the second part of "How does Infomax ICA work?"
This video is not so technically challenging here,
I'm just gonna give you some intuition of how ICA works and not delve into the mathematics.
And so here is a comparison between PCA and ICA.
So on here we have PCA. So if we have two variables variable one, variable two -- that could be
electrode one, electrode two for example -- and then here each of the dots
on the diagram could be some time points -- they are not linked by time here
but we can imagine this would be time one, time two, time three, time four, etc.
And then what PCA is going to find the axis along which the projection
of the data on this axis has maximum variance -- so the first axis is going to
be along this direction and the second axis is going to be perpendicular to the first axis.
ICA by contrast is going to find the projection of the data points that
maximize independence -- so it's going to find the axis along which the data is maximally independent.
So ICA is a method that tends to recover the original sources.
Here we can imagine that there are two processes that
influence the data, one along this direction, one along this direction, and
by multiplying by the unmixing matrix we can recover the activity of these sources.
So while PCA has simply decorrelates the output, ICA attempts to make the output
as statistically independent as possible while placing no constraints on the orthogonality of the matrix.
Here is how ICA, in particular Infomax ICA,
works so if we have some time series, some EEG time series here -- so this is the
time series and we can imagine this is a brain source, so this would be like
putting an electrode inside the brain and being able to record the activity of
that source across time -- what we can do with this source is that
we can artificially put a grid on it and count number of values in each of these
rows here and build an histogram. So this is the histogram of values and if we
have a smooth version with very small bin this is called the probability density distribution.
So far for that's for the first source, now imagine we have
a second electrode for a second source, but what we record on the scalp is really a
mixture of these two sources -- assuming there's no other source active in the brain.
And so we record the first scalp channel and a second scalp channel.
So the scalp channels are by definition a mixture of source A and source B, and
in this case, for electroencephalography, to the best approximation this is a linear mixture --
so this is perfectly adapted for ICA.
And what the central limit theorem tells us
is that the mixture of this source -- if you take an infinite number of these
source A, B, C, D, etc. and these are independent of each other -- the mixture
is going to converge to a Gaussian distribution.
So what we can do, what ICA can do, is that it starts from the scalp channels and
then it is going to rotate axes in a multi-dimensional space so it has to make the
projection as non Gaussian as possible in order to recover the sources.
So this is better illustrated in this diagram here, where we start -- this is
not very realistic for EEG sources -- but here we have two axes and two variables
and the first step is to normalize on each axis, so that's a process that's called whitening.
And this is the projection of the data on the two axes -- this is the axis for the data -- and then
what ICA is going to do it is going to rotate these axes so it has to make the
projection on this axis as independent as possible, and to rotate the axes
what it's going to do, it is going to maximize the entropy.
So why maximize the entropy?
So this is the calculation of the entropy -- this is, for example, a dice --
so the dice can take a value 1 to 6 and it's uniform, the probability of each face is 1 over 6,
and we use Shannon formulas for calculating entropy and that gives you an entropy of 2.58.
Now imagine you have a fake dice that makes the six half the time,
then it's less uniform and the entropy is going to be lower.
So the more uniform, the more random, and the higher the entropy.
So that's what we're going to try to maximize in that case.
And we're not going to simply maximize the entropy on each axis we're going
also going to maximize the joint entropy because the joint entropy is related to
mutual information as you'll see in a few minutes.
So that's Shannon formula -- how the joint entropy is the entropy of the first process
plus the entropy of the second process minus the mutual information between the two.
And we will see why it is important.
So how do you calculate the joint entropy?
Well it's very similar to calculating the entropy.
So for instance if we have a table here, with our behavioral measures -- it's just
asking people how stressed they are and how many emotional events they had in a given day --
and then we can see this contingency table where we have 19
people in this cell, 4 people in this cell, 63 people in this cell, etc.
What we can do is that we can calculate the probability for each of these cells as it
is represented here and then we use Shannon definition of joint entropy to
calculate the sum of probability times the log of a probability of each cell to get a specific number.
So how can we relate this to ICA?
So what about the ICA learning rule, how to make output statistically independent?
Well the goal could be to minimize their redundancy or mutual information.
So how do we do that? Well if we look at the formula we had before
where the joint entropy is equal to the entropy of the first output plus
the entropy of the second output minus the mutual information, really if we want
to minimize the mutual information, we can maximize the entropy.
And that's exactly how Infomax ICA works.
So the mutual information is zero if the two variables are independent.
So the learning rule is to change the weight matrix -- so to changed rotation on the axes --
in order to maximize the entropy and find an entropy extremum.
And then there is another term in the
Infomax learning rule which is called "natural gradient" of Amari,
and that's a rescaling factor that helps the weight matrix to converge faster.
So this was just to give you an intuition of how ICA works, and in the next videos
we'll see how it can be applied to EEG data.