Okay Michael. So let's try to be a little bit
more detailed about, kind of, the mechanics of how you, how
you would make this work. The algorithms for finding Independent Components
Analysis, are in all the reading. But, I do think it's
pretty easy to kind of get lost. If you don't, sort
of, turn these matrices into actual numbers. So, let me just,
sort of, give you a quick example of how we would
do this specific thing here. And see if that helps, okay?
>> Mm-hm.
>> Okay. So, how would you go about turning
this into a problem that something like, you
know, an actual algorithm that uses numbers could
do. Well, it turns out it's fairly straightforward.
And let's just take this, this particular example
here, with people talking into a microphone. Basically
you create a matrix. Which are samples of
all of the sounds. So, in our original
space, we have, you know, this handsome person here.
We have this other, let's say, similarly handsome but in a
different way person here, and we have this other person, with
my poor attempt to draw hair over here, and they're talking.
Well, what we end up doing is we basically take the sound
wave and we sample it. And what does it mean to
sample it? Well, you know, you represent sound in a computer as
a sequence of numbers, just like you represent pictures as a
sequence of numbers, and in fact, you represent words as a sequence
of numbers. And so we basically turn these sound waves
into a matrix full of values. So, as I described
before, Michael, each one of these microphones. Is actually getting
a linear combination of speech from each of these three
individuals. So if we think of microphone one, it is
seeing some kind of sound wave, which is again a
linear combination from each of these people generating a sound
wave. The same is true for microphone two and similarly.
For microphone three. Now, of course, we're talking about recording
these and we're thinking about computer programs, which means we take
this continuous sound wave and we sample it at a very
fast rate. And what we end up with is a sequence
of numbers that represent the particular pressure that we're getting
in these sound waves. So, these sound waves get converted into
a sequence of numbers and I'm just going to make up some
numbers for this. And now what we have is a matrix
where each row represents a feature in our original feature space.
And by original here, I mean, the feature space that we
see each of our microphones and every column represents a sample. Say
at time0, time1, time2, and so on. Does that make sense?
>> Yes. And so, since each of these is a linear combination
of these voices we get here. Our goal is to find a projection
like we did before and the original definition of the
problem, such that if we projected this on to here, we
would end up with a new feature space that corresponds individually
to each of these people. And you could do this with
anything, it doesn't have to be sounds, but if we think
about the. Adhoc query problem that we went through earlier, each
of these would be words, and say their presence of words
would be your features, and each of these columns would be
documents, say. And the goal would be to find
a projection given a set of documents that allowed
us to recover some underlying structure that was useful
for classification, or for information retrieval. Does that make sense?
>> Yeah. And can you say, what does
it mean for that sequence to have mutual information?
>> so as you recall. Because I know that you listen to Push Cars.
A discussion about mutual information and entropy and so
forth. All mutual information is is a measure of
how much one variable tells you about another variable.
And, the assumption here is that each of these people
talking has no mutual information. That is, they're talking
independently of one another, or the sounds that are coming
out of their mouths, don't allow us to predict
the sounds that are going to come out of another
persons mouth. So it may mean that these people are
talking to one another as we often do, Charles might
be talking to Michael, who's talking to push cars, who's
talking back to Michael, who's talking. To Charles, but the exact
sound waves that we see are actually independent of one
another. So if that turns out to be true, what independent
component analysis is trying to do is trying to recover
features, in this case. It turns out the corresponding individual speakers,
such that their sound waves, or the values
that you see, are statistically independent of one another.
Okay? And that's what this does. But, since
we can always come up with arbitrary projections that
are statistically independent, it's important that whatever our
new transformation gives us. It actually has some relationship
with the original values that we saw. So some
how we want to be able to learn from this,
into this in a way that we don't lose any
information. In much the same way that PCA was trying to
minimize the loss of information, and so we not only want
each of the individual new features, in this case people to
be statistically independent, we want the amount of data that we
get from these people. Compared to the amount of data that
we got originally from the microphones, to actually strongly predict one
another. And if the mutual information between the microphones and our
candidates for the people's voices have high mutual information, then
it means that we haven't lost anything. And so those
two things together, those two constraints. Forces into a situation
where, if this model is true, we construct the original voices.