>> Okay, so, let me give you just a few more
examples, Michael, of a, how PCA and ICA differ. And I'm going to
do this mainly by talking about certain things that ICA does. And
this is really just for your edification, but I think it really
helps, to think about, those sort of underlying models and how
they differ, by thinking about how they react differently to different kinds
of data. So, just, here's a couple of examples. The one we
covered right away, was the blind source separation problem. And of course,
what, what we recall is that ICA was, in
some sense, designed to solve the blind source separation
problem and in fact ICA does an excellent job
at solving the blind source separation problem. Meanwhile, PCA
does a terrible job at solving the blind source
separation problem and that's just because it's assuming this
kind of Gaussian distribution, it just does not recover
what's going on. But here's something that ICA does
differently from PCA. Is because it's got this kind of blind source root in it,
it's actually directional. And what I mean by
that is, you recall I drew this sort
of matrix before, where we had features this
way. And we had samples of those features
like sound, time samples of sound this way.
It turns out that for PCA, it doesn't
matter whether I give you this matrix or I give you
the transpose of this matrix. It ends up finding the same answer.
>> Hm.
>> And that should make sense, right?
Because it's just basically finding a new rotation
of the data, and these are effectively
just rotations of each other. For the purposes
of, of, if you just kind of think about it geometrically in space. ICA on the
other hand, gives you completely different answers, if
you give it this versus giving it this.
So ICA is highly directional. And PCA much less
so. But ICA, because it's making this kind of
fundamental assumption, does end up generating some really cool
results. I'm going to give you another example of
one. That's like a blind source separation. In fact,
I'm going to give you two. Okay? So, imagine
you had a bunch of inputs of faces. Okay?
So, here's my input faces. I give you bunches and
bunches and bunches of faces. What do you
think PCA would do? What do you think the
first principle component of PCA would be over
pictures of thousands and thousands and thousands of faces.
>> Over all darkness of the image.
>> Actually that is exactly right. The first thing that
PCA tends to do with images, we are actual, we
are talking pictures, not just sketches here, is it finds
the direction of maximum variance and that tends to be
brightness or luminous. Which kind of makes sense because
that's typically the kind of thing that kids vary the
most. So in fact, the first thing people often
do when they're trying to use PC on faces
is they normalize all of that away. Because the
first principal component isn't terribly helpful. It's just kind of
giving you the average light. What do you think
the second thing it would find it would be?
>> Hair versus not hair?
>> No. Interestingly enough. What it ends up finding is sort of the
average face. Which is kind of like the question you asked me earlier
about what happens if you go through the origin.
>> Mm.
>> So, there's actually names for this. It's called the
Eigen Faces because, you know, as I noted before, it's
an Eigen problem. And it basically finds kind of the
average face. And that's kind of weird. But it works for reconstruction.
I don't know what the average face here would look
like. Probably something like this. Yeah, so you can see
how useful that would be. Not even clear that's a
face. But it works for reconstruction. Do you know what ICA
ends up finding?
>> Noses?
>> Yes, it finds noses. It finds eye
selectors. It finds mouth selectors. It finds hair selectors.
And, I think, intuitively the way I would think about
this is because PCA is doing this global orthogonality thing, it's
going to be forced to find global features. ICA I didn't
say was global, and that's because it's basically local, it doesn't
care about orthogonality. So it tends to find parts of.
If you feed it the metrics the right way, it tends
to find parts of. And so it ends up finding these
little selectors. So this has a really nice outcome in cases
like natural images or natural scenes. So what do you think
happens Michael, if I take natural scenes, you know, pictures of the
forest and grass and walking around. Just things that I would see
if I were just walking out in the world and taking bunches
of pictures. And I feed it into ICA. Well the answer
for PCA, by the way, is the same as before, is you
get brightness, you get sort of the average image, things that really
make sense if your goal is to do reconstruction. But ICA actually
gives you something different. What do you think the independent
components, the underlying causes, so to speak, of natural scenes are?
>> I'm still thinking about the face parts. But, by analogy, it seems like it
would be the, you know, things in the world like, trees, and rocks, and ground.
>> Not quite, and that's I think in part because, there are too many of
those things that kind of overlap in
too many different ways. It actually finds something
more fundamental.
>> Edges.
>> That's exactly right. ICA finds edges. Now
for me, that is incredibly satisfying. It says that
the independent components of the world are edges. Now
there are two things that come out of this.
One is just the satisfying feeling you get by
realizing that in it there's a algorithm that, on
it's own, recovers something fundamental like edges. That's very
nice. But the second thing that's nice about it
is, once I use ICA to do my
feature transformation and discover that what it's learning are
edge detectors, well then I can just write edge
detectors. I can just write algorithms that are very
fast, very efficient, that can go through images of
natural scenes and pull out the edges. It's unclear
how to do that with, you know, different principled
projections, but if edges are the fundamental building blocks
of natural scenes then there are people who know how to
write edge projectors very quickly. By the way, you get a
similar result, I won't talk about it, but you get a
similar result in our information retrieval
problem where you have documents. And
you can read this in the material we gave you. But
what ICA ends up giving you for documents are topics. And
they're very easily interpreted that way. Collections of words that select
for specific topics that are themselves made up of collections of words
that select for topics and collections of uncorrelated words
that get rid of distracter documents. So the independent
components of scenes are edges and the independent components
of documents are topics. And that feels very nice, but
in both of these cases, both with edges and
with topics, it turns out that the form of these
topics are something that you can compute very, very
quickly. And sort of independently of the underlying ICA algorithm.
So why does that matter, Michael? Well, remember what I
said originally about unsupervised learning and what it was good
for, was understanding your data. Doing kind of data analysis
from a human being's point of view. So what something
like ICA, and other algorithms we might imagine, do, is
they allow you to do analysis over your data, to
discover fundamental features of them, like edges, or topics or
noses and eyes. And then once you understand that's what's making
up your data, you can simply write code that will select for it, or figure
out what the important parts of your
data are correspondingly. And so here, we haven't
just done this feature transformation problem for
the sake of doing classification, feature transformation actually
helps us to understand the fundamental underlying
causes, or at least structure of our data.