So this is called the first component, or the principle component, and this is
called the second, or second principle component
of this space. Okay, does that make sense?
>> Yep.
>> Now, here's what's interesting about principle
components analysis. You might ask me exactly
how you do this. There are several, several mechanisms for doing it. For those
of you who have dealt with linear
algebra before, something like this singular value
decomposition might be familiar to you. It's
one way of finding the principal component.
But principal components analysis basically has a lot of really
neat properties, so let me just describe some of those
properties to you. The first property is, well, the two
that I've written here. It finds directions that maximize variants and
it finds directions that are mutually orthogonal. Mutually orthogonal means
it's a global algorithm. And by global here I mean
that all the directions, all the new features that they
find have a big global constraint, namely that they must be
mutually orthogonal. It also turns out that you can prove. Which
I will try to give you a little bit of evidence
for. But I'm going to prove formally, that the PCA actually gives
you the best reconstruction. Now what do I mean by best
reconstruction? What I mean is, if you think of each of
these directions that it found, in this case. You found this
one first and found this one second. The first thing you,
I, I hope you see is that if I return these two
dimensions, I have actually lost no information. That is, this
is just a linear rotation of the original dimensions. So if
I were to give you back these two different features,
you could reconstruct all of your original data. You see that?
>> Wait. When you say features you mean, what we're
going to give it is, for each of those little black
data points, we're going to say how far along the red
axis is it and how far along the orange axis
is it.
>> Right.
>> So it really is just a, a, a kind
of a relabeling of the. of the dimensions. Right, so just
like when I think about these points in x and
y space, the original feature space, whenever I give you value
here for this feature I'm just describing how far along
a black dot is on this dimension or this projection. Now
the second dimension or the second feature just tells me how
far along a dot is. On this particular dimension or axis,
and similarly about projecting on the read and on
the orange, I'm telling how far along a point is
along this axis and along that axis. So if I were to return, if I were to take X
and Y and transform them into this new one
and two, I would have given you different values than
I did from X or Y, but they're actually just
the same point. And so I've thrown away no information.
>> So that's a pretty good reconstruction.
>> That's a pretty good
reconstruction. But what principle components analysis
does for you. Is if I take just on of these dimensions,
in particular, the first one, the principle component, I am
guaranteed that if I project only in to this space
and then try to reproject into the original space. I
will minimize what's called L2 error. So do you understand that?
>> I'm not sure let me check. So, you're saying if we instead of,
all right we take the little black dots, they now have a red dimension and
orange dimension one two, and now if we reconstruct using only the first
dimension, I guess it just puts the black dots on the red line.
>> Yep. And so there is no, they have no existance in that second dimension.
>> Correct.
>> And now you're saying of all the different ways
that I could do that to kind of project it to
a, to a linear sequence. This is the one that's
going to have the smallest. L2 error which if im not mistaken
is the same kind of reconstruction error we talked about
in all the other times we talked about reconstrucion. So it's
like squared error. Thats exactly right, its squared error. This
particular notion is called a [UNKNOWN] norm but thats really just
talking about distance. What this means is that if I
project onto this single axis here, and then I compare it
to where it was in the original space, the distance,
the sum of all the distances between those points will actually
be the minimal that I could get for any other projection.
>> Cool.
>> And you can, you can sort of prove this. It kind of makes sense if you
just think about the fact that points. Always
start out in some orthogonal space. And, I'm basically
finding in scaling and a rotation such that
I don't lose any information. And I maximize variance
along the way. By maximizing variance, it turns out
I'm maximizing or maintaining distances as best I can
in any given dimension. And, so, that gives me the
best reconstruction that I can imagine. Now, you might ask
yourself, Is there anything else nice about principal components analysis
given this reconstruction error? And there's another property of PCA that
is very, very useful. And it boils down to the
fact that it's an eigenproblem. What happens when you do
principal components analysis is you get all of these axes
back, and in fact, if you start out with. N dimensions,
you get back N dimensions again, and the job
here for a future transformation as you might recall, is
you want to pick a subset M of them hopefully
much smaller than N. Well, it turns out that associated
with each one of these new dimensions that we
get. Is its eigenvalue. That eigenvalue is guaranteed to be
non-negative, it has a lot of other neat properties.
But what matters to us here is that the eigenvalues
monotonically non-increase, that is, they tend to get smaller
as you move from the principal to the second
principal, to the third, to the fourth, to the
fifth, to the sixth, and so on to the nth
dimension. And so, you can throw away the ones
with the least eigenvalue. And that's a way of
saying that you're throw awaying these projections, or the
directions, or the features, with the least amount of variance.