okay, wait. So let me see if I can echo some of that back. So, it's almost as
if what we're doing here is we're doing a
transformation into a new space where feature selection can work.
>> Exactly, and in fact, here's something kind of interesting for you.
It turns out that if the ion value of some particular dimension
is equal to zero, then it means it provides no information whatsoever
in the original space. So, if I have a direction, if I have
a dimension that has an eigen of zero, I can
throw it away and it will not affect my reconstructioner.
>> I'm trying to remember if that makes it irrelevant or useless.
>> Well, it certainly makes it irrelevant. If there's no variance, that's the
same thing as saying it has zero entropy, because it never changes. So it's
irrelevant. Now whether it's useful or not, well, probably not, but it might
be useful for something like, our simple,
[INAUDIBLE] example that we used last time.
>>
Gotcha.
>> Okay. So does this all make sense?
>> So one question I have is in the example
that we just kind of worked through there was a blob
of data and when we drew the red line through
the maximum variance direction it went through the. The xy origin.
>> Um-huh.
>> Does it have to? Is it necessarily the case that it's going to?
Or you know is the algorithm restricted
to have to put things through the origin?
>> Well so my
answer to you is that, that's actually a very complicated
question. And in principle it, so to speak it, it
doesn't really [LAUGH] matter. But in practice what people do
and their doing something like PCA is they actually subtract the
mean of the data. Or the central of the data
from all the data points. So it ends up being
centered around an origin, for the origin. But what this
means is that you can then interprit what we're doing is
finding maximum variants. As capturing correlation.
>> Okay. That's helpful.
>> And otherwise, if you don't do that, what you end up with is
effectively a principle component that at least
intuitively kind of captures the notion of where
the origin should be, and it's not terribly helpful for what it is we're
trying to do. So my answer is, no it doesn't, and yes it does.
>> [LAUGH]
>> Okay?
>> Sure.
>> Okay. So, that's, that's basically
PCA. It's got all these neat properties.
Let's sort of summarize them again. It's, well, actually.
Let me add a couple beyond these. It is a
global algorhythm. It does give you best reconstruction error. Which
seems like a very nice thing to, thing to have.
And, it tells you, which of the. New features
that you get out are actually important with respect to
this notion of reconstruction by simply looking at their corresponding
eigenvalues. They also have a couple of other practical properties.
In particular, it's very well studied. And what that means
in this case is that there are very fast algorithms that
do a very good job, even in weird cases. Even with
large data sets, at least if they're appropriately sparse, of being
able to compute these things. People have been working on this
problem again, since before we were born Michael, and they've gotten
really good at finding principle components even for what would be
very difficult spaces. But this does lead me to a question
though which is another question, which is okay, so
you've got an algorithm that probably gives you the best
reconstruction. But what does it have to do with classification?
This is kind of like the question we had before
about relevance versus usefulness. So we find a bunch
of projections which are relevant in the sense that they
allow you to do reconstruction. But it's not clear that
if you threw away some of these projections, the ones
with low eigenvalue, that even though you'd be
able to reconstruct your. Original data is not
clear that would help you do classification later,
can you see how that would work out?
>> Sure, I mean like, it just could be that that's
not where the information is about what the labels ought to be.
>> Right, so imagine for example that one
of your original dimensions is in fact directly related
to the label and all the rest are
just, say goush and noise. But the, the variance
of that particular direction is extremely small.
>> It might end up throwing it away.
>> It'll almost certainly end up throwing it away. Which means
you'll end up with a bunch of random data that doesn't
actually later help you do classification. And that's because this looks
a lot like, if I can do the analogy, a filter method.
>> I was going to say that, yeah.
>> Oh. So this is kind of like filtering. And in fact it's going
to turn out that the other two items we're looking at too are like
filtering although their particular criterion might be
more relevant. We'll see. Okay? Alright, so do
you understand principal components analysis? Again, I'm not
asking you to understand exactly how you would
run a principal components analsyis algorithm. That
stuff's in, in all the text, but just
to sort of understand exactly what is is
trying to do, what it's trying to accomplish.
>> Yeah, I think so. So, it's, it's taking the data, it's finding
a different set of axis, that are just like regular axis in that
they're mutually orthogonal. But it lines up the varience of the data with those
axis, so that we can drop the least significant ones, and that gives us a
way to do. Feature selection. But the
whole thing is a feature transformation algorithm, in
the sense that it first moved the data around, to be able to do that.
>> That's exactly right. So, you do, transformation into
a new space, where you know how to do filtering.
>> Got it.
>> Excellent.