So let's see if we can be a
bit more formal about this notion of usefulness. Okay?
So, I erased all the stuff I had before
but basically you could summarize that last side, as
saying that relevance measures the effect on the
bayes optimal classifier. Right? So a variable is relevant
if it can make the bayes optimal classifiers performance
better or worse. So, really relevance is about information.
So, when you were talking earlier about things that you might use for
filtering you said, well, I like things that have variance or things that have
entropy or things that give me information gain. Those are all measures of
the information that is present in a
particular variable or a particular feature. Right?
>> Yeah.
>> So really from the bayes optimal
classify point of view the only thing that
matters is how much information a particular variable
provides. You know, conditioned on some label or
just in general how much information it provides so a
variable like C here which doesn't change has zero entropy, provides
no information, independent of the value of the label and therefore
cannot be relevant to the bayes optimal classifier. That make sense?
>> It does. Now I, and, and the,
this this notion of help, usefulness. Not helpfulness, but
usefulness. Is when we condition on particular predictor
and that's why we can have C being useful
in the context of a, perceptron.
>> Right. So usefulness is exactly about effect on error given a
particular classifier, or some specific model.
So usefulness is exactly about minimizing error
given some particular model or some particular learning algorithm, right? So, in
this case although C is clearly not relevant, it is in fact useful
at least for something like W transpose X. Now,
this is not useful for a decision tree, nor
is it relevant. It is not relevant to this
particular problem, but it is however useful for some algorithms.
>> Now, just to clarify so, so it
seems like the base optimal classifier has a
privileged position in this definition right, so can
we define relevant as measuring effect on, a perceptron.
>> Relevance, no.
>> No I'm
saying, but it, you just, you plugged in, a, a kind of
classifier there. Why can't we plug it some other kind of classifier?
>> For the base optimal classifier?
>> Yeah, why is that one special?
>> Because the base optimal classifier
captures this notion of the optimal thing
that you could do. It's not a specific algorithm. I mean you could write
down an algorithm that would compute the base optimal classifier, except that it
requires you looking at all possibly infinite
number of hypotheses. Right? But it is,
the, it the base optimal classifier computes
the best label, given all the probabilities
that you could, ostensibly compute over all
the hypothesis space. It doesn't have to
actually require specific algorithm to do so.
It truly is a measure of information
of variables. So, any other algorithm you
have has a bias, in particular inductive bias.
>> Okay.
>> [LAUGH]
Okay, that's a fine answer. So, at the very
beginning of our discussion, Michael, you actually asked me this
question of what the criteria was? When I said that
features selection was about maximizing
some criteria, removing features according
to some criteria and I told you that eventually we'd
kind of get to the point of what the criteria is.
The notion of relevance versus usefulness gives us an idea
of thinking about what that criteria is. Ultimately we've been
talking about unsupervised learning, mostly in a kind of vacuum but
presumably you know wheather some
particular description compact or otherwise, some
particular label is in fact a good one based on how
it's used later on. So, one way of thinking about this
is the labels that I come up with for a set of
data are exactly good ones in so far as they help
me to do something else like classification later. All that clustering
that we did before, like with k means and E M.
You could think of those as a kind of feature transformation algorithm
which is what we'll be talking about next where you've taken a bunch
of features and you've converted them into something simple like a label. And
whether that label is a good label or a bad label, depends entirely
upon whether you can then do some
kind of classification or regression problem later.
>> Label in this case meaning the cluster name? Right.
>> Got it.
>> Okay.
>> That's interesting.
>> So, you'll actually find if you go through the
literature and you, you look at some algorithms people have
predicted, that a lot of the, measures like information
gain or entropy or whatever, often end up being couched
in terms of relevance. But ultimately, what we really
care about is usefulness, or at least one could argue.
>> Cool.
>> Okay. So that's pretty much what I wanted to talk about
for a feature selection. There's a lot of algorithms and, I mean,
if you go and you look at the material we've made available
to everyone you'll be able to see some of these algorithms discussed
in more detail. But at a high level these are the, the key
issues that I wanted you to see. So, with that Michael, let's wrap up.