Okay Michael. So let's see if we can jump
on that notion of useless and some other word
you used. I'm going to define what it was you
were trying to say [LAUGH] by calling it relevance, okay?
>> Okay.
>> So it turns out this thing that you were hanging onto where you
were saying that well, C which you thought was useless and then it turned
out to be useful and you were. You were coming up with reasons why
doesn't change anything, it doesn't provide any
information. We can actually make very precise informal.
So here's a formal definition of the notion
of relevance and i'm just going to go through
this. Let me know if it makes sense. So we have a set of features and let's just
say they're XI, so there is X1, X2, X3. The X end features
and any particular feature X and I. Is going to be strongly relevant,
exactly in the case if removing that feature degrades
the Baze optimal classifier, that is what BOC stands for, does that make sense.
>> Did we talk about what A BOC is? Yes, way back
in the supervised learning. Lesson on bashe learning. We ended up talking about
the bayes optimal classifier, the one that takes the weighted average of
all of the hypotheses, based on
their probability of being the correct hypothesis.
>> Okay.
>> Remember that?
>> Yeah.
>> And we, and what we actually said is that the bayes optimal classifier is
the best that you could possibly do on average, if you could actually find it.
>> Right.
>> That coming back to you?
>> Yeah i mean we need a notion of
priors and stuff to be able to define strongly relevant.
>> No it really just says that there is a base optimal classifier which
is to say the best you could do. That X of i is strongly
relevant in the case if you didnt have that feature
you couldnt do as well as the base optimal classifier
that had access to all the features. In the quiz
that we did before we know that the actual function
that we're looking for really was a and b. So
if I remove a, I can't actually compute a and
b. Similarly if I remove b, I can't actually compute
a and b. So both a and b are strongly relevant.
>> Okay, I mean in this case
is uses that fact that not only that it's a and b but there is
nothing else that has the same information as a and the same information as b.
>> Right, exactly. And that little, that, that difference you just
noted is the difference between being
strongly relevant and being weakly relevant.
>> Oh.
>> So, a variable, a feature is called
weakly relevant if it's the case that it's
not strongly relevant. So, the order of these
definitions matter. And it turns out that there
exists some subset of your features, let's call that subset s, such that if I
added the feature to that subset s, it
would in fact improve the Bayes Optimal Classifier.
>> So in this case, we're talking about
the Bayes Optimal Classifier on. Oh, I guess
in both cases, we're talking about. The Bayes
Optimal Classifier on the reduced set of features.
>> Right.
>> Comparing it to the Bayes Optimal Classifier on
the full set of features in the first case.
>> Right, or just on
any subset of features. So in particular, in the strongly relevant
case, that's the case. We're saying, well, what would the Bayes
Optimal Classifier be on all of the features, versus the Bayes
Optimal Classifier on all of the features except. X sub i,
and if removing X sub i degrades the performance, then we
say it's strongly relevant. On the other hand, if X sub
i is not strongly relevant, that is, removing it doesn't actually
hurt the base optimal classifier. It can still be weakly relevant
in the case where there is some subset of
the features. Such that if I added x of
i to that subset it would improve the Bayes
optimal classifier on just that subset. So I can
make that a little more concrete: Imagine that we
had another variable, let's call it e, ok? Which
had these values. So, if you look carefully, you'll
notice that e is in fact not a. Mhm.
>> So that means
neither A nor E is strongly relevant, because I can remove
A and still learn B and A by basically making it
B and not E, or I could move E and still
learn A and B by simply using A and B. Agreed?
>> Yea, so its so then B would be strongly relevant but the other two not.
>> Right. However both A and E are
weakly relevent. Because there exists a subset such that
adding it back in gives you better performance.
In particular, A is weakly relevant for any
subset that doesn't include E. And E is
weakly relevant for any subset that doesn't include A.
>> Huh? And that includes the null set in this case?
>> Yes, exactly. If you have a particular feature
which is not strongly relevant, and is not weakly
relevant, then we call it irrelevant. So, in this
case, when, what you were calling C as being useless.
What it actually is is irrelevant. Because it provides no
information to the classifier. You might as well have just simply
had the normal subset and just always output no and you
would do just as well as having the value of c.
>> And yet it somehow turned out to be helpful for the perceptron case.
>> Right. And that's because there's another notion that
we could think about. Which is not relevance but usefulness.
So, let me define that for you, okay? So.