>> Okay, Michael. You got an answer for me?
>> Yeah, I think so. I kind of like the method of dropping features
that we don't need. So I'm going to see if I can apply that idea.
>> OK.
>> So, for the decision tree case, if I had all
4 features, could I even learn this thing? I think I
could. Let's see, I could do A. And if A is
true, then I can split on B. And if B is true,
then I output A plus, otherwise I output minuses.
>> So, hm. Is that a function? Do you know what function that is?
>> It's a decision tree.
>> Yes, but what, do you know if
there's a Boolean function that decision tree represents?
>> I don't know. Do you want to draw it and I can look at it?
>> Yeah, sure. Let's do that. So first you split on A. Then you said A is
either zero or 1. And then in the case where it was a 1,
you split on B. And if that were a zero, you output no, and if that is
a 1, you output yes. And over here, you output no. That's what you said, right?
>> Yep. And that works for this example.
>> Mm hm.
>> Or this set of examples. And, let's see, so with 1 plus, oh
I see. So it's it's AND right? So it, if A and B are true,
then the output's true, otherwise not.
>> Right. And in fact that is exactly the function that I was thinking of.
>> Oh.
>> So this, the true underlying function here is in fact AND of A and B.
>> All right. And oh, I see, so in particular, now we've
answered the first question. Because I, I need, well okay I, I, I
only need A and B. And in fact, having just one of them
alone isn't going to be enough. If I just had A then the
decision tree can't learn it. And if I just had B,
then the decision tree can't learn it. It needs more information.
>> Right, and even in these examples you haven't,
you can't really luck up and have picked C or
D to learn this. Because C is always 1, and
so you can't output the correct answer for all cases.
And D is 1 in the wrong place, and
zero in the wrong place, but zero in the right
place a couple of times. So you can't do something
like take the not of D and haven't gotten lucky.
>> I see.
Well yeah, and C in particular seems exceptionally useless.
>> Ooh, I like that word.
>> [LAUGH]
>> I am going to say useless. Useless. Okay, fine.
So, that's right. So, it, and maybe this helps you
a little bit with the next one. I know that
you now know what the true function is. It's AND
A or B. And you've decided that D doesn't help
a lot, and C is completely useless. Why'd you say
it was useless, by the way? Because it's the same
in all the cases. It doesn't discriminate between the positive and
the negative labels.
>> That's right. So it is useless in that regard. So, good. This is
the answer I was looking for, for
decision trees. What about for the simple perceptron?
>> Okay, right. So I'm not great at doing this kind of
thing in my head. But I guess the good news is, oh, how
about this? How about this? Since we got it down to only these
two features matter, we can draw it as a two dimensional plane, right?
>> OK.
>> So mark 0,0. Put a minus at 0, 0.
>> OK.
>> Put
a minus at 0,1.
>> Mmhm.
>> Put a minus at 1,0.
>> Mmhm.
>> And put a plus at 1,1. So that's
when we're using the A and the B features.
>> Yep.
>> And this is great, because it's linearly separable. So we just need to,
yeah, we just need a set of weights that can do that. And let's
see. So, how do we do that? So, if we add, let's see. If
we do a weight of 1 on A, and a weight of 1 on B.
>> Mm-hm.
>> Then.
>> >> Then.
>> Let me, let me draw it.
Maybe that will, that will help. So, you want to pick a
weight vector which is, you said, 1 for A and 1 for B.
>> Mm-hm.
>> Okay, all right. Go.
>> And that will give us for the first example, that gives us an
output of zero. For the next one an output of 1, the next one
an output of 1, and the last one an output of 2. So, we
would like, if it was like, uh-oh. Wait. So, if it, if the weight
vector was that, then what we really want it to be
is WTX greater than 1. But we can't. It says zero there.
>> Mm hmm. In fact, wait, isn't it the case that if you don't have, if it's
greater than or equal to zero, it's, we're
really talking about lines that go through the origin.
>> Yep that's right. This is an origin limited perceptron.
>> So it seems like we're hosed, right? There isn't going to be a separator
that goes to the origin that lops off the plus.
>> That's right.
>> So, I guess what we usually do in
perceptrons is we include, was it like a bias
unit, like a, like a, something that's all 1's.
And then that let's us move the threshold around.
>> Right. So normally this would look like
this. W transpose X plus B is greater
than zero. But I didn't let you have that, so you can't have that.
>> But maybe I can. You say I can add B in.
What if I could add C in? Because if you look at this
>> Well, what would that give you?
>> Well, look at the C column, it's all 1's. It's the same as if
it was, it like, the thing that you called B a moment ago, the, this,
this bias unit. So if we actually include A, B, and C as our features,
then we can represent this with a,
what did you call it, origin limited perceptron.
>> Right. So then, and I'll even draw it in a
different color, if we had WB, what did you have? 1, 1,
and.
>> I'm going to say negative 1.
>> Negative 1, and that's A, B, and C.
Then what would that give you, in the first case?
>> So negative 1, zero, zero, 1.
>> Yep.
>> So it's only, the positive example is the
only one that's greater than zero, strictly greater than zero.
>> Yep, that's exactly right. So,
you still think C is useless?
>> I see. Well, when I said useless.
>> Mm-hm.
>> I meant it didn't give any information.
>> And that's exactly right. It doesn't give
any information. But that does not make it useless.
>> Interesting. Okay, well, I now regret that word choice.
>> C's everywhere across the world are thanking you. [LAUGH]
>> So this actually brings me to the
last point that I want to make about feature
selection, which really gets down to noticing the difference between useless
and what I'm going to call relevant. So let's dive into that, okay?
>> Yeah, that, that would be really helpful. Is
that, and that's the right answer? The first three?
>> Yes, that's the right answer.
>> Cool.