Alright, Michael. So, you got it? Which line do you think is best?
>> Alright, so if I choose one green and one red [CROSSTALK]
that means there's nine different possible
lines. But they all separate the points.
>> That is correct.
>> I'm pretty sure. I, you know, I think they're intended to
even if they don't quite. But I think they do, I think they
actually all separate the positives from
the negatives [CROSSTALK]. The positives on
the upper right side and the negatives on the lower left side. So,
I mean the only one that seems further special
is the middle one. If I choose the middle green
and the middle red it's kind of, you know, ecstatically
pleasing, there's a lot of space on, on each side.
>> Okay.
>> So, I'm going to go with that but it's not clear, you
know again, best if I was a photographer that might be the best.
>> Hm, you are a photographer. So, let's pretend I can draw
a straight lines. That's pretty good. That's the line that you've chosen.
>> Yes.
>> And you think that's best. Well, I'm going to give you a
hint, Michael, and tell you that you were correct. That's a pretty good hint.
>> Yeah.
>> But now I want you to figure out
for me why that line is better than this line.
>> Interesting. Alright, well so one thing that's, that
second line. The. Orange line, as oppose to the orange
line, has against it, in some sense it seems
to get really, really close to that bottom minus point.
Maybe it's a little too close. Like maybe that
minus is near other minuses that we just can't see.
And since we drew really really close, a line really
really close to it, it could be we ended chopping
of some of the minuses that. That we might
want to have kept on the other side. And so
maybe by drawing it in the middle it's sort of,
we have the biggest, I don't know like demilitarized zone.
>> No, I like that. So that is a very good explanation
for why you would prefer the middle line over the.
Other line that isn't the middle line. And you can make
a similar argument for any other of the nine lines that,
that or the other eight lines that you could possibly draw.
Let me draw another line for you and you tell
me if you think this one is good or bad and
why it might be better or worse than the middle one.
See if you give me a different answer. I can take
one of the parallel lines which again we will pretend is a straight line and
put it there. Or I can take in fact the other line that is meant to be parallel
to it and why aren't those lines better? Or just as good.
>> I guess I'd use the same explanation as before
which is that these, the, the line that was, is really
close to the pluses maybe is a little too close
for comfort and it, it gets very close to that plus.
The, kind of, I don't know. I could point to it, but maybe
you should point to it. The plus that's really close to the line.
>> I'm pointing to it.
>> Because Again if there's a just, you
know, other data points near there, it's going
to make this distinction between the positives and
the negatives that isn't really warranted by the data.
>> Right, so I could make the op the
opposite argument with you, Michael, which is that listen, they
all separate the points. They completely explain the data
as we see them. So, all three of those lines
explain the data. In fact, all nine of those lines explain
the data. So, why aren't they warranted by the data. What's the
problem that you might run into if you put one line very
close to the positives or one line very close to the minuses?
>> Well, again...so, so...it may be that they fit
this data. But, this is just a sample from the
population. And so... You know, we don't know what's
going to happen really close to that other plus is.
So like in the nearest neighbor algorithm for
example which you always try to make me remember,
it's going to want to, you know, if it's
closer to the pluses, it should be a plus.
>> Right. That makes sense and I think that's exactly right so your
intuition and I think the intuition that people should take with them is that
lines that are too close to the positives. Or lines that are too close
to the negatives, sort of have this
feature where you're believing the training data
that you've got too much. You've decided that all of these
boundaries are fine because of the specific set of training data
that we've got. And while you want to believe the data, because
that's all you've got, you don't want to believe the data too much.
>> That's overfitting.
>> That's overfitting. So, the problem with those two
lines, or one way to think about the problem with
those two lines, is that these lines are more likely
to overfit. And why is that? It's because they're believing
the data too much. So, given that we're trying to avoid
overfitting, what could you say about the middle line? So, here's
the argument that I would make, and you tell me if
you buy it, Michael. This line, or the line it's intended to
represent anyway, is the line that is consistent with the data
while commiting least to it. That seems like a good way
of saying what I was feeling, yeah. It is funny though
because like, overfitting up to this point, we you, we were generally
talking about overfitting as being something where there's
a great deal of model complexity in some sense.
>> Mm-hm.
>> And it doesn't seem like those lines that are
closer to the pluses or closer to the minuses. Are
inherently more complex, they're still lines. It's interesting that they,
they, they kind of maybe behave as if they are.
>> Right, and in fact they, they're,
it's a more, sort of, literal interpretation
of the words over and fit, right? You, you have decided to fit the data,
and you believe it too much, and what you really want to
do is commit. The least that you can commit to the
data while still being consistent with it right. So this basic
idea of, uh,uh, finding the line of least commitment in the
linear separable set of data, is the basis behind support vector
machines. So what I want to do next is I want to
see if we can come up with some equation that would
help us define such a line. It's easy in this case
cause we're staring at it but if you imagine these points were in
700 and. Thirteen thousand dimension it would pretty difficult to find such a
plane just by staring at it so let's see if we can try
to work out how you go about finding this least commitment line okay
>> Cool.