>> All right, Michael. What's the answer?
>> All right. Well, let me start off with what I think the
answer isn't. So, the last one, boosting tends to overfit, if boosting trains
too long. You just told me a story about that not being true.
So I'm going to eliminate that
one from consideration. Boosting training too long.
>> Oh, nice to know you were listening.
>> [LAUGH] Boosting training too long, seems like
not a good reason for it to overfit.
>> You're correct.
>> All right. Boosting tends to overfit if it's
a nonlinear problem. So, that doesn't seem right. I
mean I guess, no, this one just doesn't seem right
at all. Like I don't see why, why the problem
being linear or nonlinear, has anything to do with overfitting.,
>> Okay.
>> A whole lot of data is the opposite of what tends
to cause overfitting. If there's lots of data then you'd think that it
would actually do a pretty reasonable job of, you know, there's a
lot to fit. There's a lot going on there. It's unlikely to overfit.
>> Right, and
in fact if a whole lot of data included all of the data, and you actually
could get zero training error over it, then
you know you have zero training zero generalization error.
>> because it'll work on the testing data as well, because it's in there.
>> Right.
>> All right. Weak learner uses artificial neural network with
many layers and nodes. So I'm guessing that you wanted
me to think about that being something that, on its
own, is prone to overfitting, because it's got a lot
of parameters.
>> Sure.
>> So, if, and now we're doing boosting
over that. So we fit a neural net, and
then we fit another neural net, and we
fit another neural net. And we're combining all the
outputs together in the correct, weighted way. It's
not obvious to me that that should be a
good thing to do. I'm not sure it would overfit, but it seem like it sure could.
>> OK, so you're, you're, so for now let's
put a little question mark to it. You think that
might be the right answer, but you want to think about it some more?
>> Yeah let me, let me look at the first
one. Weak learner chooses the weakest output. Well, I mean
boosting is supposed to work as long as we have
a weak learner. . And it doesn't matter if it
chooses the weakest or the strongest. All that matters is
it does significantly better than a half. So, like I
feel like the only one, the only one of these
choices that is likely to be true is the second one.
>> And that is, in fact, correct. So let me give you an example
of when that would be correct. So let's imagine
I have a big powerful neural network that could represent
any arbitrary functions. Okay, got lots of layers and lots
of nodes. So, boosting calls it, and it perfectly fits
the training data, but of course overfits. So then it
returns, and it's got no error, which means all of
the examples will have equal weight. And when you go
through the loop again, you will just call the same
learner, which will use the same neural
network, and will return the same neural network.
So every time you call the learner, you'll get zero training error, but you will
just get the same neural network over and over and over again. And a weighted
sum of the same function is just that
function. So if it overfit, boosting will overfit.
>> Interesting. And not only will it overfit, but it'll
just, it'll be stuck in a horrible loop of error.
>> Right. So that's why this
is the sort of situation where you can imagine
boosting a lower fit. If the underlying learners all
overfit and you can never get them to stop
overfitting, then there's really not much you can do.
>> Interesting.
>> Now, I do want to have a little semantic argument
with you for a moment, Michael. You used the word strongest at
some point, when you were talking about using the weakest output. And
I just want to point out that, that doesn't really mean anything.
>> What do you mean, it doesn't mean anything?
>> Well, so what's a strong, what would you call
a strong learner?
>> One that is far away from it. If a
weak learner just has to do a little bit better
than a half, it seems like a strong learner would
be something that would be very close to being accurate.
>> Right. Of course, on the other hand, if
by that definition all strong learners are also weak learners.
>> Sure.
>> Because anything that does better than a half is still doing better
than a half, which is all it requires to be a weak learner.
>> Yeah, but that's kind of true of people
too. Like a strong person is also a weak person.
>> No.
>> Well it depends how you define it. So,
if you say a weak person is someone who can
at least lift their own arms, then strong people are
also weak people in that they can lift their arms.
>> Yes if you define it that way and if I define
blue to be purple, then I can say blue is purple. But that's
not how people define weak people. They define weak people, by saying they
can't lift more than, not that they can lift at least as much.
>> I see. So it's this piece of terminology that boosting uses that is in
error, not me.
>> That's one interpretation. It's not the one that I would use, but
it's one interpretation. When you say something like a strong learner, I mean,
it makes sense to use that kind of term, and sort of throw
it around, and say, well, by a strong learner I mean someone who's,
or a learner that's going to overfit, or is going to always do
really well on the training data. But in kind of a technical definition
it's very difficult to sort of pin down. So don't get too caught
up what a strong learner means if you want to write a proof.
Seems fair?
>> Good point yeah, also, also that this whole notion that strong
is sometimes defined as not weak. And it is not the case that
if you have something that's not a weak learner that it's, then
it's a strong learner. In fact, it's no lear, no learner at all.
>> Exactly. So, a weak learner's just defined in a way that
basically says, it gives me at least some information. Good. Let me
just throw one more thing in here and then we can stop
talking about this. There's another, a couple of other cases where boosting
tends to overfit. The one that matters the most, or
comes up the most, is in the case of pink noise.
>> Did you say, peak noise?
>> I said, pink noise. I even wrote it in red, which
looks like pink. It's a strong pink as opposed to a weak pink.
>> [LAUGH]
>> I'm sorry. There's no way for that to be obvious from what we've
talked about, but as a practical matter,
pink noise tends to, cause boosting overfit.
>> Okay, but this is not a term I'm familiar with
unless you're critiquing the musical stylings of a particular performer.
>> [LAUGH] No. Although I did recently see, see them in concert. But
that's a whole other conversation. Okay, so pink noise just means uniform noise.
>> I thought white noise was uniform noise.
>> No, white noise is Gaussian noise. Okay, so pink noise is uniform
noise and white noise is Gaussian noise. This is why, Michael, by the way,
if you ever try to set up a studio or a cool stereo system in your house, you
want a pink noise generator. So that it covers
all the frequencies equally, not just the white noise. generated.
>> Hm.
>> But boosting tends to overfit in those sorts of circumstances. And you
can read more about it in the notes if you want to. But
the one that I want I really want people to get is, that
if you have an underlying weak learner that overfits, then it is difficult
for boosting to overcome that. Because fundamentally you've already done all of
your overfitting and it's, there's really not much for those things to do.
>> Okay. Got it?
>> Got it.
>> Excellent. It all ties back into margins, and it's all one
big story, which I think is the lesson of all of machine learning.