Alright, Michael, so, here's what you have before you. You have the same
housing data that we've looked at a couple of times before. I've, for the
sake of readability, I've drawn over, some of the data points so that
they're easier to see. This is exactly the data, that we've always had. Okay?
>> Okay.
>> Now, you'll notice that I marked one of
them as green, because here's what we're going to do.
I'm going to take the housing data you've got, I'm
going to do some ensemble learning on it. And I'm
going to hold out the green data point. Okay? So of
the nine data points, you're only going to learn on 8
of them. And I'm going to add that green data point
as my test example and see how it does. Okay?
>> Okay. So that sounds like, cross validation.
>> It does. This is a cross validation. Or you could just say,
I just put my training set and my test set on the same slide.
>> Okay.
>> Okay, Michael, so the first thing I'm going to do is I'm going to pick
a random subset of these points. And just for the
sake of the example, I'm going to pick five points
randomly. And I'm going to do that five times. So
I'm going to have five subsets of five examples. And by the
way, I'm going to choose those randomly, and I'm going to choose
them with replacement. So we're not going to end up in the
situation we ended up just a couple of minutes ago
where we never go to see the same data point twice.
Okay?
>> Yeah.
>> Alright. So 5 subsets of 5 examples, and then I'm
going to learn a third order polynomial. And I'm going to take those
3rd order polynomials. I'm just going to learn on that subset, and then
I'm going to combine them by averaging. Want to see what we get?
>> Oh, yeah, sure.
>> So here's what you get, Michael. Here I'm showing you a plot over
those same points, with the five different
3rd order polynomials. Can you see them?
>> Yeah. They're, right. There's like a bunch of wispy hairs.
>> Just like most third order polynomials. And as
you can see they're, they're kind of you stare at
them and you see their kind of similar. But
some of them they veer off a little bit because
they're looking at different data points. One of them
actually very hard to see because it's only one like
this. Actually veers off like this because just, purely
randomly, it never got to see the two final points.
>> I see.
But they all, but they all seem to be pretty much in
agreement, like between points three and
four. There's a lot of consistency there.
>> Right. Because just picking five of the
subsets you seem to be able to either get
things on the end, or you get things in
the middle. And maybe one or two things on
the end it sort of works out. Even the
one that doesn't see the, the last two points
still got to see a bunch of first ones
and get this part of this space fairly right.
>> Cool.
>> Okay.
So the question now becomes how good is
the average of these compared to something we might
have learned over the entire data set? And here's
what we get when do that. So what you're
looking at now Michael, is the red line,
is the average of all of those five third
order polynomials. And the blue line, is the fourth
order polynomial that we learned when we did this
with simple regression, a couple of lessons back.
>> Okay.
>> And you actually see them pretty close.
>> Why is one of them a fourth order, and one a third order?
>> Well what I wanted to do is try a simpler set of hypothesis,
than we were doing, than when we were doing full blown regression. So third
order simpler than fourth order. So, I thought we'd combine a bunch of simpler
rules. Then the one we had used before and see how well it does.
>> You want to know how well it does?
>> I would!
>> Well it turns out that on this data
set and I did this many, many, many times
just to see what would happen with many different
random subsets. It typically is the case that the
blue line always does better on the training set,
the red points, than the red line does. But
the red line almost always does better on the
green point on the test set or the validation set.
>> Interesting.
>> That is kind of interesting. So wait, so let me get this straight.
It seems sort of magical. So, so it learns an average
of third degree polynomials, third order polynomials, which is itself a
third order polynomial. But you're saying it does better by doing
this kind of trick than just learning a third order polynomial directly.
>> Yeah, why might you think that might be?
I have a guess, you tell me what you think.
>> Wow, so well, I mean, you know, the danger is often
over fitting, over fitting is like the scary possibility. And so maybe
by, by kind of mixing the data up in this way and focusing
on different subsets of it. I don't know. Somehow manages to find the
important structure as opposed to getting
misled by any of the individual datapoints.
>> Yeah. That's the basic idea. It's kind of the
same thing, at least that's what I, I think that's
a good answer. It's basically the same kind of argument
you make for cross validation. Alright. You take a random
bunch of subsets. You don't get trapped by one or two points that happen to
be wrong because they happen to be wrong
because of noise or whatever. And you sort
of average out all of the variances and
the differences. Hm. And often times it works.
And in practice this particular technique of ensemble learning
does quite well in getting rid of overfitting.
>> And what is this called?
>> So, this particular version, where you take
a random subset and you combine by the mean, it's called bagging.
>> And I guess the bags are the random subsets?
>> Sure.
>> [LAUGH]
That's how I'm going to think of it.
>> That's how I'm going to think of it. It also has another name
which is called bootstrap aggregation. So I
guess the different subsets are the boots.
>> [LAUGH] No,no, no, no bootstrap usually
refers to pulling yourself up by your bootstraps.
>> Yeah, I
like my, I like my answer better. So, each of the subsets
are the boots and the averaging is the strap. And there you go.
So, regardless of whether you call it bootstrap aggregation or you call it
bagging, you'll notice it's not what I said we were going to talk about
during today's discussion. I said we were going to talk about boosting. So we're
talking about bagging but we're going to
talk about boosting. The reason I wanted
to talk about bagging is because it's really the simplest thing you can
think of and it actually works remarkably well. But there are a couple
of things that are wrong with it, or a couple
of things you might imagine you might do better. That
might address some of the issues and we're going to see
all of those when we talk about boosting right now.