(The Dog by Chucklehead plays) ...so what's really interesting is, no matter what trick
you are doing, you are always getting a treat.
SO that's the decision tree.
So the complexity doesn't really matter.
Hey Ingo, I'm sorry to interrupt your important meeting with our chief furry officer but it's
time for...
Five Minutes with Ingo?
Five Minutes with Ingo.
OK, great.
Do you remember, Graeme, we have been starting here, talking about decision trees, but then
actually we talked about K Nearest Neighbors?
Oh, do you mean this episode?
You know what, forget about decision trees, that's a stupid and boring topic anyways.
We will talk about this another time.
OK.
Yeah, that one.
So, let's talk about decision trees today!
Sure.
We have a dataset here about getting a dog.
So people might make a decision, 'oh, do I pick this dog or not?' and we collected this
information.
This is about the cuteness here, so we have a column here about cuteness.
Cuteness might be low or high.
We have the size of the dog, small or large, and then people decide and we just collect
this information.
Yup.
So that's a table, pretty simple actually.
And we would like to create a predictive model, a decision tree model, which we can then use
for a new case, for a new combination of those values to actually map, to make this prediction,
how are people going to react.
And this is how such a decision tree could look like, OK?
So here you can see we first pick the cuteness and then if the cuteness is high of course
people are getting the dog.
'Aaaaawe!'
And if the cuteness is low, they are also paying attention to the size, and if the size
of the dog is unfortunately large, poor dog.
Well, it's not picked.
So that's that.
OK, anyway.
So that is the result, and this is the data, so the big question is how do we get from
here to here.
And the algorithm for building such a decision tree is actually quite simple.
There's a formula in there, and it's not THAT simple, but the algorithm is super simple.
It's what we call a recursive, top-down strategy.
So we start on the top of this tree and we pick, we select the first attribute, the first
column, we use for making the split.
And after we did this, we then take the different values for this column, and we separate the
data set into the two buckets here.
So in all cases with cuteness equals to high, those actually are then certain cases, exactly
these three rows here, and then there are three other rows which are going into this
side here.
OK so, and then if we figure out, well it's not, it's not completely clear here, we can
make another split, we pick another attribute.
In this case it's simple because we only have one left.
But even for more complex cases, you can use the same algorithm.
And then we repeat this whole thing.
So, the big question is, how do me make this pick, how do we pick this first attribute
at the root.
So, here is a different way to look at the same data.
So let's have a look at cuteness.
We can separate the data now in those two buckets 'low' and 'high', and here are the
classes for those two buckets.
So if the cuteness is low we have one case with yes, and two cases with no.
And if the cuteness is high, all three cases are 'yes.'
Sure.
We can do the same thing for size.
So Graeme, looking at this data now, this re-representation of the data, what attribute
would you pick actually, cuteness or size?
Hmm, I would pick cuteness.
Why?
Because that is a clean list of data.
It yes, yeah, exactly.
So this group here, those three cases there, it really like covers over 50% of the complete
data, and it's what I would call, like it's a pure bucket, it's only yes cases.
And that's really something we would like to achieve, we would like to end up with very
pure cases here.
So that is good, and even in this case two-thirds are no cases, so we have a clear separation,
while here the pure bucket is smaller and here it is just 50%, so it's less clean.
Sure.
And there is actually a formula for this.
Oh, how convenient.
Yeah yeah, just brought this piece of paper in here.
So, we call this the information gain.
And we use something called entropy for calculating the information gain, by just using the weighted
sum of the entropy's for those two cases.
So here, the split is one-third versus two-thirds, and here it's basically a pure play.
And whenever the entropy is pure, the entropy is actually zero.
So that's good.
So we end up with the information of this split of 0.14 and for size it's 0.2.
And then we can actually compare this to the entropy before we did the split, which is
another value, and then we can calculate the information gain, and you can see, if you
subtract this here, that the information gained for cuteness is higher, 0.13, than for size,
.07.
That's why we pick cuteness at the root of this chain.
Got it.
So that's the basic algorithm.
Very very simple, it's a good heuristic, leading to a good predictive model, and that's, well,
all on decision trees from my side.
Fantastic, thanks Ingo!
And this has been your five minutes with Ingo... and Marla!