Ingo! Ingo! Hey man, it's time for Five Minutes with
Ingo. Do you remember last time that we talked about how to assess model quality, and
models help people. They make them feel good about themselves. All right, okay.
We have been talking about model validation. We have been talking about how to calculate how well a model is predicting the future.
So, and I promise that we are going to talk about a very powerful concept for
that which is called cross validation. So look at those data points here. So I've
put numbers on top of them: 1 and 7 10, 2 and so on. So we have all the data
points here and what we can do now is actually take one of those data points
out. We use all the other data points for building a model. So let's, for example,
start with number one. We take it out, put this somewhere else. We actually
now build a model on the rest of the data points, and then we apply this model
on this one data point we take out. So have a quick look. Or play with the car. Oh sorry. So this
model is actually pretty easy to find. So you could for example say 'whenever
you see a cube the color is red.' So you're predicting the color of those building blocks here, so
in this case, it's a cube so it's red. So that's correct. So you put this back
in here. You take number 2, same model would be created. It's a cube
so it's red as well. Now for the first time you get a blue triangle here. But if
you look at this data, still we can use the model of red cubes here. So we
would say then probably this one here is a triangle, and that's also correct.
You like Chevy's do you? Oh, sorry, yeah. So this one here, number four, is
interesting because now you can clearly see all the cubes are red, all the triangles
are blue. So we would, the model is one hundred
percent sure it's going to be blue, but the model is wrong. It's actually red,
as you can see. So that can happen. It's, it's not like a model needs to be
correct all the time, all of the cases. We have like noise, we have actual situations
the model is not covering. You don't want to run into this risk of overfitting. So
that's a normal thing. He's absolutely right. Thank you.
So, so far we've been correct three times, we haven't been correct one time
and just believe me that for all the other cases we would be correct as well. Okay,
which means we are correct in ninety percent of all the cases and that's how
you can come up with the validation performance or the model
performance by using a cross-validation. So, but the big problem here is, as you
can see we built, we have like ten data points. So, if you would do this like that, you
would also build ten models, which is okay for small data sets, but if you have like 10 billion data points you can't do this.
Another idea would be this one here. So instead of doing what we just did, by looking
at one data point after the other, you can also put the data into batches. Okay
hmm right now I have like 25 data points here. I have five batches,
each batch consists of five data points, and for each batch I can now calculate
how often am I correct. So we have the same data points here and, for example,
for this batch I'm correct. I think it's in 80% of the cases, yeah. Here it's
is a 60? It's 60. Cool, and here 80, 100, 80. 100, 80. So now we can just build the average of those
correctnessess - that's not even an English word. That's definitely Genglish but go on. Anyway, so
it would be 80%. So now you're doing this on a much larger data set. But
we don't build like whatever 25 models, but only five models, so we build five
different models. And we never build a model on the same data as the test set
here we are applying the model on. So we always take one out here, build the model
here, apply the model there, and by that we are also assuring that you're not
running into this risk of overfitting. So I think you had a question. If you have five data batches,
does that mean you have to train five different models? Serious? I just, I just
told you that a moment ago. That's exactly what it means. Okay, all right. Oh,
oh! And I have a follow-up question. Okay. So which one is the best? So that's
actually the most frequent question we are getting from, from RapidMiner users
as well. So because you built five different models, and they have different
accuracy. So you could think this one here is the best because it's correct
most often. But what you are doing there, if you could just take this model build
on those data points, those 20, you would actually do some overfitting on
this particular test data set and that's also not a good idea. So in general you
would like to train the model on as much data as possible.
So instead of taking one of those five you just build a completely new model on all
twenty five data points and that's your final model. So what this really is
doing is just estimation of the accuracy, but it's not really creating one single
model. You build the model on the complete data. And that's it for today! And this has
been your Five Minutes with Ingo!
Very interesting.