Hey guys!
Hey!
Hi Ingo, hey Simon!
Simon Fischer, our SVP of Engineering.
Great to have you here.
Hey Graham, great to be in Cambridge!
Nice, great!
How fun.
Uh, Ingo, you know what time it is.
Time for...
Five Minutes with Ingo.
All right.
So, you remember like two weeks ago we talked about cross validation.
Yes, yup.
We used those building blocks like representing data points or even here like groups of data,
and we have seen how you can distribute data into training data and testing data, make
a split, and then you can build the model on the training data and then apply the model
on the testing data, so you want to make sure that you are never, well, applying the model
on training data, data you have already used to, that you've been using to train the model.
So, this leads to a fair estimation, that's great, and Simon and I actually have been
working on a cross validation anyway in RapidMiner so we thought that let's take this from here.
Cool.
So here is an operator very quickly going in and it's almost complete.
You see we built a decision tree here in the training phase, and then the second part of
the data, not the same part, it's a different part of data.
And we now apply the model.
And here lets use this recommended operator down here, drag it in, and then connect it
and now the process is finished.
OK.
So I can go up again, everything is good, and we can execute it now.
Alright, so that's the result.
Oh, cool.
Yes, exactly, so that's the result, and what you see here is now the average over multiple
of the iterations, ten of the iterations and cross validation.
The operator that Ingo just dragged into performance operator, measured the performance once and
here we see the average, or the aggregate over all of the ten iterations.
Let's just look at one, or a couple of those numbers here.
So, in the columns what you see are the true values that are in the data, where as on the
rows we have what RapidMiner predicted.
So for instance, if we look at the middle-center row here, there were 31 cases where the truth
was iris-versicolor and four where the truth was iris-virginica.
And in all of those cases in this row, RapidMiner predicted versicolor.
So that means almost always was RapidMiner correct.
In 31 cases, and in four it was incorrect.
So that's a total of 35 where RapidMiner decided versicolor, and the ratio in which it was
correct is here on the far right, that's 88.57%.
And you get lots more insight here in the other rows for the other classes, and in the
columns you get the invert.
So there you get the ratio, the so-called re-core, so the items that RapidMiner finds
where iris-versicolor would have been correct.
Of those 33, RapidMiner found 31.
So you see this basically: for every single class, you see this recall for the three different
class, also the precision - how often it is correct, which is great.
And then the overall accuracy - how well this model is working - is number one on the top
left here correct?
Exactly, that's the average over all the iterations.
So a pretty good model actually with 94%.
Yeah that's pretty good, yeah.
Do you remember actually the times when we had to be coding this whole thing?
Some time ago, that was tough.
It was pretty tough Graham, I tell you.
I think that years ago, cross-validation-.
It was one of the first things we built.
It was actually, yeah really, besides a couple of machine learning methods, one of the first
things.
You mean, instead of using the GUI you're showing me here, building this whole thing
out using code.
Well yeah, we needed to program this because otherwise you couldn't use it as part of the
Graphical User Interface.
Right, of course.
And actually, I just changed back here to the process.
So if you pick this one here, and you see all the different functionality like the number
of validations, which is really the number of different test data sets, data sets you
use.
So all those different parameters, the assembly type, that's so much work because you need
all those different settings.
The problem is really we need to make all this work and coding isn't really that fun,
it's a hard time.
Yeah, so many rules.
Yeah right, it's like a fight, coding.
Rules... hmm.
Gentlemen... welcome to Code Club.
The first rule of Code Club is: you do not talk about code club.
Second rule of code club: you do not talk about code club.
Third rule of code club: if someone checks in, breaks a task, and goes for coffee, the
hack is over.
Fourth rule: only two keys to a joint.
Fifth rule: one line at a time fellas.
Sixth rule: no suits, no ties.
Seventh rule: debugging will go on as long as it has to.
And the eighth and final rule - What?
If this is your first night at Code Club, you have to code.
Well, uh, yeah, anyway so, yeah that was also a fun time modelling.
It was of course it was fun.
Yeah well for us fun.
I think it was not for everyone.
Well no actually I'm happy that other people can actually just use that one here.
Anyway, I would say that's what we can say about cross validation and how this looks
in practice.
It's actually pretty, pretty nice if you see this, if you see this separation between training
and test data, so you can see the distribution you can have a look into every single example
to figure out where, like, the predictive model is spot on and where it's wrong, so
the overview table Simon just described.
But, overall it's really a technique everyone should know to estimate the performance accuracy
of a model.
So, try it out.
Cool.
And this has been your Five Minutes with Ingo.
Data Scientist Number 7.
Thanks.