(Complaining about Ingo) Where is Ingo?
Where - Oh my...
All right.
Ingo, I know it's you.
What are you doing?
You know how difficult it is to be a data scientist?
So you built a fort to sit in?
Sure everybody is coming to you and asking you all these questions you need to answer
and asking you to predict the future and use all the big data and everything.
And you're the only person who can solve that!
That's all my life is.
So from time to time I need to hide and get my work done.
No way!
Fair enough, look I was thinking, now that you're in a box fort, we could take five minutes
and do Five Minutes with Ingo!
Well sure, yeah, but don't tell the others.
OK, so remember last time we talked about over fitting?
And, and over fitting really is about this trade-off between complexity and correctness.
So you would like to build a model to predict data points, or the future in general, as
accurate as possible.
But at the same time you don't want to create a model that is unnecessarily complex, because
that tends to be less good for unseen data points, for new situations, and that's exactly
why we're doing this whole thing.
We want to be as good as possible for new situations.
So we have this trade-off and because of also over fitting control.
And this comes actually with another interesting question, and that is very important for you
to detect over fitting.
For the models you are building.
And this question is: How do we actually know how well this model is working for new and
unseen data points, because we've never seen them before.
So you need to come up with an estimation of the model prediction accuracy.
And how can you do this, because, well, the model is predicting the future so we never
have seen the future before.
So how can you calculate, how well the model is working?
So, that's the topic for today.
Mmhmm.
That sounds good.
Ok, so, let's go back to K Nearest Neighbor.
We touched upon this before, so we had those two classes of data points here, the blue
class and the red class, and you know by now how K Nearest Neighbors works.
If I ask you for example for a data point like here, the answer is - what class is this,
blue or red?
Blue.
It should be, because it's closer to the blue points.
So, that is easy now, but how can we, in general, say that where ever a data point is here,
or in general understand that first.
Wherever a data point is, we can make a decision, if it's a blue point or a red point.
But we can't make the calculation for all data points because they are all basically
an infinite amount.
So that's not working.
So, what other ideas do we have?
We could, for example, use the same data points I've used for building the model also for
calculating how well this model works.
But that's really risky and I don't recommend at all to do this.
I see this actually happening in practice all the time.
We call this recalculate the training error.
If you ask me, never do that.
It's, it's always leading to the wrong decision.
And here's why.
Let's simplify this a little bit, down to only two points.
So if the K Nearest Neighbor model now, for those two data points, so it's pretty clear
that everything below this line is red and everything above this line is blue.
Yup.
So now for calculating the model accuracy if we just use those two data points, and
only those two data points, well, what is the class of this data point?
Red.
That seems to be correct.
And this data point is blue, that was also very easy.
So now we say like 'OK, in 100% of all cases our prediction is correct.
Our training error is zero, because we are correct in all of these.
Every time!
Exactly.
Yeah!
But that is true for every one nearest neighbor model.
So if you only look at the closest neighbor and you do this on the training data itself,
then we always will end up with a 100% correct model.
So we think 'oh yeah, cool!
The model is perfect, it's going to work like a charm.'
But reality is, if you have a new and unseen data point, we don't really know what the
decision is.
We don't know how often we are correct and how often we are not correct because we have
no clue what color those data points are, which are further away or, well, not exactly
on the decision boundary here.
So, using the training error is really really a bad idea.
So what else can we do?
So I will show you one thing.
So usually we have a table of data points, so we have a couple of columns here: C1 to
CN.
And then here we have one specific column, we call this often the label or the target
variable, we want to predict.
So we take all of the values here in the first couple of columns, create a predictive model
to predict the value of this last column.
So whenever you have a situation like that, and it's pretty common, you can really separate
the data into two parts.
And we say, OK we have training part here, and we have the test part here.
So we use all those data points to train the model and use the second half to actually
calculate how this model is working.
I like it a lot!
And that's much better now, because now we are no longer using the same data points we
used for training as for testing, so we're not running into this risk.
By the way, a good way to detect over fitting often is: if the training error is much smaller
than the testing error, then you're often in an over fitting case.
So that's for this week, but next time I'm also talking a little bit about cross validation
which is actually even a better technique to come up with those model accuracy estimations.
So I think really I need to get back to my work now.
All right.
And this has been your Five Minutes with Ingo.