(Magic by the Cars playing) What?
So building models really is somewhat complex.
Oh yes.
One does not simply train a model in Mordor.
Yeah that's right.
Let's train you actually so you can be more efficient.
What do you think?
Very good.
All right, all right.
Hey guys, hi Gandalf how are you?
Welcome to headquarters.
We're glad you're here for training.
Ingo, hate to bother you but it's that time again.
For Five Minutes with Ingo?
For Five Minutes with Ingo.
Excellent, let's do that.
OK, did you actually see Lord of the Rings?
Oh yeah absolutely.
Do you remember the scene down in the Mines of Moria of this stone bridge over Khazad-dûm?
Oh wait, do you mean this scene?
Yeah, that scene.
So, we have a bridge here.
Oh, how convenient.
Right.
So where they have been running on the bridge?
In the scene they were running down the middle of the bridge.
Down the middle of the bridge.
Yup.
Why not on the edge?
Because it's dangerous, it's not safe.
Right, why is it dangerous?
Because it's just one misstep and actually falling down to Khazad-dûm.
Right, right there's not enough room.
So you run in the middle because you have safety space, and that's a good idea.
Now let's talk about Support Vector Machines.
OK.
Support Vector Machines actually take the same idea of safety space really.
So look at those data points, blue and red like always.
I asked you 'can you take some stick or something like, like a ruler and show me where you put
a separating line?'
Oh yeah uh um, Gandalf, can I use the wand please?
Oh my god, OK, whatever.
So, you do this one.
Oh, yeah.
OK, sure, I'll do it.
So, there.
Right in the middle?
Wherever you would like to place it.
Right in the middle.
Yup, that looks good.
Like we talked about with the bridge.
But why?
This one here is equally correct, and this one here is correct as well, and this one's
correct, and this one's correct so we'd be using the same concept, we need the safety
space.
So the idea here is really, we don't know where other data points are.
We only know these here.
But we believe that other data points are following the same distribution so it's very
likely that there might be another red data point here, and if you go over the model like
that one here, actually we might have a higher risk for making an error, like we have a higher
risk for falling down.
OK, so that's, that's the basic idea of Support Vector Machines, to keep as much safety space
as possible.
Got it.
We also call this safety space, by the way, a margin.
So, if you would like to optimize this margin, there's a little bit of formulas here, you
can actually say 'we would really like to minimize the complexity' and we've seen this
a couple of episodes before, that it's always a better idea to go with a simpler model,
so we can actually minimize this function.
And then at the same time we can also calculate how often do we do errors.
So, let's have another look here.
So for example, if one of those blue data points is here, it might be a good idea to
say like 'yeah, you know what?
Just accept this error here, we do some mistake here.
But who cares, we just keep the hyper plane where it is because it's better to accept
this one error.'
But then we need to sum them all up.
And this parameter 'C' here is a trade off between this risk, on these data points, and
the complexity.
So then there's a lot of math happening, a lot of magic is happening here in this place.
Haha, magic.
Nice.
So, and then from there you are ending up somewhere here.
And by the way, if you're interested to see a full video on this topic, on SVMs, you can
just click here.
So, this is the result, and that looks maybe a little bit complex in the beginning, but
it's rather simple.
So we have a couple of sums here, and this sum is very interesting because you just sum
up all the dot products between all the combination of data points, so we've all the combinations
between Xi and Xj, two data points in this data set you have.
So in a dot product linear space, really it's calculating like a similarity.
And that's something I mentioned before, that machine learning often is about optimization
- so we are maximizing something here - and similarities.
So that's happening here as well.
When you multiply with the actual labels, and then there are those weighting factors
here, you could also call them Lagrange multipliers, and they say how important is the data point
for this formula, for this model here.
And those Lagrange multipliers, they are somewhere between zero, and they have an upper bound
of 'C', and that's actually this trade-off factor.
So if you say like errors are really a bad thing, then we can give example of higher
rates.
So we have a higher upper bound.
So, two more things which are interesting on SVMs: first thing is, it's very efficient.
So to optimize this function, typically you are ending up with a cubic run time.
So, if 'n' is the number of data points you have, that means that really the run time
is to the power of 3, but it's linear in the number of dimensions.
And that's pretty good actually, adding additional dimensions to your data space is not very
costly.
Mmhmm.
And the last thing is a dot product is a linear thing, but you can actually instead of use
the dot product here directly, you can replace it with a so-called 'kernel function', and
such a kernel function is a dot product in a higher dimensional space.
And you get into the space by taking all those data points, and just transforming them into
data points of this new space.
And that means, if in the regular space, you have something like this here, they're actually
not linearly separable, you can transform those data points into a new space and then
in this new space, you can clearly see, you can separate them by linear function again.
All right, SVMs are pretty cool.
They're cool because they're using this maximum margin idea.
This means you have less over fitting, that's great.
They're doing this in a very efficient way, so it's very fast to calculate them.
They even, like linear and other dimensions.
And another linear, type of linear you can use them for linear problem, specification
problems, and also non-linear problems.
So that's pretty great.
OK.
Did you have like a question or something?
Uh, oh right, real quick, sorry.
Why are they called Support Vector Machines?
Excellent question, as always.
OK, so you've seen those weighted feign factors for every example alpha?
So, what happens if one of those factors is actually zero?
That means this example's not important because this whole part here is becoming zero.
So all of a sudden, this data point doesn't matter any longer.
And that happens actually for most of the data points - we actually say that it's a
sparse algorithm - and only those data points which are very close to the separating hyper
plane here, they are actually important to define the position of the type of plane.
But you can also say they are supporting the location, the position, they are support vectors.
That's why we call this Support Vector Machines.
Fascinating.
And this has been your Five Minutes with Ingo!
What?!
You're a true magician.