what is going on everybody and welcome to another support vector machine tutorial where we left off we were running the
calculation for every support vector
So we have that now and what we're going to move into is the calculation for the width that
Separates the hyper planes that go through these support vectors, so let's just visualize that real quickly
so you're going to have some vector space and
You're going to have some features, and then real quickly. What are our?
Support vectors, we can look at this and just know right we know for sure
This is [a] support vector, and then we know this is a support vector
And this is a support vector
The reason why we know this [as] these are the points where the two parallel hyperplanes with the most distance between them would be going?
through
But we also know mathematically that what what is the definition of the support vector itself like?
Mathematically, how do we know it's a support vector for sure because this equation is actually
something is a more like a constraint that we
The scientist are imposing on a support vector
That doesn't necessarily tell us that we have a support vector because first of all we don't even know at this stage
What w is and what b is we just simply know that?
We want this equation to equal [zero] and as you'll soon see
We're going to go through a lot of work to make that the case
Based on the value we have for x sub I so
We this is not the equation that we're going to figure out
We're just going to run this equation and somehow it derive what the support vectors are
Instead a support vector is the [let's] [say] feature set that
If you were to move it like let's say we took this plus this
Bottom plus, and moved it to here would that affect this hyperplane?
absolutely right no longer might that be the best separating hyperplane it might be something else and
So so we we call this a support vector because if we moved it it would affect the rest it would affect the best
separation so we know
what our support vectors are and then we have our nice beautiful hyperplane and
The objective though if we go all the [way] back to that first video was to actually come up with the best
separating hyperplane between the Data sets so that
separating hyper Hyper plane can be calculated
simply by taking this right here right the the width
and
Then you could take [that] width divide it by two
And then basically from here you could just plus that width and you would find you would get yourself to that hyperplane
So we know that we want to maximize that width and now what we're going to talk about is
how we
How we do that?
So the equation is for width
equals x plus
minus x negative and these are the vectors
Dotted with vector w over the magnitude of Vector w?
So again, what are we wanting to do here? We want to maximize
this equation and
Then what are the basically the variables in this equation? Well we've got of course now here. Here here here
But we actually already know
What this and this are equal to?
Because in the previous tutorial we worked out what that what the equation is
for those values
Right for x minus and x plus this [is] the equation that we're using
So pulling that down
we get right remember that's y sub i x I
W plus B?
Minus 1 equals 0 and then it was actually the same thing here right both of [those] equations
and if we were to do the Algebra in this example
we would find that the x plus is actually it would come down to being 1 minus B, and
then we'd Have 1
plus B for the for the x minus and
[then] again if we worked out all the Algebra [we'd] find out that this entire width equation comes down to being 2
vector the Norm Magnitude of Vector w
width
Is equal to 2 over the magnitude of w?
and
We want to maximize
This so we know we want to minimize that
And then we just simply rewrite we say yeah, we want [to] minimize
the magnitude of Vector W and then for just simple mathematical convenience
We're going to say
Well, yeah
We want to we [want] to minimize vector w or we want to minimize the magnitude of vector?
W that means we also would [want] to mat minimize?
[one-Half] the
magnitude of Vector w
Squared right so long as we want to minimize
Just simply the magnitude of vector w we'd also want to minimize
half the magnitude of
Vector w squared right that works so we do that for mathematic convenience that we're about to plug it into an equation
And that's just going to make it a little easier for us
So we want to minimize
[one-Half] the Magnitude of Vector W
squared
So we want to minimize [one-half] of the magnitude of Vector w?
Squared with constraint to our support vectors which we have defined as being y?
sub I
Times x sub I by w plus b
minus 1
That is our
constraint ah
Someone said the C word
There's a knock at [the] door. [it's] L'orange he brought his multipliers and together we're going to work on this problem
So then we're gonna have our lagrangian, and it's going to state that the L
W
[B]
Equals what we already decided was going to be our statement for the magnitude of [w] and [that] was going to be?
one-Half the Magnitude of Vector W
squared minus
the
sum over
Alpha I
Where alpha I is our lagrangian multiplier
And then the constraint we have is the constraint that we define as being our support vectors, so that was
Y
sub I
multiplied by x sub I
w+ b
[-1] and that is our initial statement here that we need to optimize
and so again
We want to minimize
W and we actually want to maximize?
B, so why do we want to maximize?
B and what exactly is B?
Well if we look at the really simple equation for a hyperplane so the equation [for] a hyperplane is
just w x
Plus B. What's that look like that looks a lot like Mx plus b
Right for the simple line and what happened when you modified B
well, what even was be right what b was you're just that was just [your] y-intercept and if you modified where the line intercept to
The y that would just move it up or [down] the same is basically true for a hyperplane
So to exemplify this let's draw some vector space
and
Then let's add a couple [hyperplanes] actually we'll start. Just by adding a single hyperplane
so this is a hyperplane right it's all filled in and
Then what if you were to add b. Well if we added b. Let's Say B
This is b. We're adding B moves you up, so then if you had another hyperplane you'd have this
and
What you've done here is you've added to B
So you've just you've basically shifted this hyperplane and b in terms of the support vector machine is
Generally going to be called bias. That's kind of why we're actually using b here
So heading back to our lagrangian here. We need to differentiate
L with [respect] to W and we also need to differentiate L with [respect] to B, so
differentiate L with respect
To w and we also need to differentiate L with respect to B
Okay, so when you did let's say first, we'll do differentiate
L with respect
to w
You're going to find that you've got basically it's going to come down to being Vector w
minus the sum over Alpha I
y I
x I
Which then simply becomes?
When you move this [w] basically, but we'll just add the equal sign here, so it simply becomes
vector w equals the sum over alpha iyi x I and
I'll go ahead and make x I official okay, so then
We can differentiate
L with [respect] to B now so we can say differentiate L with respect
to whoops
differential L with respect to B and
What we get there is going to be minus the sum over alpha I
y I
And you would say that is equal to zero
But then you can actually just you would come up [with] L with respect to B
Is the sum [over] alpha I y i then what we come up is L
equals the sum I
over Alpha, I minus
one
half of
The double sum of I and J
over
Alpha I alpha
j y I
y j dotted
with vector x I dot x
J and
We want to maximize
This so we know we still have this constraint optimization problem, but right away
We should recognize this alpha by alpha this alpha Squared
when you see that we recognize uh-oh we had of a
quadratic
programming problem
So understandably some of you might be a bit fuzzy at this point
If that's you that's totally fine, and honestly should be expected if you're not lost or confused and this is your first time
Learning about the Svm
You're a robot
So what I've shown so far is the more formal and traditional
calculation for the support vector machines
Optimization along with some of the facts and assertions that an Svm just just has so it's important to at least be
Familiar with this stuff as you're going to see these formulas if you continue working with Svm's also
You need to know them if you're going to work with an optimization library, and you definitely need to know
How the Svm works if you intend to you know make your own like we do in the series it is also important
I think to cover the more formal calculation of an Svm or the typical
Calculation because I think it really highlights some of the downsides and upsides to the Svm
so one of the downsides is the just
complexity both in Math, but in also the
optimization problem itself
The other major downside of the Svm. Is that you need all of your feature sets in memory to optimize
So if you have a really large data set this might not be feasible like like a data set
That's a hundred gigabytes or something that's not
outrageous okay, so I mean that's a huge day set but I mean it's not it's not too crazy in this world and
The fact the chances of having 100 gigabytes on your typical machine. I mean you might have a big giant machine, but
Probably not so that's gonna be pretty hard now of course this was a problem with K nearest neighbors as well
but this is one of the major downsides to the SVM now there are methods to
[kind] of work with small or larger data sets and you can do it with like mini Batches, basically
Probably the most popular choice nowadays is sequential minimal optimization or Sm
Oh, which was created by John platt actually quite a while ago at Microsoft, but that's probably now
The most common used one if I was to make a guess, but anyways either way the SvM can be pretty cumbersome it
Now the upside though is once you have trained a support vector machine you do not need
the you don't need the feet the old features anymore and a classification for [a] new data set is just
W x plus B and not even just w x plus B
it's the sign of w explodes me, right is and not not sign as
In cosine sine and sine as in is it a positive or negative if it's positive
It's a plus class and if it's a negative it's a negative anyway in the next video
I'm going to hopefully clear up some of the fuzziness if you have it here by
Dramatically simplifying our problem which we can do since we have a pretty decent foundation
I [think] now of the support vector machine and its constraints and then
Hopefully, I'll [be] able to
Get you completely clear when we go to write our own
Support vector machine from Scratch in Python because We are actually going to write our own optimization?
Algorithm in python and I think once you do that and you?
Recreate the support vector machine and do the prediction the training and all that
I think that's going to help tremendously at least for me. I was pretty fuzzy until I actually tried to do it myself
So hopefully it will help you as much as as it
did me
So anyway
I kind of suggest you hold your questions until at least the next video when we kind of do it basically a recap and
Simplify the math that we've seen so far
But you [can] still leave one below if you want otherwise you can of course leave comments below and as always
Thank you [for] watching