we are going to use inbuilt data Irish which has 150 observations
five variables four variables are numeric and variable species is a factor
variable with three levels so using these numeric variables we want
to build a classification model that will help us introducting correct
species so let's use a scatter plot from ggplot2 package
so let's plot petal length and petal width
this is the scatterplot we get so let's color code and see whether there is any
possibility of separation so I will put comma hit enter
color.this by species so you can see now there is a separation based on the color
red ones is the first species and clearly it is very far away from the
other two groups between blue and green there is some small overlap
let's first get a quick overview of what is support vector machines so these are
data which are color coded represent two different classes so basically we are
looking for optimal separating hyperplane between these two classes and
that we achieve by maximizing this margin around this hyperplane once that
lie on this boundary are called support vectors and this middle line is
our separating hyperplane so in situations where we are not able to
obtain a linear separator data points are then projected into higher
dimensional space so that data points can become linearly separable for this
projection we make use of kernels so a program that helps us in performing
these activities is called support vector machine
we will use a library which is e1 0 7 1 which is commonly used for support
vector machine so we will develop a support vector machine model using SVM
we want to predict species in this data set and then we will use all other
variables so remember we have 4 variables which are numeric so by
putting a dot we can make use of all those variables our data is Irish and
let's store this in my model
so the modern created so let us look at a summary
so it gives it was used and the data that was used
and then there are some parameters SVM type is C which is for classification so
since we are dealing with species which is a factor variable classification
becomes the default type if our response variable was quantitative or continuous
variable in that case we would have done regression but right now it is
classification so colonel use walls radial basis function cost is cost of
constraint violation and default is 1 so that's why this is 1 here and then we
have another parameter gamma which is 0.25 and number of support vectors is 51
it belonging to the first type of species 22 belonging to second type and
21 belonging to the third type and these are the three classes for species so
let's plot the results and see what we get
and since we have four variables we will have to provide some extra information
in this plot if we had only two variables like petal length and petal
width then this would have been sufficient but since we have four so
let's specify few other things so let's say we want this plot for beetle width
versus little length so petal width on y-axis
and petal dot length on x-axis and for the other two quantitative variables
let's make use of slice
so we keep possible but with constant at three and simple
dot length constant at four and we run this
to get the plot so this visualization shows us the data support vectors which
are represented by this cross and also decision boundaries that you can see
here for the three types of species these data points belong to the first
species and these data points belong to the second one and this one belongs to
the third type of species whereas predicted class regions are provided by
the colors in the background so you can see this color here for the first
species and this color represents predicted class for the second species
and this pink color represents predicted class or the third species we had 51
support vectors and there are eight of them belonging to the first species
so you can see there are eight crosses here one two three four five six seven
and eight so these are the support vectors for the
first species similarly you have these red crosses representing support vectors
for the second species and green crosses represent support vectors for the third
category
so let's predict each species using our model called my model that we created
and we'll make the prediction for the entire dataset in Irish and let's store
this in PR Edie and then we create a table
PR IDI versus actual
so let's all this as predicted and this one as actual
and we can store this table in tab
so this gives us our matrix so out of the total there were 50
data points belonging to the first species and the model also predicted
that these 50 belong to the first species similarly there were 48 data
points belonging to second category and model also correctly predicted this so
this is a correct prediction and similarly there were 48 that belong to
the third category and they were correctly predicted by the model two
data points belong to this third category but the model predicted them to
be from the second category so this is a Mis classification similarly there were
two data points that belong to second category but the model predicted them to
be in the third category so this is another miss classification so if we add
the 50 plus 48 plus 48 and divide by 150 we get accuracy or if we add this miss
classify data points 2 and 2 which is 4 and divide 4 by 150 we get miss
classification rate we can calculate sum of the diagonal items in the tab divided
by sum of the tab and we do 1 minus 2 get miss classification rate
is about 2.7 percent note that this model used radial basis function as a
kernel so other than radial basis function we
can use linear so if you run this and look at the summary of the model so now
you can see kernel has changed to linear and number of support vectors are now 29
based on linear kernel and these are the support vectors in these three
categories
part of the model for these two variables you can see there is a slight
change the way these are classified so this was with radial basis function
and this one is with linear similarly if you calculate conclusion matrix and this
classification error this is what we get so you can see the slightly higher miss
classification so this classification rate has
increased 23.3% similarly if we try polynomial
so you can say we have polynomial kernel now we have 54 support vectors 6 26 and
22 in each of these 3 classes if you look at the plot so we can see plot
looks slightly different compared to earlier ones
and let's do the confusion matrix and miss classification miss classification
has now increased to seven and all of them are misclassifying this third
category as second and error has gone up to 4.7 percent
try sigmoid condom so with sigmoid we have 54 support
vectors and 626 and 22 in each of the three classes
is much higher now in fact seven plus nine plus one so seventeen
misclassifications and error rate is about eleven point three percent for
this data set obviously performance by sigmoid kernel is the worst where as
radial basis function was the best let's see if we can fine tune our model to get
better classification
so tuning is also called hyper parameter optimization and it helps to select the
best model let's set a random feed at one two three
so we will make use of tool function for doing the tuning and we will use support
vector machine model species as function of all other four
variables and our data is still Irish we'll make use of ranges
so we'll make use of a sequence which
starts with zero and goes up to one with the increment of 0.1 so basically values
for epsilon will be 0.1 0.2 0.3 0.4 up to 1 so we will have 11 different values
another parameter that we want to use for fine-tuning is cost
so cause captures the cost of constraint violation and default value for cost is
one it cost is too high it will mean high penalty for non-separable points
and what may happen is the model may store too many support vectors and that
will lead to overfitting whereas if the cost value is too small we may end up
with under fitting and model may not be very accurate so that's why we make use
of a large range so that we are able to capture optimal cost value so that's why
we make use of 2 to the power 2 up to 9 so 2 to the power of 2 is 4 and then 2
to the power of 3 which is 8 and it goes up to 2 to the power of 9 which is 512
basically this gives us eight different cost values and here we have 11
different epsilon values so 11 times 8 is 88 so we'll have 88 different
combinations so when we are giving this range we have to be careful with large
data sets if there are too many combinations it will take a very long
time to really come out with the optimal solution so since we have only 150 we
are using 88 different combinations we should be fine so let's store this in T
Model T for tuning or tool model so we run this once we successfully run this
we can plot T model
this gives us performance evaluation of svm for these two parameters that we had
used and darker regions means better results are in these regions which means
lower values for cost and various different values for epsilon so in fact
instead of going from 2 to the power of 2 which is 4 up to 512 even if we do our
search within maybe less than 200 we should be fine so instead of 9 if I do 7
here and rerun the model and then look at the plot so you can see
again lower values are better so let us look at summary of this T model
so the fam method used was 10-fold cross-validation
best parameters epsilon zero and cost around 8 which is 2 ^ 3 you have the
results for best performance and then we have detailed results
so using these results we can choose our best model
so within t model you have
at model and let's store this in my model
to look at summary type is classification because our Y
variable is a factor variable and the best kernel is radial so one thing to
notice is generally for classification purposes radial basis function is the
best cost eight gamma parameter is about 0.25 and there are 35 support vectors
six 15 and three for each of these three classes
we can also plot this as we had done earlier
you
if we run this
what we get
so let's look at the confusion matrix and this classification error so you can
see out of 150 data points only two are misclassified and this classification
error is now 1.3 percent so this is significantly less than what we had
earlier