[Music]
previously you learned that
classification models predict the
discrete value or values of a response
using one or more predictors
you can think of this as categorizing
some unknown items into a discrete set
of classes
classification is widely applicable
it is used for email filtering
speech and handwriting recognition
medical diagnosis and much more
so how does classification work
in this video you'll learn the basics of
the most popular classification models
suppose you're in new york city
and you want to predict if a given trip
will have a toll or not based on
features like the pickup and drop-off
locations
notice here you're not trying to predict
the value paid for the tool
you just want to categorize a trip as
having a tool or not having a tool
meaning you have two classes in this
problem
you might use labels such as tool and no
tool if you use categorical data
or you could use a logical true false
regardless of how you name the class
labels and pick your variables the goal
with classification is to determine the
class label for an unlabeled test case
in this case you have two classes which
is known as binary classification
but you can also have a more complicated
problem where you want to identify the
source of the tool
problems with three or more classes are
known as multi-class classification
to introduce classification concepts
this video will focus on just binary
classification
a variety of classification models are
available
some of the most common ones are shown
here
you already learned the basics about
decision trees in regression
the main difference in the case of
classification is that the response
variable is now discrete
since possible outcomes are
predetermined from your list of classes
rather than computed based on the data
so let's take a closer look at logistic
regression
k-nearest neighbors and support vector
machines
keep in mind every model works with any
number of predictors
let's start with logistic regression
logistic regression
is analogous to linear regression
in linear regression you find an
equation to predict a continuous
response using predictive variables
in logistic regression the goal is also
to find an equation but now to estimate
a binary response variable such as yes
no
or true or false
all of which can be encoded as 0 or 1.
to do this instead of fitting a line to
the data logistic regression fits an
s-shaped logistic function
also called the sigmoid function
this curve goes from zero to one
and it estimates the probability that
the trip will have a toll based on your
predictor feature
so logistic regression still uses a
formula but one that is a better fit for
a binary
problem as with the case of linear
regression the task is to find the
equation's coefficients
it's important to note that you can
choose a threshold value
if the probability is greater than this
threshold value it's predicted the trip
will have a toll
otherwise it's predicted it will not
it is tempting to assume that the
classification threshold should always
be 0.5
but thresholds are problem dependent and
in many cases you must tune the
threshold value
for example
you might set a high threshold to
classify spam
so that you don't filter out important
emails
you should consider logistic regression
anytime you have a binary response
variable that's what this model is
uniquely built for
also this model works well when you have
fairly well-behaved data
and relationships that are not too
complex
because it's fast to train this model is
great for an initial benchmark
sometimes your data will be too
complicated for a simple mathematical
formula
in cases like this
using a classification model known as
k-nearest neighbors or knn is a good
approach
this classification model assumes that
similar things exist in close proximity
or in other words are near to each other
k n predicts a response by looking at a
given number k of neighboring
observations
to better understand how knn works
consider an example in which you've set
k equal to three
for a new data point the classification
will take into account the point's three
nearest neighbors
here notice two of the data points are
labeled as toll
and one is labeled as no tool
since the majority of the neighbors are
toll
the new data point is classified as toll
this is also known as a majority voting
mechanism
a k n model is different from logistic
regression
and that making a new prediction
requires referencing it to all existing
data rather than running it through a
mathematical equation
therefore k n models can be
computationally expensive for large data
sets
also you need to be mindful of the right
value for k
a value of 1 might lead to predictions
that are less robust to noise or
outliers
larger values of k will produce more
stable predictions due to majority
voting
but eventually a very large value of k
will make less accurate predictions as
it becomes difficult to capture complex
behavior
you'll need to adjust k to find the most
appropriate value for a particular data
set
depending on the value of k
it is common to use the terms fine
medium and coarse when describing knn
classifications
in general the k n classification model
is among the easiest to understand and
interpret
and as you'll see later in this module
it can be quite accurate
k n's main disadvantage is that it
becomes significantly slower as the
volume of data increases
this can make it an impractical choice
in environments where predictions must
be made rapidly
or where there are tight memory
constraints since all the data must be
available when making a prediction
the final type of model covered in this
video is support vector machine or svm
you may remember svm models as being an
option for regression
svm models are also a popular choice for
classification because of their
flexibility
in the binary classification problem
suppose you want to separate the orange
squares representing no toll from the
blue circles representing toll
any line shown on this plot is a viable
option
they would all perfectly separate the
orange squares from the blue circles
but is there an optimal line or decision
boundary
in order to best capture the behavior of
the data the goal is to find the line
that will most accurately classify new
observations into one of the two classes
you would probably want a line that is
evenly spaced between these two classes
and provides a buffer for each class
that's exactly what svm does
the algorithm tries to find a line
that's right in the middle of your two
classes
maximizing the distance between the two
called the margin
to find the line that maximizes the
margin
the svm algorithm first finds the points
closest to the line from both classes
these points are called support vectors
thus the svm algorithm tries to find a
decision boundary in such a way that the
separation between the two classes is as
wide as possible
in this two-dimensional case
that decision boundary corresponds to a
line
but this boundary is generally known as
a hyperplane which is applicable in
higher dimensions
in short a support vector machine is a
classifier that finds an optimal
hyperplane that maximizes the margin
between two classes
in real examples it's usually impossible
to find a hyperplane that perfectly
separates the two classes
a point inside the margin but correctly
classified is called margin error
a point on the wrong side of the
separating boundary is a classification
error
the total error is the sum of the margin
error and the classification error
what happens when the data cannot be
separated by a straight line or
hyperplane as shown here
in these situations you can use a kernel
method
which projects the data into an extra
dimension
instead of a decision line there is now
a decision surface that separates the
points
this concept can be generalized to
higher dimensions
with the kernel method you map data into
a higher dimensional space
where the data is linearly separable
the mathematical function used for the
transformation
is known as the kernel function
and there are different types of kernel
functions
linear is the most common but other
options include polynomial
radial basis function and particularly
gaussian
each of these functions has its own
characteristics and its own expression
the kernel method is a real strength of
svm
as it enables you to handle non-linear
data efficiently
however the kernel function must be
properly chosen
to avoid increasing the training speed
drastically
in this video you learned about some
widely used classification models that
can be tuned to work with any number of
predictor variables
each has its own advantages and
disadvantages in terms of accuracy and
training speed
the only way to know which one works
best on a particular data set is to try
them out and see how each one performs
next you'll learn how to quickly train
these models in matlab using the taxi
data
[Music]