Hey guys, in this video I'm going to go over an example of how you do discriminant analysis in
R. I have the sample code written here and I'll post it online and it'll be in the description
box below. And before I get into the details of how you do discriminant analysis in R,
let me give you some background on how I'm going to use discriminant analysis and what
it is. Usually in the machine learning context, it's used for classification purposes. This
sample dataset, let me go ahead and read it in, we have 14 variables in total. We have
13 numeric predictor variables and we're going to use that to predict one categorical variable that
has three levels. And the levels are one, two, and three. In this case, I'm going to subset the data,
or randomly partition it into training and testing data, and I'm going to build a
model using discriminant analysis that's going to predict what category, whether one, two,
or three, the response variable falls in and then I'm going to test that data using a subset of the
original full dataset. This type of classification problem, if you have two--if you have a response
variable that has two levels, I think people are very familiar with that type of classification
using logistic regression. Discriminant analysis is similar and from what I've read you can use
logistic regression mostly just for two, for a response variable that has two levels and you
can also do discriminant analysis for two levels but I think usually you can just stick with a
logistic regression. Discriminant analysis though you can use it for any number of levels. If you
have three or more levels, you're going to want to use discriminant analysis instead of logistic
regression. And also I've read that if the model assumptions for discriminant analysis hold,
which it has more stringent like model assumptions than logistic regression, but if those assumptions
are true I've read that discriminant analysis is more powerful than logistic regression. But again
my recommendation would just be if you have two levels just do regression but if you have three
or more do discriminant analysis. The first thing I did was get a public dataset using this library
called 'rattle'. This has a couple different data science functions and stuff but I'm just using it
for the public dataset called 'wine'. And let me go to a description of the dataset. This is
a public dataset from the University of California at Irvine's machine learning repository. They have
a website with a bunch of free datasets and a lot of them are already loaded in certain packages in
R. This one has basically a 178 observations of samples of wine taken from three different like
cultivating--cultivars I think they're called in Italy. They have basically three different
vineyards' wines from Italy and they're measuring 13 different values of the wine. And we want to
subset the data and model it and then use that to predict if we can tell based on the 13 variables
where the wine comes from. This part of the code in the beginning is just loading in the
public data. This second line, I'm going to use that for plotting later on. I loaded that in.
And then these two lines just load in and attach the wine dataset. This I've already checked this,
but this is code that you can use to check the class of your variables. Like I said in the intro,
we want our predictor variables to be numeric and then our response variable is the only
categorical variable, so that's fine. And in this next line of code I check the assumptions
for linear discriminant analysis. Discriminant analysis is most commonly done using either
linear discriminant functions or quadratic discriminant functions. LDA stands for linear
discriminant analysis and then you also commonly do QDA, which is quadratic discriminant analysis.
Linear discriminant analysis has a really strict assumption on the distributions of the data,
which says that the covariance-variance matrices for each group have to be equal to each other. You
have to have homoscedasticity across--it's basically you have to have multivariate
homoscedasticity. And this is an assumption just for LDA. There are a couple different ways you
can test that assumption and one way of doing it is shown here. It's called Box's M test. I'm
going to run this code and then check whether the null hypothesis, which is homoscedasticity,
is met or not. This gives me a small p-value, so we reject the null hypothesis that the
covariance-variance matrices are equal to each other. Actually you don't have homoscedasticity,
you have heteroscedasticity and you don't want to use linear discriminant analysis. Now,
Box's M test, I'm using it because it was the only function that I found that worked. The alternative
tests for this are something called Bartlett's test and Levine's test and I think there's also
a non-parametric test as well for this but I tried running it and I think I have too
many predictor variables because it didn't really work when I tried to run it in R. I don't know if
that's a thing that like if you have multiple predictor variables, you should use boxes M
test but that's just how it went when I ran it. But when I've been reading about Box's M test,
I've read that they're basically like a lot of problems with this test and it has a tendency to
reject the null hypothesis. It's kind of like a conservative test and also--it also assumes that
the data is normally distributed, which I don't think is a problem because of our large sample
size. But I've read that it's also very sensitive to non-normality. But anyway, whether or not this
is an accurate test, this model assumption of homoscedasticity for the--for the multivariate
data, it's just an assumption of LDA, it's not an assumption in quadratic discriminant
analysis. Basically, I'm going to trust that this is true and not run LDA. I'm going to try running
a quadratic discriminant analysis instead. It's kind of like just a check. If this were true,
then you could go ahead and try to fit the model using linear discriminant but if it's false,
then I would just run a QDA. In the next part of the code before I run the quadratic discriminant
analysis, I'm going to randomly partition the data. I'm going to train the model
using 75% of the 178 observations and then I'm going to test the model accuracy--well,
and then I'm going to fit the model--and then I'm going to test the model accuracy based on
the 25% test data. To do that, I just got this code online, I didn't even write it myself. I'm
going to get a sample size that's 75% of the original data, which in our case is going to
be 133 observations. Then I'm going to set seed so that we can replicate the partition, and then
I'm going to randomly partition 75% of the data into a variable called 'train' and then 25% of
the data into a variable called 'test'. And then now that the data has been randomly partitioned,
I'm going to fit a quadratic discriminant analysis model to the training data. To do that,
I'm going to load in another package called 'MASS' and this lets you run the code QDA. I'm going to
fit the QDA, which has the categorical response variable on all 13 predictors and then I'm going
to predict the response, or the classification, of the three possible levels using the test data.
And then I'm going to do a kind of confusion matrix of what the actual groups are--one,
two, and three--and what the predicted groups are--one, two, and three. And this is for the test
data. If you can see here, the QDA, the quadratic discriminant analysis, predicts everything almost
100% correctly. There's literally just one error where the model predicted that one sample was
level 1 when actually it was level 2. And then if you wanted to quantify the exact model accuracy,
I wrote this code that will do that and it tells you that this code has a 97.8% accuracy. The model
based on this data can classify the different groups with 97.8% accuracy. And for this last part
of the code, I added in code for showing how to do partition plots. You need a package called 'klaR'
to do that and then it's just this function. And again I think I had too many predictor variables
because it didn't let me load in this really big image using all the predictors, so I just
selected a subset to show you guys what the code does. Basically it will show you--let me wait for
it to load. It will show you a comparison using two of the predictor variables at a time of what
the discriminant functions look like. Basically, the classification separations. And in this case,
it's three colors because we have three groups and the black will show you the correctly specified
samples and the red will show you what's been incorrectly specified. That's really cool. For
example in white, the model is predicting that this is two but as you can see there are some
threes and even a one that's peeking through and then this was predicted to be one and then
this was predicted to be three. That's just for here and it will also give you the error rates.
Now if you can see here, this is--this seems to have a lot of incorrect predictions and I think
it's because I couldn't fit the full number of predictors. I think it's predicting based on the
subset and not the full model that I ran that had a 97% accuracy. But this is just kind of to show
you an example of how you can visualize what the data is doing--what the model is doing with the
data, I mean. All right, so that's pretty much it for this video and thank you guys for watching!
*chiptune music*