Hey guys, in this video I wanted to go over one way of doing variable selection for regression
models in R. And specifically I'm going to be talking about the branch and bound technique,
which is one application that's found in the leaps package in R. Before, I go into my little
demo, the leaps package is named after this paper, which was from the 70s from these forest
guys, and they talked about regression by leaps and bounds.
The branch and bound algorithm comes from computer science and for the first time in
the 70s it was applied to regression variable selection.
They start talking about it on page 74.
And without going into too much of the theoretical details about what it does is, with variable
selection in regression, you might have several possible predictors that you want to add to
your model.
And you want to pick the best subset and if you have even a moderate amount, or
number of, predictors, the computation time of testing every possible model can be very
big.
So if you think about all the possible models as like leaves on a tree, what branch and
bound does is it basically prunes some of the leaves off the tree and it makes it
computationally easier to find the best subset of regression.
So this video is going to focus mostly just on the application of how you would do this
in R and again the only package you need to install is the leaps package.
This second package that I have here, the faraway package, is for sample data.
This gives me a sample dataset and if you're interested in what type of datasets it has,
the CRAN package has this webpage that has all the different datasets.
So I was just looking around and picking one that had lots of variables so that we could
do a demo of how to do variable selection.
So these are all different datasets with different numbers of observations and variables.
So I picked this one called "fat" and if you do like question and then the name of the
dataset, in this window here, it will give you a description found from the CRAN package
website about what the data is.
This is percentage of body fat and body measures.
It has two different measures of body fat and then a bunch of different measurements
taken for 252 men it says.
So this has 18 variables and so there are 2^18 different possible models that you could
fit a regression on.
Now let me show you how you can use the branch and bound technique to pick the best possible
subset of those predictors.
If you read into the dataset, there are two measures of body fat.
I only used the first one, the brozek or however you pronounce it.
To run the model in R, you're going to set your x as all the predictors.
You're going to do it as model dot matrix.
And then this was the y that I picked.
So y regressing on all the predictors.
The models, you have to take the intercept out, so by doing -1 you take out the intercept.
And because I don't want this other possible body fat predictor, or response, I removed
that because I thought of it as basically like a second possible response in the model.
I'm considering basically a model that has the brozek body fat as the response and then
all the other predictors as possible subsets for the model minus the siri body fat.
And then you just write that the data is equal to fat to reference back to the data set that
we're using.
We're going to next set this variable y, which has the response.
It's the brozek.
And then we're going to fit the x and y using the leaps function.
So bestmods is equal to leap(x,y) and then the number of best is just going to be--we
want to just have one best model.
The output will give you, depending on how many predictors you input as your x, it gives
you the best possible subset for all the numbers of predictors that you can consider.
We have 16 possible predictors, or x's.
In this case, the output is going to give me 16 possible different best models based
on how many predictors you want.
So the first line, it gives me the best possible model if I only want one predictor.
And the true or falses tell you which predictor you want to include in that best model.
In this case, if I only wanted one predictor, it says that the best predictor is number
one by calling it "true" and the rest of them are false.
In contrast, if I wanted a best possible model with 8 predictors out of the 16, then it would
be the first predictor, the third, the fifth, and so on.
Then at the bottom, you have these values.
So this is picked--this automatically picks it based on Mallow's Cp, so the idea with
that--you can also use other information criterion, or criteria to pick your best model out of
all the possible subsets for each number of predictors, but it will automatically use
Mallow's Cp.
With these, with AIC or BIC or Mallow's Cp, the best model will minimize these values.
In this case, it will give you the Mallow's Cp values for all the 16 different possible
values.
If you want to know which model is the best out of all these possible models, then you
just have to pick the one with the minimum Cp.
In this case, the minimum Cp, if you look for it, is 4.482491.
So it tells me here that this is model number 9.
So out of the 16 possible predictors, based on Mallow's Cp, using the leaps and bounds
or branch and bounds variable selection technique, the model that you want to pick is the one
that has 9 predictors and the 9 predictors are given here in the model.
You can go back and look at what variables or predictors those are and then, re-run your
regression using those 9 predictor variables.
And so this is one application of the leaps package, and one way of doing variable selection.
There are other possible popular methods doing variable selection for regression, like forward
selection or backward elimination, or just stepwise regression in general and also the
information criteria that you use can also give you different results.
So it's not that there's only one right answer for how many variables to include, it kind
of depends on how many predictors you want to limit yourself to and how much complexity
or bias you will allow in your model.
*chiptune music*