Hey everyone, in this video I wanted to talk about logistic regression and kind of go over
the basics of how logistic regression works. I'm going to start by comparing it to a typical linear
model which would be multiple linear regression or simple linear regression, which is when you have
a variable Y which is your outcome variable and you regress Y on one or more predictor variables.
I'm just going to say as an example X1 X2 X3. This could be any number of predictor variables.
In this type of linear model, X1 X2 X3 and Y are all continuous random variables. That means
that the values of Y and all the Xs can be any number. These values when they're continuous
can exist anywhere from negative infinity to infinity. And so when you have a linear model
where your outcome variable Y is continuous and your predictor variables are continuous, the
model--the linear regression model--is basically that Y is equal to a linear combination of
the predictor variables. You're going to have in this example a model like this
where Y is equal to a linear combination of the Xs and because the Xs are continuous, this can exist
anywhere from negative infinity to infinity on the right-hand side and on the left-hand side a linear
combination of terms that can exist from negative infinity to infinity also has the same domain. It
can also exist from negative infinity to infinity. When you're doing logistic regression, you're
going to have a slightly different type of model. The idea with logistic regression is that instead
of having a continuous random variable Y, the outcome variable is now going to be categorical.
Theoretically for a logistic regression, there's such a thing as multinomial regression where Y
could be any number of levels but usually when we're talking about the basics of logistic
regression we're just going to consider Y as a binomial random variable. Y can have two outcomes.
This could be any categorical variable with any levels. For example, it could be gender where the
outcome could be male, female. That's one possible outcome variable. The outcome could be yes, no;
it could be disease, no disease. I'm just giving examples of categorical variables that could be
your outcome variable Y. And mathematically when we do a logistic regression model,
we're just going to model the categorical variable as 0, 1. The number 0 is going to refer to some
level and the number 1 is going to refer to the other level. And so this is how you would write
out mathematically a binomial random variable. We see from the linear model that we had before,
on our right-hand side the predictor variables are still going to be continuous. On the right-hand
side, these values can still exist anywhere from negative infinity to infinity. But when
we change the outcome variable to only be equal to either 0 or 1, now we have a problem where
the left-hand side does not equal the domain of the right-hand side. We can't just use a linear
regression model to predict an outcome variable that's categorical and this is basically where the
idea of logistic regression comes in. We have to transform the left-hand side, which is the
outcome variable, to have the range of negative infinity to infinity. The way logistic regression
solves this problem is that instead of predicting Y as 0 or 1, we're going to predict on the
left-hand side the probability that Y is equal to 1. Since Y can take on either of the values
0 or 1, we're going to say we want to model the probability that Y is equal to 1 and again we're
going to regress this on the predictor variables. I'm going to label the probability as lowercase p
and the probability of any--any probability can be between the values of 0 and 1 inclusive. Now
instead of just having two possible values on the left-hand side, now we have an infinite
number of possible values that can range from 0 to 1. But this still doesn't completely solve
the problem because we can't go less than 0 or more than 1, whereas on the right-hand side we
still can be anywhere from negative infinity to infinity. We still have to do a transformation
on the probability that Y is equal to 1 to get it to a range of negative infinity to infinity.
The way logistic regression does that is the left-hand side is going to be equal to the log
of p over 1 minus p and so this is known as the log odds of the probability that Y is equal to 1.
A probability over 1 minus the probability, that's known as the odds. That's the equation for an
odds. We're taking the log odds of p, which is the probability that Y is equal to 1. And this is the
model that's known as logistic regression. This is your left-hand side and then that's equal to
a linear combination of the predictor variables. And this equation is known as logistic regression.
This transformation that we do on p, the probability that Y is equal to 1,
we do it basically now because the log odds of the probability that Y is equal to 1 does have
the desired range, which is from negative infinity to infinity. Basically when we graph out what the
logistic model looks like, a logistic regression basically models an S-shaped curve. A logistic
curve is also known as an S-shaped curve. And basically what's going to happen is if you
have very large values on your right-hand side, so your linear combination is very large, what that's
going to do is this is going to go to 1 and this approaches 0. If you have large predictor values,
the linear combination of that is basically going to predict observations that have a probability
close to 1 and on the other hand if you have very small values, the linear combination is going to
predict probabilities that approach 0 and then there are a few probabilities in between that
are in between 0 and 1. And so this is the idea of logistic regression in a nutshell. And then
if you're using logistic regression to predict an outcome variable, if you want to predict whether
Y is 0 or 1, the final step is you're going to have some kind of probability cut off.
I'm going to write that as p alpha. Let's say the probability cutoff is 0.5. The way we use the
probability cutoff is we are going to predict the probability that Y is equal to 1 and so if that is
greater than or equal to the probability cut off, then that means that we're going to label Y as 1.
Let's say using our logistic regression model we find that the probability that Y is equal to 1 is
equal to 0.56. If our probability cut off is 0.5, then that means because 0.56 is greater than 0.5
we're going to label that observation as Y is equal to 1 or that outcome as Y is equal to 1.
Probability cut off is something that you can pick as the statistician. You can have for example a
high probability cut off. You can say, I only want to label Y as equal to 1 if the probability that Y
is equal to 1 is 0.9 or greater. You might only want to label Y as equal to 1 if you're really,
really sure. What you pick as your probability cut off, it's going to affect things like the
sensitivity of the model or the specificity of the model and also the accuracy. That's something that
you can play around with to try to maximize whatever diagnostic metric you want to use to
pick a good model. And one last thing I want to note about the logistic regression before I move
on to an example is that you can see that once we do the transformation on the left-hand side,
we do get that the left-hand side is equal to a linear combination of terms on the right-hand
side. This is not considered directly a linear model but it belongs to a family of models known
as GLMs, or generalized linear models. This is a type of model where if you have a function of an
outcome variable equal to a linear combination of terms, that's known as a generalized linear model.
It's basically like a special type of linear model where you have to have some kind of function on
the left-hand side. And in the last part of the video, I just want to do this example problem.
In this problem, we're given an equation for a logistic regression and it has two predictor
variables, X1 X2. Then we're given the values of X1 and X2. We're asked to calculate the
probability that Y is equal to one and then with that probability classify Y as equal to
0 or 1 using the probability cut off of 0.5. For this type of problem--and you might do this on a
computer, but basically the way that you solve for p, which is equal to the probability that
Y is equal to 1. This is always the same formula. I'll go ahead and derive it in this video but this
formula is basically going to be e to the linear combination of terms, B0 plus B1 X1 plus . . . Bm
Xm over 1 plus e to the linear combination of terms, B0 plus B1 X1 plus . . . Bm Xm.
And so this is always going to be the formula for this type of problem to solve for the probability
and again if you're doing logistic regression on a computer this is basically how the computer is
calculating these probabilities. But let me go ahead and solve this problem and show you how
I come to that formula. I'm going to first plug in X1 and X2 into the equation. Then we're just
going to have to solve for p and we're going to have numbers for all the other values.
On the left-hand side that's going to be the log of the odds of the probability, which is p
over 1 minus p is equal to -1.55 plus 0.73 times -0.9 minus 3.11 times -0.77. This entire equation,
if we add and subtract everything, is going to be equal to 0.1877. When we say we take the log odds,
this is usually assumed to be a base of e. It's usually assumed to be the natural log. To get
rid of the log on the left-hand side, I'm going to take e to the both sides, the left-hand side and
the right-hand side, of the equation. That's going to give me on the left-hand side p over 1 minus p
and then the right-hand side is going to be e to the 0.1877. I'm going to multiply both sides by
the denominator. I'm going to multiply by 1 minus p. That's going to give me p is equal to 1 minus
p times e to the 0.1877. I'm going to distribute the e to the 0.1877. That's going to give me e to
the 0.1877 minus p times e to the 0.1877 is equal to p. I want to solve for p. I'm going to add this
term to both sides. It's going to give me p plus p times e to the .1877 is equal to e to the 0.1877.
And then I'm going to draw out the p. That's going to be equal to p times 1 plus e to the
0.1877 is equal to e to the 0.1877. And to solve for p I'm going to divide both sides by this term.
That's going to give me p is equal to e to the 0.1877 divided by 1 plus e to the 0.1877. That's
how you solve this particular equation but again in general the way the formula always is is it's
going to be e to the linear combination of terms over 1 plus e to the linear combination of terms.
When I actually solve for this number, that's going to equal approximately 0.5468.
Since we have in the problem that the probability cut off p alpha is equal to 0.5
and because the probability p is 0.5468, since p is greater than the probability
cut off we're going to classify Y as equal to 1. And we are done with this problem.
*chiptune music*