If I were in Hawaii
I'd be sitting on a beach
In the shade of a tree
Watching snack quest
Hello, I'm Josh Starmer and welcome to StatQuest today
We're gonna cover logistic regression and we're gonna dive deep into the details. This is part one of a series of videos
I'm gonna do on logistic regression this time we're talking about coefficients
This stat quest follows up on logistic regression clearly explained
Which provides the big picture of what logistic regression is and how it works
In this video. I want to dive deeper into how logistic regression works
Specifically we'll talk about the coefficients that are the results of any logistic regression
We'll talk about how they are determined and interpreted
we'll talk about the coefficients in the context of using a continuous variable like weight to predict obesity and
We'll talk about the coefficients in the context of testing if a discrete variable
Like whether or not a mutated gene is related to obesity
Before we dive into the details
Let's do a quick review of some of logistic regressions main ideas to make sure we're all on the same page in
This example the y-axis is the probability a mouse is obese
It goes from 0 the mouse is not obese to 1 the mouse is obese
The dotted line is fit to the data to predict the probability a mouse is obese given its weight
If we wait a heavyweight Mouse
Then the corresponding point on the line indicates that there is a high probability
close to one that it is obese and
if we weight a middleweight Mouse
Then there's an intermediate probability close to 0.5 that it is obese
Lastly a lightweight Mouse has a low probability close to 0 of being obese
Ok, those are all the basics we need for this stat quest
One last thing before we get started
I want to mention that logistic regression is a specific type of generalized a linear model often abbreviated
GLM
Generalized linear models are a generalization of the concepts and abilities of regular linear models
Which we've already talked about in many stat quests
That means that if you're familiar with linear models, then you are well on your way to understanding logistic regression
We'll start by talking about logistic regression when we use a continuous variable like weight to predict obesity
This type of logistic regression is closely related to linear regression a type of linear model
So let's do a super quick review of linear regression
Shameless self-promotion if you're not already familiar with linear regression
Check out the stat quests on linear regression and linear models part one
Ok, we start with some data and we fit a line to it and this is the equation for the line
It has a y-axis
intercept and a slope and
We plug in values for wait
to get predicted values for size
So even though we didn't measure a mouse that weighed 1.5
we can plug 1.5 into the equation and
predict that Mouse to have size
1.9 1
And a mouse with weight 1.5 and size 1.9 one would end up on the line at this point
Now even though it's silly the equation can predict the size of mice with weight equals zero
that's just the y axis intercept zero point eight six and
We can even predict the size of mice with negative weights
I'm pointing this out because the fact that we are not limiting the equation to a specific domain and range
makes it easier to solve and
This has a big effect on how logistic regression is done
So over on the left side we have the linear regression and on the right side we have the logistic regression
with linear regression
the values on the y-axis
Can in theory be any number?
Unfortunately with logistic regression the y axis is confined to probability values between 0 and 1
To solve this problem
the y axis in logistic regression is
Transformed from the probability of obesity to the log odds of obesity. So just like the y axis and linear regression
It can go from negative infinity to positive infinity
So we can see what we're doing let's move the logistic regression to the left side
Now let's transform the y-axis from a probability of obesity scale to a log odds of obesity scale
We do that with the logit function that we talked about in the odds slash log odds stat quest
P in this case is the probability of a mouse being obese and
Corresponds to a value on the old y axis between 0 and 1
The midpoint on the old y axis
corresponds to P equals 0.5 and
When we plug P equals 0.5 into the logit formula and do the math
We get 0 the center of the new y axis here
Is P equals zero point 7 31 on the old y axis if?
We plug P equals zero point seven three one into the logit function and do the math
we get one on the new y axis if
We plug P equals zero point eight eight into the logit function and do the math
we get two on the new y axis if
We plug P equals zero point nine five into the logit function and do the math
We get three on the new y axis
Lastly these blue points from the original data are at P equals one
If we plug P equals one into the logit function and do the math
Well, technically you can't divide by zero. However
the log of one divided by zero
equals the log of one minus the log of zero and the log of zero is defined as negative infinity and
Since something minus negative infinity equals positive infinity this whole thing is equal to positive infinity
This means the original samples that were labeled obese are at positive infinity on the new y-axis as
a result the original
y-axis from 0.5 to 1 is
Stretched out from 0 to positive infinity on the new y-axis
Similarly
0.5 to 0 on the old y-axis is stretched out from 0 to negative infinity on the new y-axis
Ultimately we end up with the log of the odds of obesity on the new y-axis
And the new y-axis transforms the squiggly line into a straight line
The important thing to know is that even though the graph with the squiggly line is what we associate with logistic regression
The coefficients are presented in terms of the log odds graph
In the stack quest logistic regression details part 2 fitting a line with maximum likelihood
We'll talk more about how this line is fit to the data
But for now just take my word for it that this is the best fitting line
Just like with linear regression the best fitting line has a y-axis intercept and a slope
The coefficients for the line are what you get when you do logistic regression
The first coefficient is the y-axis intercept when weight equals zero
It means that when weight equals zero
the log of the odds of obesity are negative three point four seven six in
Other words if you don't weigh anything the odds are against you being obese, duh
here's the standard error for the estimated intercept and
The Z value is the estimated intercept divided by the standard error
In other words, it's the number of standard deviations. The estimated intercept is away from zero on the standard normal curve
That means that this is the Wald's test that we talked about in the odds ratio stat quest
Since the estimate is less than two standard deviations away from zero. We know it is not statistically significant and
This is confirmed by the large p-value the area under the standard normal curve
That is further than one point four seven standard deviations away from zero
The second coefficient is the slope
It means that for every one unit of weight gained
the log of the odds of obesity increases by one point eight to five
Here's the standard error for the slope
Again, the Z value is the number of standard deviations. The estimate is from zero on the standard normal curve and
Again, the estimate is less than two standard deviations from zero. So it is not statistically
significant, this is no surprise with such a small sample size and
This is confirmed with the large p-value
Double bam
Now we know all about the logistic regression coefficients when we use a continuous variable like weight to predict obesity
now let's talk about logistic regression coefficients in the context of testing if a discrete variable like
whether or not a mouse has a mutated gene is related to obesity on
the left side we have mice that have a normal gene and
On the right side. We have mice with a mutated gene
Just like before some of the mice are obese in some of the mice are not obese
This type of logistic regression is very similar to how a t-test is done using linear models
So let's do a quick review of how a t-test is done using linear models
Shameless self-promotion
If you're not already familiar with how a t-test can be performed using a linear model then check out the stat quest
linear models part two
t-tests and anova
Okay, we start with some data in
this case we've measured the size of mice that have a normal gene and
the size of mice with a mutated version of that gene
Then we fit two lines to the data
the first line represents the mean size for the mice with the normal copy of the gene the
Second line represents the mean size of the mice with the mutated copy of the gene
These two lines come together to form the coefficients in this equation
the mean value for size for the mice with the normal copy of the gene goes here and
The difference between the mean size of the mice with a mutated gene and the mean size of the mice with the normal gene
goes here
We then pair this equation with a design majors to predict the size of a mouse
Given that it has the normal or mutated version of the gene
This is the design matrix for the observed data
The first column corresponds to values for b1 and it turns on the first coefficient the mean of the normal mice
The second column corresponds to values for b2 and turns the second coefficient
the mean of the mutant mice minus the mean of the normal mice off or on
Depending on whether it is a zero or a one
For example the first row in the design matrix corresponds to a mouse with a normal copy of the gene
We predict its size by replacing b1 with 1 in
replacing b2 with 0 and
then we just do the math and
We see that this mouse is associated with the mean of the normal mice
This row in the design matrix corresponds to a mouse with the mutated gene
We plug in 1 for b1 and
Plug in 1 for b2 and we see that the mean of the mice with the normal gene
plus the difference between the two means
Associates this mouse with the mean for the mice with a mutated gene
When we do a t-test this way we are basically testing to see if this coefficient
The mean of the mutant mice minus the mean of the normal mice is equal to zero
But you already know all this t-test stuff what you really want to know is how it applies to logistic regression
The first thing we do is transform the y-axis from the probability of being obese
To the log of the odds of obesity
Now we fit two lines to the data
For the first line we take the normal gene data and
Use it to calculate the log of the odds of obesity for mice with the normal gene
Thus the first line represents the log of the odds of obesity for the mice with the normal gene
Let's call this the log of the odds for gene normal
We then calculate the log of the odds of obesity for the mice with the mutated gene
Thus the second line represents the log of the odds of obesity for a mouse with the mutant gene
Let's call this the log of the odds gene mutated
These two lines come together to form the coefficients in this equation
the log of the odds gene normal goes here in
the difference between the log of the odds gene mutated in the log of the odds gene normal goes here and
since subtracting one log from another
Can be converted into division this term is a log of the odds ratio
It tells us on a log scale how much having the mutated gene
Increases or decreases the odds of a mouse being obese
Okay, now that we know what the equation is all about let's substitute in the numbers
the log of the odds for gene normal is just the log of 2 divided by 9 and
The log of the odds for gene mutated is just the log of 7/3
Now we just do the math and
that gives us these coefficients and
Those are what you get when you do logistic regression
the intercept is the log of the odds for gene normal and
The gene mutant term is the log of the odds ratio
That tells you on a log scale how much having the mutated gene
increases or decreases the odds of being obese and
here are the standard errors for those estimated coefficients and
Here are the z values aka the Wald's test values that tell you how many standard deviations
The estimated coefficients are away from 0 on a standard normal curve
The Z value for the intercept negative 1.9 tells us that the estimated value for the intercept negative
1.5 is less than 2 standard deviations from 0 and thus not significantly different from 0 and
this is confirmed by a p-value greater than
0.05
The z-value for gene mutant the log odds ratio that describes how having the mutated gene
Increases the odds of being obese is greater than two
Suggesting it is statistically significant. And this is confirmed by a p-value less than
0.05
Double bam
now we have seen how some of the linear model concepts for regression apply to logistic regression and
We have seen how some of the linear model concepts for T tests apply to logistic regression
in short in terms of the coefficients
Logistic regression is the exact same as good old linear models, except the coefficients are in terms of the log odds
This means that all those fancy things we can do with linear models like multiple regression and
ANOVA
Can be done using logistic regression all we have to remember is that the scale for the coefficients is log odds
Triple bam
hooray
We've made it to the end of another exciting stat quest if you like this stat quest and want to see more of them
Please subscribe and if you want to support stat quest well
Click the like button below and consider buying one or two of my original songs
Alright until next time quest on