We're gonna do a lot of math step by step by step by step by step
StatQuest!!!
Hello, I'm Josh Starmer and welcome to StatQuest
Today we're going to talk about maximum likelihood for the normal distribution
and it's gonna be clearly explained
Not: This StatQuest follows up on the StatQuest "Maximum Likelihood Clearly Explained"
as well as the StatQuest
"Probability Versus Likelihood"
Lastly this StatQuest assumes you are already familiar with the normal distribution
If not, check out the StatQuest "The Normal Distribution Clearly Explained"
Let's start with this nasty-looking equation
It's the equation for the normal distribution or normal curve
It has two parameters
the first parameter, the Greek character μ (mu)
determines the location of the normal distribution's mean
a smaller value for μ moves the mean of the distribution to the left
and a larger value for μ moves the mean of the distribution to the right
the second parameter, the Greek character σ (sigma), is the standard deviation
and determines the normal distribution's width
a larger value for σ makes the normal curve shorter and wider
and a smaller value for σ makes the normal curve taller and narrower
In this StatQuest we're going to use the likelihood of the normal distribution
to find the optimal parameters for μ the mean and σ, the standard deviation,
given some data x
Let's start with the simplest data set of all: a single measurement
Here we've measured a single Mouse and it weighs 32 grams.
Now
just to see what happens
We can overlay a normal distribution with μ equals 28 and σ equals 2 onto the data
To determine the likelihood of the data given this curve we can plug the numbers into the likelihood function
and then plug the numbers into this equation
and here's the equation with μ equals 28
σ equals 2 and x equals 32 plugged in
Now just do the math
plug and chug plug and chug
and the likelihood of the curve with μ equals 30 and σ equals 2, given the data,
is 0.03
thus the y-axis value here is 0.03
Now we can shift the distribution a little bit to the right by setting μ equals 30 and then calculate the likelihood
Again we just plug the numbers into the likelihood function
and now we just plug and chug
and the likelihood of the new curve with μ equals 30 and σ equals 2, given the data, is 0.12
Thus the y-axis value here is 0.12
If we decide to fix σ equals 2 so that it is a given just like the data
Then we can plug in a whole bunch of values for μ and see which one gives the maximum likelihood
For example, if we start with the mean of the distribution over here on the left at 20 grams
then we plug μ equals 20 into the equation
and we get a crazy small likelihood and
Then we can plot the likelihood on a graph
The y-axis is the likelihood value and the x-axis is for the different values that we plug in for μ
Each time we change μ we calculate the likelihood and plot it
We can identify the peak in the likelihood graph by determining where the slope of the curve equals zero in
this case the slope equals zero when μ equals 32
Now we can fix μ equals 32 and treat it like a given just like the data
And we can plug in different values for σ to find the one that gets the maximum likelihood
Note: You actually need more than one measurement to find the optimal value for σ
I'll leave the reason why as an exercise for the viewer
That said, if we had more data
Then we could plot the likelihoods for different values of σ
And the maximum likelihood estimate for σ would be at the peak, where a slope of the curve equals zero
The goal of this super simple example is to convey the basic concepts of how to find the maximum likelihood
estimates for μ and σ
To solve for the maximum likelihood estimate for μ
We treat σ like it's a constant and then find where the slope of its likelihood function is 0
And to solve for the maximum likelihood estimate for σ
we treat μ like it's a constant and then find where the slope of its likelihood function is 0
The example with one measurement kept the math simple, but now I think we're ready to dive in a little deeper
So let's use a two sample data set to calculate the likelihood of a normal distribution
Crazy Times!
To keep track of things, let's call the mouse that weighs 32 grams X sub 1
And the mouse that weighs 34 grams X sub 2
And again, just to see what happens
Let's overlay a normal distribution with μ equals 28 and σ equals 2 onto the data
We've already seen how to calculate the likelihood for this curve given X sub 1,
the mouse that weighs 32 grams
And we can calculate the likelihood for the curve given X sub 2 by plugging in 34 into this likelihood function
but what's the likelihood of this normal curve given both X sub 1 and X sub 2
In other words, what's this?
Because these measurements are independent (i.e. weighing X sub 1 did not have an effect on weighing X sub 2)
The likelihood of a normal distribution with μ equals 28 and σ equals 2
given the data X sub 1 equals 32 and X sub 2 equals 34
is just the likelihood of the distribution given X sub 1
times
the likelihood of the distribution given X sub 2
So we just plug in the numbers and do the math
And that gives us a really small number
If we had a third data point then we just add it to the given side of the overall likelihood and
Add another individual likelihood term to our multiplication
With n data points
We add all n data points to the given side of the overall likelihood function
Then multiply together all n individual likelihood functions
BAM!!!
Now that we know how to calculate the likelihood of a normal distribution when we have more than one measurement
(we just multiply together the individual likelihoods).
Let's solve for the maximum likelihood estimates for μ and σ
BRACE YOURSELF FOR A LOT OF MATH!!!
But don't freak out,
we'll go through it one step at a time
Here's the likelihood function without any value specified for μ and σ
it equals the product of the likelihood functions for the N individual measurements
and here's what the equation looks like
What we need to do is take two different derivatives of this equation
One derivative will be with respect to μ. When we treat σ like it's a constant
and we can find the maximum likelihood estimate for μ by finding where this derivative equals zero
the other derivative will be with respect to σ when we treat μ like it's a constant
and we can find the maximum likelihood estimate for σ by finding where this derivative equals zero
But before we try to take any derivatives, let's take the log of the likelihood function
We do this because it makes taking the derivative way way easier
In the likelihood function and the log of the likelihood function both peak at the same values for μ and σ.
I know it's hard to tell that the log likelihood function for σ peeks here, but just take my word for it. I double-checked!
Here's the log of the likelihood
Now we're going to go, step by step, through all of the transformations that the log has on this function
Just a heads up - the log is going to do a lot of transformations to this function, so hang in there!!!
First the log transforms the multiplication
into addition
Now we log transform the individual likelihood functions
Let's focus on this one first
Step 1: Move the log of the first likelihood function to the top left-hand side of the screen for reference
Step 2: Convert the multiplication
into addition
Step 3: Convert 1 over the square root
into the exponent -1/2
and convert the exponent
into multiplication
Step 4: Convert the -1/2 exponent
into multiplication
and the ln(e)=1
So we can remove it
Step 5:
The log can convert the multiplication of 2πσ²
into addition of two times π and σ squared
and there's nothing more to do to this term
so it doesn't change
Step 6: Convert the exponent in the log of σ squared into 2 times the log of σ
Step 7: Lastly, the 2/2 term cancels out
BAM!!!
We started with this
and after the log transformation ended up with this!!!
I know all of this log transformation stuff looks bonkers the first time you see it
Just remember, it will make it easier to take the derivative.
So now we can go back to the original likelihood and turn this
into this
And by following the same steps, we can transform the remaining parts of the sum
into this
Now let's simplify this equation
Just to be clear about how we simplify, keep in mind that since we have n data points
that means we have a term for the first data point, X sub 1
and that this represents the terms for the remaining n minus 1 data points.
So we can combine this from the first term
with the remaining n minus 1 occurrences.
1 plus n minus 1 equals n
Then all n of the negative log of σ's can be combined
and the last parts of each term stay the same.
This is the log of the likelihood function after simplification, and it is what we will take the derivative of.
So, let's move it to the top for reference
We'll start by taking the derivative with respect to μ
This derivative is the slope function for the log of the likelihood curve and we'll use it to find the peak
a.k.a where the slope equals zero
The first term doesn't contain μ, so it's derivative is 0
The second term doesn't contain μ either, so it's derivative is also 0
The third term contains μ, so now we have to work
Specifically, the numerator contains μ and we have to apply
The Chain Rule
So we start with the outer derivative of negative 32 minus μ squared
and then we multiply that by the derivative of 32 minus μ
and then we simplify
Because the derivative is with respect to μ, σ is a constant and, thus, the denominator doesn't change.
So we can combine the two pieces
and the 2s cancel out
and that means the derivative of this
is this
Likewise, we apply
The Change Rule
To the remaining terms and get
these!
BAM!!!
Now we can simplify the equation
These zeros go away
and we can pull the σ squared out
and add the numerators together.
Then we can combine the measurements
and , lastly, we can combine the μs
Thus, this is the derivative of the log-likelihood function with respect to μ
DOUBLE BAM!!!
Now, let's take the derivative of the log-likelihood function with respect to σ.
This derivative is the slope function for the log of the likelihood curve
and we'll use it to find the peak, which is hard to see in this graph!
So, from here on out, because they peak at the same spot
I'll show you the likelihood functions instead of the log-likelihood functions
The first term doesn't contain σ, so it's derivative is zero.
The derivative of the second term is just n over σ.
The derivative of this term isn't tricky but it's easier to figure out when we rewrite 1 over σ squared
as σ to the negative 2
Since we're taking the derivative with respect to σ, this part can be treated like a big constant.
and the derivative of this
is this.
The two negatives cancel out
and then the 2s cancel out
and lastly we put σ back into the denominator by changing the sign of the exponent
So the derivative of this
is this!
So that's what we put into the equation.
Likewise, the derivatives of the remaining terms
simplify to these.
BAM!!!
This is the derivative of the log-likelihood function with respect to σ.
We can simplify the equation by omitting the zero
leaving the second term the same
and we can pull the σ cubed out
and add up the remaining terms.
DOUBLE BAM!!!
At long last here are the two derivatives!!!
To find the maximum likelihood estimate for μ
we need to solve for where the derivative with respect to μ equals zero
because the slope is zero at the peak of the curve
Likewise to find the maximum likelihood estimate for σ
we need to solve for where the derivative with respect to σ equals zero
because the slope is zero at the peak of the curve
We'll start by setting the derivative with respect to μ to 0
and solve for μ
We start by multiplying both sides by σ squared, that makes the σ squared go away
then we add n times μ to both sides
divide both sides by n and solve.
Thus, the maximum likelihood estimate for μ is the mean of the measurements.
So that is where the center of our normal curve will go.
DON'T GET TOO EXCITED. WE'RE NOT DONE YET!!!
DANG!!!
Now we need to set the derivative with respect to σ to 0
Now multiply both sides by σ
add n to both sides
multiply both sides by σ squared
divide both sides by n
and take the square root of both sides
and at long last
we see that the maximum likelihood estimate for σ is the standard deviation of the measurements
Thus, we use the formula for the standard deviation to determine the width of the normal curve that
given the data, has the maximum likelihood
In summary
The mean of the data is the maximum likelihood estimate for where the center of the normal distribution should go
and the standard deviation of the data is the maximum likelihood estimate of how wide the normal curve should be
Note: These solutions may be obvious, but now we have the math that proves that our intuition is correct.
wait for it
TRIPLE BAM!
Hooray! We've made it to the end of another exciting StatQuest.
If you like this StatQuest and want to see more of them
please subscribe,
and if you want to support StatQuest, well, please consider buying one or two of my original songs
Alright, until next time
QUEST ON!!!