Hey everyone, in this video I want to go over this statistics and probability proof. Basically I want
to prove a well-defined relationship between the mean square error of an estimator and its bias and
variance, which is that the mean square error of an estimator--in this case our estimator is
going to be theta hat--the mean square error of an estimator is equal to the bias squared plus
the variance. This is a well-defined relationship and I just want to prove it using the expectation
definitions, or equations for MSE, bias, and variance. Using expectation equations,
the mean squared error of an estimator theta hat is equal to the expected value of theta hat minus
theta squared. And we can also define the bias and the variance of an estimator using expectation.
The variance of an estimator theta hat is equal to the expected value of theta hat squared minus the
expected value squared of theta hat. And the bias of an estimator theta hat is equal to the expected
value of theta hat minus theta. In this case, theta--wherever we see that--that's going to be
a parameter and it's considered to be a constant instead of a random variable but the estimator
theta hat is going to be a random variable. Using these expected value equations for MSE,
variance, and bias, basically what I'm going to do is I'm going to decompose the expected value
definition of MSE and try to get it into a form where the expected value equations are equal to
the variance plus bias squared. Breaking down the expected value of theta hat minus theta squared,
that is equal to--just by expanding this binomial expression--that's going to be equal
to the expected value of theta hat minus theta times theta hat minus theta. If we distribute
that expression, that's going to be equal to the expected value of theta hat squared my
minus 2 theta hat theta plus theta squared. We can distribute the expected value to each term
individually. That's going to be equal to the expected value of theta hat squared. And then
2 and theta are constants, so we can pull that out, so that's going to be minus 2 theta, the
expected value of theta hat, plus theta squared. For MSE, I would stop simplifying it there and
then next I'm going to show that bias squared plus variance can simplify to this expression
and if it can simplify to that expression, then the two expressions are equal to each other,
and then that means that bias squared plus the variance is equal to MSE. Before I do that,
I'm also going to note that the bias squared of theta hat is equal to the expected value of theta
hat minus theta times the expected value of theta hat minus theta. If we expand that expression,
that's going to give us the expected value squared of theta hat minus 2 theta expected
value of theta hat plus theta squared. Then the variance of theta hat plus the bias squared of
theta hat is going to be equal to the expected value of theta hat squared minus the expected
value squared of theta hat. This is the variance portion written as the expected value equation
plus the bias squared, which is equal to the expected value squared of theta hat minus
two theta times the expected value of theta hat plus theta squared. In this expression,
we have minus the expected value squared of theta hat plus the expected value squared of theta hat,
so these two terms cancel out to zero. The variance of theta hat plus the bias squared
of theta hat is equal to the expected value of theta hat squared minus two theta expected value
of theta hat plus theta squared. And if we go back to how we defined the mean squared error,
we're able to say that that was equal to the expected value of theta hat squared minus two
theta expected value theta hat plus theta squared. Because these two terms, or these two expressions,
are equal to each other therefore that means that the variance of theta hat plus the bias squared
of theta hat is equal to the mean square error of theta hat. And we are done with this problem.
*chiptune music*