Let us now introduce the linear least mean squares formulation.
The setting is the usual one-- we
have an unknown random variable and another random variable,
which is our observation.
We're given enough information so that we can, for example,
calculate the joint distribution of these two random variables.
What we would like to do in the least squares
methodology is to come up with an estimator, such
that the mean squared error of this estimator
is as small as possible.
And we have seen the general solution to this problem.
If we consider arbitrary estimators,
it turns out that the best possible estimator,
the best possible function g, is this particular function
of the observations.
Our estimator is a conditional expectation of Theta, given X.
Now, let us look at an example that we considered earlier.
Suppose that X and Theta have a joint PDF, which
is uniform over this particular region.
We did consider this example and we
found that the optimal estimator was
a function that had this particular shape.
So this blue curve here corresponds
to the function, which is the conditional expectation
of Theta, given the value of the observation
that we have obtained.
We notice that this function is nonlinear,
but it is only mildly nonlinear.
The fact that it is nonlinear is a little bit of a nuisance.
It makes it somewhat of a complicated function.
Wouldn't it be nicer if our estimator had turned out
to be a linear function of the data, such as this one?
It would have been nicer, but, unfortunately,
that's not the case.
By what if we impose it as a constraint,
that we will only look at estimators
that are linear functions of the data.
What does that mean?
Mathematically speaking, it means
that we will only consider estimators
that depend linearly on the data X.
Now, a and b here are some parameters
that are for us to choose.
If I choose a and b differently, I'm
going to get a different red curve here.
Which one is the best red curve?
Well, we need a criterion.
But let us stick to our mean squared error criterion.
And in that case, we're led to the following formulation.
We want to find choices for a and b.
That is, we want to choose a particular red line,
so that the resulting estimation error, the resulting
mean squared estimation error, is as small as possible.
So what we have here is a random variable.
Here is the value that's going to be
given to us by our estimator.
And we look at the associated error,
square it, and take the expectation.
So this is the linear least mean squares formulation.
We're looking for an estimator, which
is a linear function of the data.
And we want to choose the best possible linear function.
How does it compare to the earlier problem
of picking the best estimator?
Here we were considering an arbitrary function g
and we were trying to find the best possible function
of the data, which would be our estimator.
So this was really an optimization
over all possible functions.
Here we only have an optimization with respect
to two numbers.
So at least mathematically, this should be a simpler problem.
And we will see that it has a simple solution.
Before going on to the solution, however,
let me make one comment that in some cases,
the linear least squares estimation
problem is relatively easy to solve.
And these are the cases where the conditional expectation
turns out to be linear in the data.
This is the best possible estimator.
If it happens to be linear, it's at least
as good as any other linear estimator,
so it's also going to be the optimal linear estimator.
That is, if the optimal solution turns out
to be already linear, by imposing the extra constraint
of sticking to linear estimators is not
going to make any difference.
But for the general case, in general,
this is not going to be the case.
The conditional expectation may well
turn out to be a nonlinear function of the data,
as in this example.
And in those cases, the linear least mean squares estimator
is going to turn out to be different.