In this lecture sequence, we introduced
quite a few new concepts and went
through a fair number of examples.
So for this reason, it is useful now
to just take stock and summarize the key ideas and concepts.
The starting point in a Bayesian inference problem
is the following.
There's an unknown parameter, Theta,
and we're given a prior distribution
for that parameter.
We're also given a model for the observations, X,
in terms of a distribution that depends
on the unknown parameter, Theta.
The inference problem is as follows.
We will be given the value of the random variable X.
And then we want to find the posterior distribution
of Theta, that is, given this particular value of X, what
is the conditional distribution of Theta?
In the case where Theta is discrete,
this will be in terms of a PMF.
If Theta is continuous, this would be a PDF.
We find the posterior distribution
by using an appropriate version of the Bayes rule.
And here we have four different combinations or four choices,
depending on which variables are discrete or continuous.
This is a complete solution to the Bayesian inference
problem-- a posterior distribution.
But if we want to come up with a single guess of what Theta is,
then we use a so-called estimator.
What an estimator does is that it calculates a certain value
as a function of the observed data.
So g describes the way that the data are processed.
Because X is random, the estimator
itself will be a random variable.
But once we obtain a specific value of our random variable
and we apply this particular estimator,
then we get the realized value of the estimator.
So we apply g now to the lowercase x,
and this gives us an estimate, which is actually a number.
We have seen two particular ways of constructing
estimates or estimators.
One of them is the maximum a posteriori probability rule
in which we choose an estimate that maximizes the posterior
distribution.
So in the case where Theta is discrete,
this finds the value of theta, which
is the most likely one given our observation.
And similarly, in the continuous case,
it finds a value of theta at which
the conditional PDF of Theta would be largest.
Another estimator is the one that we
call the LMS or least mean squares estimator, which
calculates the conditional expectation
of the unknown parameter, given the observations
that we have obtained.
Finally, we may be interested in evaluating
the performance of a given estimator.
For hypotheses-testing problems we're
interested in the probability of error.
And we have the conditional probability of error.
Given the data that I have just observed
and given that I'm using a specific estimator,
what is the probability that I make a mistake?
And then there's the overall evaluation of the estimator--
how well does it do on the average
before I know what X is going to be?
And this is just the probability that I
will be making an incorrect decision.
For estimation problems, on the other hand,
we're interested in the distance between our estimates
from the true value of Theta.
And this leads us to the following conditional mean
squared error, given that we have already
obtained an observation.
And we come up with an estimator.
In particular, the value of the estimator at this time
would be completely determined by the data that we obtained.
But Theta, the unknown parameter remains random.
And there's going to be a certain squared error.
We find the conditional expectation
of this squared error in this particular situation, where
we have obtained a specific value of the random variable,
capital X. On the other hand, if we're
looking at the estimator more generally,
how well it does on the average, then
we look at the unconditional mean squared error,
and this gives us an overall performance evaluation.
How do we calculate these performance measures?
Here, we live in a conditional universe.
And in a Bayesian estimation problem at some point
we do calculate the posterior distribution of Theta,
given the measurements.
So these calculations involved here
consist of just an integration or summation
using the conditional distribution.
For example, here we would integrate this quantity
using the conditional density of Theta,
given the particular value that we have obtained.
If we want to now calculate the unconditional performance,
then we would have to use the total probability
or expectation theorem.
And in that case, we can average over all the possible values
of X to find the overall error.
So all of the calculations involve tools and equations
that we have seen and that we have used in the past,
so it is just a matter of connecting those tools
with the specific new concepts that we have introduced here.