We can finally go ahead and introduce the basic elements
of the Bayesian inference framework.
There is an unknown quantity, which
we treat as a random variable, and this is what's special
and why we call this the Bayesian inference framework.
This is in contrast to other frameworks
in which the unknown quantity theta is just
treated as an unknown constant.
But here, we treat it as a random variable,
and as such, it has a distribution.
This is the prior distribution.
This is what we believe about Theta
before we obtain any data.
And then, we obtain some data, which are some observation.
That observation is a random variable,
but when the process gets realized,
we observe an actual value, numerical value,
of this random variable.
The observation process is modeled,
again in terms of a probabilistic model.
We specify the distribution of X,
but we actually specify the conditional distribution of X.
We say how X will behave if Theta happens
to take on a specific value.
These two pieces, the prior and the model of the observations,
are the two components of the model
that we will be working with.
Once we have obtained a specific value for the observations,
then we can use the Bayes rule to calculate
the conditional distribution of Theta,
either a conditional PMF if Theta is discrete
or a conditional PDF if Theta is continuous.
And this will be a complete solution, in some sense,
of the Bayesian inference problem.
There's one philosophical issue about this framework, which
is where does this prior distribution come from?
How do we choose it?
Sometimes we can choose it using a symmetry argument.
If there's a number of possible choices for Theta
and there's a reason to believe that they're all
equally likely, we have no reason
to believe that one is more likely than the other, then
the symmetry consideration gives us a uniform prior.
We definitely take into account any information
we have about the range of the parameter Theta,
so we use that range and we assign 0 prior probability
for values of Theta outside the range.
Sometimes, we have some knowledge about Theta
from previous studies of a certain problem, that tell us
a little bit about what Theta might be,
and then when we obtain new observations,
we refine those results that were obtained
from previous studies by applying the Bayes rule.
And in some cases, finally, the choice
could be arbitrary or subjective just reflecting our beliefs
about Theta, some plausible judgment
about the relative likelihoods of different choices of Theta.
Now, as we just discussed, the complete solution
or the complete answer to a Bayesian inference problem
is just the specification of the posterior distribution
of Theta given the particular observation that we
have obtained.
Pictorially, if Theta is discrete,
a complete answer might be in the form of such a diagram that
tells us that certain values of Theta
are possible with certain probabilities.
Or if Theta is continuous, a complete solution
might be in the form of a conditional PDF that again
tells us the conditional distribution of Theta.
To appreciate the idea here, consider the problem
of guessing the number of electoral votes
that a candidate gets in the presidential election.
The electoral votes are certain votes
that the candidate gets from each one
of the states in the United States.
And there is a certain number that the candidate
needs to get in order to be elected president.
One possible prediction could be a statement
that I predict that candidate A will win,
but actually a more complete presentation
of the results of a poll could be
a diagram of this kind, which is essentially a PMF.
Here, a particular pollster collected all the data
and gave the posterior probability distribution
for the different possible numbers of electoral votes.
And this diagram is a lot more informative
than the simple statement that we expect a certain candidate
to get more than the required electoral votes.
So what is next?
As we just discussed, the complete solution
is in terms of a posterior distribution,
but sometimes, you may want to summarize this posterior
distribution in a single number or a single estimate,
and this could be a further stage
of processing of the results.
So let us talk about this.
Once you have in your hands the posterior distribution
of Theta, either in a discrete or in a continuous setting,
and if you're asked to provide a single guess about what
Theta is, how might you proceed?
In the discrete case, you could argue as follows.
These values of Theta all have some chance of occurring.
This value of Theta is the one which is the most likely,
so I'm going to report this value
as my best guess of what Theta is.
And using a similar philosophy, you
could look at the continuous case
and find the value of Theta at which the PDF is largest
and report that particular value.
This particular way of estimating Theta
is called the maximum a posteriori probability rule.
We already have in our hands the specific value of X,
and therefore, we have determined
the conditional distribution for Theta.
What we then do is to find the value of theta
that maximizes over all possible thetas the conditional PMF
of this random variables capital Theta.
And similarly in the continuous case,
the value of theta that maximizes the conditional PDF
of the random variable Theta.
This is one way of coming up with an estimate.
One can think of other ways.
For example, I might want to report instead, the mean
of the conditional distribution, which in this diagram
might be somewhere here, and in this picture,
it might be somewhere here.
This way of estimating theta is the conditional expectation
estimator.
It just reports the value of the conditional expectation,
the mean of this conditional distribution.
It is called the least mean squares estimator,
because it has a certain useful and important property.
It is the estimator that gives you
the smallest mean squared error.
We will discuss this particular issue
in much more depth a little later.
Now, let me make two comments about terminology.
What we have produced here is an estimate.
I gave you the conditional PDF or conditional PMF,
and you tell me a number.
This number, the estimate, is obtained
by starting with the data, doing some processing to the data,
and eventually, coming up with a numerical value.
Now, g is the way that we process the data.
It's a certain rule.
Now, if we know the value of the data,
we know what the estimate is going to be.
But if I do not tell you the value of the data
and you look at the situation more abstractly,
then the only thing you can tell me
is that I will be seeing a random variable,
capital X, I will do some processing to it,
and then I will obtain a certain quantity.
Because capital X is random, the quantity that I will obtain
will also be random.
It's a random variable.
This random variable, capital Theta hat,
we call it an estimator.
Sometimes, we might also use the term estimator
to [refer to] the function g, which
is the way that we process the data.
In any case, it is important to keep this distinction in mind.
The estimator is the rule that we use to process the data,
and it is equivalent to a certain random variable.
An estimate is the specific numerical value
that we get when the data take a specific numerical value.
So if little x is the numerical value of capital X,
in that case, little theta hat is the numerical value
of the estimator capital Theta hat.
So at this point, we have a complete conceptual framework.
We know, abstractly speaking, what
it takes to calculate conditional distributions,
and we have two specific estimators at hand.
All that's left for us to do now is
to consider various examples in which we can discuss what
it takes to go through these various steps.