In the context of the problem of estimating
the unknown bias of a coin, we encountered this distribution,
which is known as a Beta distribution.
It's a probability density for a random variable, Theta,
that takes values in the interval from 0 to 1.
So this formula is valid for thetas in this range.
And k here is a non-negative integer.
Now, this coefficient here, d(n,k),
is a normalizing constant, which is needed so that this is
a legitimate PDF, that it integrates to 1.
And so in particular, d needs to be
equal to the integral of what we have in the numerator.
This is the choice that makes this whole expression integrate
to 1.
And this integral is calculated and can
be found to be equal to this particular expression.
How do we derive this expression?
One way is to carry out a long exercise in calculus.
We have this integral here.
You might either expand it and then integrate and collect
terms, or you could try to demonstrate this equality
by applying integration by parts.
But this is complicated.
Is there some simple way of arguing and deriving
this expression?
We will see that there is a very simple probabilistic argument
for deriving this equality.
What we will actually derive is this same equality,
but in slightly different notation.
Instead of k, we will use alpha.
Instead of n minus k, we will use beta.
So here we have alpha factorial times beta factorial.
In the denominator, we have the sum
of these two coefficients plus 1,
so this corresponds to alpha plus beta plus 1 factorial.
This is what we want to demonstrate.
What we will do will be to consider the following setting.
We start with alpha plus beta plus 1,
that many independent random variables that
are uniform on the unit interval,
and we will consider the following event
and its probability.
This is the probability that these random variables
happen to be ordered in some particular order.
Let us call this event A, so this is the probability of A.
Now, this probability is not hard to calculate.
We have alpha plus beta plus 1 random variables-- independent,
identically distributed.
By symmetry, any particular way of ordering
these random variables is equally likely.
How many ways are there to order alpha plus beta
plus 1 random variables?
It's the factorial of the number of items
that we're trying to order.
We're talking about the probability
of a particular permutation, so this probability
is equal to 1 over the number of permutations
of alpha plus beta plus 1 objects.
So this is one expression for the probability of this event
A.
Now, we're going to calculate this probability
in a different way.
What we will do is we're going to apply the total probability
theorem.
We're going to condition on Z. We're
going to calculate the conditional probability of A
given that Z takes a specific value,
and then weigh those probabilities according
to the probability density of the random variable Z.
So this is just the total probability theorem
applied to this particular context.
And now to make progress, what we will need to do
is to find this conditional probability.
We fix a constant little theta, and we want the probability
that this event happens.
What is this event?
It is the event that all of the X's fall inside this interval,
all the Y's fall inside this interval,
and furthermore, the X's are sorted and the Y's are sorted.
So let us write this out.
It's the probability that all of the X's happen
to be less than theta, all the Y's
happen to be bigger than theta, and also, not just that,
but the X's are sorted, and furthermore, the Y's
are sorted as well.
Clearly, if I give you the value of theta
so that Z is equal to theta, for this event to happen,
we must have all these events here happen as well.
So now, let us try to calculate the probability of this event.
We're going to use the multiplication rule.
First, take the probability of this event
and then the conditional probability of that event.
The X's and the Y's are independent,
so we can take the probability of this event and then multiply
with the probability of this event involving the Y's.
How about the probability of this event,
that all of the X's are less than theta?
Since the X's are independent, this
is going to be equal to the probability
that X1 is less than theta.
What is this probability?
Since X1 is uniform on the unit interval and this is theta,
the probability of falling in this interval
is equal to theta.
Times the probability that X2 is less than theta.
This probability is, again, theta and so on.
We have alpha many terms of that kind,
so this probability that all of these random variables
are less theta is equal to theta to the power of alpha.
Similarly, about the Y's.
For any particular Y, the probability
that it falls in this interval is
equal to the length of this interval, which
is 1 minus theta.
This is the probability for each one of the Y's.
There's beta many Y's.
The Y's are independent.
So the probability that all of them fall in this interval
is going to be this number to the power of beta.
So suppose that I told you that all the X's are
less than theta, and then I ask you,
given this information, what is the probability
that the X's that you got are arranged
in this particular order?
Now, because of the complete symmetry of the problem,
even if I told you that all the X's fall inside this interval,
any order of the X's is equally likely.
So the probability of this particular order
is going to be 1 over the number of possible orderings.
How many ways are there that alpha items can be ordered?
There are alpha factorial possible orderings,
so the probability that I obtain one particular ordering
is 1 over alpha factorial.
And similarly, if I tell you that the Y's all
fell in this interval by symmetry,
the probability of a particular order
is going to be 1 over the [number of possible]
orders, which is beta factorial.
All right.
So we have this conditional probability,
and now we can go back to this formula and substitute,
and what we obtain is the integral of this expression,
theta to the alpha, 1 minus theta [to the] beta, 1
over alpha factorial times 1 over beta factorial.
Then we have the density of Z, but since Z is uniform,
the density is equal to 1.
And then we have a factor of d theta.
So what have we achieved?
We calculated the probability of the event A
in two different ways, and we got two seemingly different
answers.
But these two answers have to agree.
Therefore, this expression is equal to that expression.
And now if you take this factor, 1 over alpha factorial times 1
over beta factorial, and send it to the other side
of the equation, what we obtain is exactly the formula
that we wished to derive.
This example is an instance of the following.
There are certain formulas in mathematics
that are somewhat complicated to derive,
and their derivations using, for example,
calculus are quite unintuitive.
But once you interpret the various terms that
appear in such a relation in a probabilistic way,
you can sometimes find very easy derivations and explanations
why such a formula has to be true.