We have seen so far an example of a probability law on a
discrete and finite sample space as well as an example
with an infinite and continuous sample space.
Let us now look at an example involving a discrete but
infinite sample space.
We carry out an experiment whose outcome is an arbitrary
positive integer.
As an example of such an experiment, suppose that we
keep tossing a coin and the outcome is the number of
tosses until we observe heads for the first time.
The first heads might appear in the first toss or the
second or the third, and so on.
So in this example, any positive integer is possible.
And so our sample space is infinite.
Let us not specify a probability law.
A probability law should determine the probability of
every event, of every subset of the sample space.
That is, the probability of every
set of positive integers.
But instead I will just tell you the probability of events
that contain a single element.
I'm going to tell you that there is probability 1 over 2
to the n that the outcome is equal to n.
Is this good enough?
Is this information enough to determine the probability of
any subset?
Before we look into that question, let us first do a
quick sanity check to see whether these numbers that we
are given look like legitimate probabilities.
Do they add to 1?
Let's do a quick check.
So the sum over all the possible values of n of the
probabilities that we're given, which is an infinite
sum starting from 1, all the way up to infinity, of 1 over
2 to the n, is equal to the following.
First we take out a factor of 1/2 from all of these terms,
which reduces the exponent from n to n minus 1.
This is the same as running the sum from n equals 0 to
infinity of 1/2 and to the n.
And now we have a usual infinite geometric series and
we have a formula for this.
The geometric series has a value of 1 over 1 minus the
number whose power we're taking, which is 1/2.
And after we do the arithmetic, this turns out to
be equal to 1.
So indeed, it appears that we have the basic elements of
what it would take to have a legitimate probability law.
But now let us look into how we might calculate the
probability of some general event.
For example, the probability that the outcome is even.
We proceed as follows.
The probability that the outcome is even, this is the
probability of an infinite set that consists of
all the even integers.
We can write this set as the union of lots of little sets
that contain a single element each.
So it's the set containing the number 2, the set containing
the number 4, the set containing the
number 6, and so on.
At this point we notice that we're talking about the
probability of a union of sets and these sets are disjoint
because they contain different elements.
So we can use an additivity property and say that this is
the probability of obtaining a 2, plus the probability of
obtaining a 4, plus the probability of
obtaining a 6 and so on.
If you're curious about doing this calculation and actually
obtaining a numerical answer, you would proceed as follows.
You notice that this is 1 over 2 to the second power plus 1
over 2 to the fourth power plus 1 over 2 to the sixth
power and so on.
Now you factor out a factor of 1/4 and what you're left is 1
plus 1 over 2 to the second power, which is 1/4, plus 1
over 2 to the fourth power, which is the same as 1/4 to
the second power and so on.
And now we have 1/4 times the infinite sum of a geometric
series, which gives us 1 over 1 minus 1/4.
And after you do the algebra you obtain a numerical answer,
which is equal to 1/3.
But leaving the details of the calculation aside, the more
important question I want to address is the following.
Is this calculation correct?
We seem to have used an additivity
property at this point.
But the additivity properties that we have in our hands at
this point only talk about disjoint unions of finitely
many subsets.
Our initial axiom talked about a disjoint union of two
subsets and then later on we established a similar property
for a disjoint union of finitely many subsets.
But here we're talking about the union of
infinitely many subsets.
So this step here is not really allowed by what we have
in our hands.
On the other hand, we would like our theory to allow this
kind of calculation.
The way out of this dilemma is to introduce an additional
axiom that will indeed allow this kind of calculation.
The axiom that we introduce is the following.
If we have an infinite sequence of disjoint events,
as for example in this picture.
We have our sample space.
We have a first event, A1.
We have a second event, A2.
The third event, A3.
And so we keep continuing and we have an infinite sequence
of such events.
Then the probability of the union of these events, of
these infinitely many events, is the sum of their individual
probabilities.
The key word here is the word sequence.
Namely, these events, these sets that we're dealing with,
can be arranged so that we can talk about the first event,
A1, the second event, A2, the third one, A3, and so on.
To appreciate the issue that arises here and to see why the
word sequence is so important, let us consider the following
calculation.
Our sample space is the unit square.
And we consider a model where the probability of a set is
its area, as in the examples that we considered earlier.
Let us now look at the probability of the overall
sample space.
Our sample space is the unit square and the unit square can
be thought of as the union of various sets that consist of
single points.
So it's the union of subsets with one element each.
And it's a union taken over all the
points in the unit square.
Then we think about additivity.
We observe that these subsets are disjoint.
If we're considering different points, then we get disjoint
single element sets.
And then an additivity property would tells us that
the probability of these union is the sum of the
probabilities of the different single element subsets.
Now, as we discussed before, single element subsets have 0
probability.
So we have a sum of lots of 0s and the sum of 0s should be
equal to 0.
On the other hand, by the probability axioms, the
probability of the entire sample space
should be equal to 1.
And so we have established that 1 is equal to 0.
This looks like a paradox.
Is it?
The catch is that there is nothing in the axioms we have
introduced so far or the properties we have established
that would justify this step.
So this step here is questionable.
You might argue that the unit square is the union of
disjoint single element sets, which is the case that we have
in additivity axioms.
But the additivity axiom only applies when we have a
sequence of events.
And this is not what we have here.
This is not a union of a sequence of
single element sets.
In fact, there is no way that the elements of the unit
square can be arranged in a sequence.
The unit square is said to be an uncountable set.
This is a deep and fundamental mathematical fact.
What it essentially says is that there are two kinds of
infinite sets.
Discrete ones or in formal terminology countable.
These are sets whose elements can be arranged in a sequence,
like the integers.
And also uncountable sets, such as the unit square or the
real line, whose elements cannot be
arranged in a sequence.
If you're curious, you can find the proof of this
important fact in the supplementary materials that
we are providing.
After all these discussion, you may now have legitimate
suspicions about the models we have been looking at.
Is area a legitimate probability law?
Does it even satisfy countable additivity?
This question takes us into deep waters and has to do with
a deep subfield of mathematics called Measure Theory.
Fortunately, it turns out that all is well.
Area is a legitimate probability law.
It does indeed satisfy the countable additivity axiom as
long as we only deal with nice subsets of the unit square.
Fortunately, the subsets that arise in whatever we do in
this course will be "nice".
Subsets that are not nice are quite pathological and we will
not encounter them.
At this stage we are not in a position to say anything more
that would be meaningful about these issues because they're
quite complicated and mathematically deep.
We can only say that there are some serious mathematical
subtleties.
But fortunately, they can all be overcome
in a rigorous manner.
And for the rest of this class, you can just forget
about these subtle issues.