We will now take a step towards abstraction, and
discuss the issue of
convergence of random variables.
Let us look at the weak law of large numbers.
It tells us that with high probability, the sample mean
falls close to the true mean as n goes to infinity.
We would like to interpret this statement by saying that
the sample mean converges to the true mean.
However, before we can make such a statement, we should
first define carefully the word "converges." And we need
a notion of convergence that refers to convergence of
random variables.
Here's a definition.
Suppose that we have a sequence of random variables
that are not necessarily independent.
We say that this sequence of random variables converges in
probability--
that's a particular notion of convergence we're defining.
It converges to a certain number if the
following is true--
no matter what epsilon is, as long as it is a positive
number, the probability that the random variable falls far
from this number--
that is, epsilon or further away from that number--
that probability converges to 0 as n increases.
That is, as n increases, there is higher and higher
probability that Yn will be close to this
particular number a.
This is the notion of convergence
that we have defined.
And notice that this notion of convergence corresponds
exactly to what is happening in the weak
law of large numbers.
And so in particular, we can describe the weak law of large
numbers as saying that Mn, the sample mean, converges to mu
as n goes to infinity, but in a particular sense--
in the sense of convergence in probability.
Let us now try to understand a little better what convergence
in probability really amounts to.
And we will do that by making a comparison with the ordinary
notion of convergence of real numbers.
When we're dealing with convergence of numbers, we
start with a sequence of numbers, and we are interested
in the statement that this sequence converges to a
certain limit.
What does that mean?
What we mean is that the elements of the sequence
eventually--
that is, when n is large enough--
will get and stay arbitrarily close to this particular
number a, which is the limit.
In terms of a picture, here is a, the limit.
Here is n.
We take a small band around this number a.
And what we require is that the elements of the sequence
eventually get within this band around the number a.
They might get outside the band, get inside again.
But eventually--
that is, after some time--
the elements of the sequence will only
stay inside this band.
Now to translate this into a more formal mathematical
statement, which is the mathematical definition of the
notion of convergence, we have the following--
if I give you some epsilon, epsilon could be
a very small number.
I form a band around a that goes from a minus epsilon to a
plus epsilon.
What I want is that there exists a certain time, n0--
in this picture, n0 would be here--
such that for all times after n0, our sequence stays within
epsilon from a.
That is, our sequence stays inside this band.
Now let us move to the case of random variables, and see what
convergence in probability is talking about.
Here, instead of a sequence of numbers, we have a sequence of
random variables.
And we're interested in the meaning of the convergence of
the sequence of random variables to
a particular number.
In words, what this means is that if I fix a certain
epsilon, as in this picture, then the probability that the
random variable falls outside this band converges to 0.
So the picture would be as follows.
We have, again, our limit.
We fix some epsilon, which could be an
arbitrarily small number.
For any fixed choice of epsilon, we take this band,
and for any given n, we look into the probability that our
random variable falls inside that band.
So if I am to draw the distribution of our random
variable, a distribution might be something like this--
so there is a certain probability that it falls
outside this band.
But when n becomes large, this probability of falling outside
this band becomes very small.
So the probability of falling outside the band becomes tiny.
So the bulk of the distribution--
that is, most of the probability--
is concentrated inside this band.
And this is true, no matter how small epsilon is.
If I take a much narrower band around a, I still want all of
the probability to be eventually
concentrated inside that band.
Of course, it might take longer.
It might take a larger value of n, but I want that when n
is very large, the bulk of the distribution is, again,
concentrated inside this narrow band.
So in words, convergence in probability means that almost
all of the probability mass of the random variable Yn, when n
is large, that probability mass get concentrated within a
narrow band around the limit of the random variable.
We finally point out a few useful properties of
convergence in probability that parallel well-known
properties of convergence of sequences.
Suppose that we have a sequence of random variables
that converges in probability to a certain number a, and
another sequence that converges in probability to
some other number b.
We do not make any assumptions about independence.
We do not assume the Xn's are independent of each other.
We do not assume that the sequence of Xn's is
independent of Yn.
We then have the following properties--
if g is a continuous function, then the function of the
random variables converges to the function of the number.
So for example, the sequence of random variables Xn squared
is going to converge to a squared.
Another fact is that the sum of these two random variables
converges to the sum of their limits.
Both of these properties are analogous to what happens with
ordinary convergence of numbers.
And they tell us that, in some sense, convergence in
probability is not a very different notion.
We will not prove those properties at this point, but
they're useful to know.
However, there's an important caveat.
Xn might converge to a certain number in probability.
However, the expected value of Xn does not necessarily
converge to that same limit.
So convergence of random variables does not imply
convergence of expectations.
And we will be seeing an example where this convergence
does not take place.