One situation where covariances show up
is when we try to calculate the variance
of a sum of random variables.
So let us look at the variance of the sum
of two random variables, X1 and X2.
If the two random variables are independent,
then we know that the variance of the sum
is the sum of the variances.
Let us now look at what happens in the case where
we may have dependence.
By definition, the variance is the expected value
of the difference of the random variable we're
interested in from its expected value, squared.
And now we rearrange terms here and write
what is inside the expectation as follows.
We put together X1 with the term minus the expected value of X1
and then X2 together with negative the expected value
of X2.
So now we have the square of the sum of two terms.
We expand the quadratic to obtain
expected value of the square of the first term
plus the square of the second term plus 2 times a cross term.
And what do we have here?
The expected value of the first term
is just the variance of X1.
The expected value of this second term
is just the variance of X2.
And finally, the cross term, the expected value of it,
we recognize that it is the same as the covariance of X1
with X2.
And we also have this factor of 2 up here.
So this is the general form for the variance
of the sum of two random variables.
In the case of independence, the covariance is 0,
and we just have the sum of the two variances.
But when the random variables are dependent,
it is possible that the covariance will be non-zero,
and we have one additional term.
Let us now not generalize this calculation.
Here is for reference and comparison
the formula for the case where we add two random variables.
But now let us look at the variance
of the sum of many of them.
To keep the calculation simple, we're
going to assume that the means are zero.
But the final conclusion will also
be valid for the case of non-zero means.
Since we have assumed zero means,
the variance is the same as the expected value
of the square of the random variable involved,
which is this one.
And now we expand this quadratic to obtain the expected value
of: we will have a bunch of terms
of this, where i ranges from 1 up to n.
And then we will have a bunch of cross terms of the form Xi, Xj.
And we obtain one cross term for each choice of i
from 1 to n and for each choice of j from 1 to n,
as long as i is different from j.
So overall here, this sum will have n squared minus n terms.
Now, we use linearity to move the expectation
inside the summation.
And so from here, we obtain the sum
of the expected value of Xi squared, which
is the same as the variance of Xi,
since we assumed zero means.
And similarly here, we're going to get this double sum over i's
that are different from j of the expected value of Xi, Xj.
And in the case of 0 means again, this
is the same as the covariance of Xi with Xj.
And so we have obtained this general formula
that gives us the variance of a sum of random variables.
If the random variables have 0 covariances,
then the variance of the sum is the sum of the variances.
And this happens in particular when the random variables
are independent.
For the general case, where we may have dependencies
and non-zero variances, then the variance of the sum
involves also all the possible covariances
between the different random variables.
And let me finally add that this formula is also
valid for the general case where we do not
assume that the means are zero.
And the derivation is very similar,
except that there's a few more symbols
that are floating around.