The basic function underlying a normal distribution, aka a Gaussian,
is e to the negative x squared. But you might wonder "Why this function?" Of all
the expressions we could dream up that give you some symmetric smooth graph with mass
concentrated towards the middle, why is it that the theory of probability
seems to have a special place in its heart for this particular expression?
For the last many videos, I've been hinting at an answer to this question
and here we'll finally arrive at something like a satisfying answer. As a quick refresher on
where we are - a couple videos ago, we talked about the Central Limit Theorem
which describes how as you add multiple copies of a random variable, for example
rolling a weighted die many different times or letting a ball bounce off of a peg repeatedly,
then the distribution describing that sum tends to look approximately like a normal distribution.
What the Central Limit Theorem says is as you make that sum bigger and bigger under
appropriate conditions, that approximation to a normal becomes better and better.
But I never explained why this is actually true, we only talked about what it's claiming.
In the last video, we started talking about the math involved in adding two
random variables. If you have two random variables each following some distribution,
then to find the distribution describing the sum of those variables, you compute
something known as a Convolution between the two original functions. And we spent a lot of time
building up two distinct ways to visualize what this convolution operation really is.
Today, our basic job is to work through a example which is to ask what happens
when you add two normally distributed random variables which, as you know by now, is the
same as asking what do you get if you compute a convolution between two Gaussian functions.
I'd like to share an especially pleasing visual way that you can think about this calculation
which hopefully offers some sense of what makes the E to the negative X squared function special
in the first place. After we walk through it, we'll talk about how this calculation is one of
the steps involved in proving the Central Limit Theorem, it's the step that answers the question
of why a Gaussian and not something else is the central limit. But first, let's dive in.
The full formula for a Gaussian is more complicated than just e to the negative
x squared. The exponent is typically written as negative one-half times X divided by Sigma
squared, where Sigma describes the spread of the distribution. Specifically the standard
deviation. All of this needs to be multiplied by a fraction on the front which is there to
make sure that the area under the curve is one making it a valid probability distribution.
And if you want to consider distributions that aren't necessarily centred at zero,
you would also throw another parameter, µ into the exponent like this. Although
for everything we'll be doing here, we just consider centred distributions.
Now, if you look at our central goal for today, which is to compute a convolution
between two Gaussian functions, the direct way to do this would be to take the definition of
a convolution - this integral expression we built up the last video and then to
plug in for each one of the functions involved the formula for a Gaussian.
It's kind of a lot of symbols when you throw it all together but more than anything working
this out is an exercise in completing the square and there's nothing wrong
with that. That will get you the answer that you want but of course you know me,
I'm a sucker for visual intuition and in this case, there's another way to think about it
that I haven't seen written about before that offers a very nice connection to other aspects
of this distribution like the presence of pi and certain ways to derive where it comes from.
And the way I'd like to do this is by first peeling away all of the constants
associated with the actual distribution and just showing the computation for the simplified form;
e to the negative x squared. The essence of what we want to compute
is what the convolution between two copies of this function looks like.
If you remember, in the last video, we had two different ways to visualize convolutions
and the one we'll be using here is the second one involving diagonal slices. And as a quick
reminder of the way that worked, if you have two different distributions that are described
by two different functions, F and G, then every possible pair of values that you might get when
you sample from these two distributions can be thought of as individual points on the xy plane.
And the probability density of landing on one such point, assuming independence,
looks like f(x) times g(y). So what we do is we look at a graph of
that expression as a 2-variable function of x and y which is a way of showing the
distribution of all possible outcomes when we sample from the two different variables.
To interpret the convolution of f and g evaluated on some input s, which is a way of saying how
likely are you to get a pair of samples that adds up to this sum, s. What you do is you look at a
slice of this graph over the line x plus y equals s and you consider the area under that slice.
This area is almost but not quite the value of the convolution at s.
For a mildly technical reason, you need to divide by the square root of two.
Still, this area is the key feature to focus on. You can think of it as a way to combine
together all the probability densities for all of the outcomes corresponding to a given sum.
In the specific case where these two functions look like e to the negative
x squared and e to the negative y squared, the resulting 3D
graph has a really nice property that you can exploit. It's rotationally symmetric.
You can see this by combining the terms and noticing that it's entirely a function of x
squared plus y squared. And this term describes the square of the distance
between any point on the xy-plane and the origin. So in other words,
the expression is purely a function of the distance from the origin.
And by the way, this would not be true for any other distribution. It's a property that
uniquely characterizes bell curves. So, for most other pairs of functions, these diagonal slices
will be some complicated shape that's hard to think about and honestly, calculating the area
would just amount to computing the original integral that defines a convolution in the
first place. So, in most cases, the visual intuition doesn't really buy you anything.
But in the case of bell curves, you can leverage that rotational symmetry.
Here, focus on one of these slices over the line x + y = s for some value of s.
And remember, the convolution that we're trying to compute is a function of s. The
thing that you want is an expression of s that tells you the area under this slice.
Well, if you look at that line, it intersects the x-axis at (s, 0) and the y-axis at (0, s) and a
little bit of Pythagoras will show you that the straight line distance from the origin
to this line is s divided by the square root of two. Now, because of the symmetry,
this slice is identical to one that you get rotating 45° where you'd find
something parallel to the y-axis the same distance away from the origin.
The key is that computing this other area of a slice parallel to the y-axis is much
much easier than slices in other directions because it only involves
taking an integral with respect to y. The value of x on the slice is a constant.
Specifically, it would be the constant s divided by the square root of two.
So, when you're computing the integral, finding this area, all of this term here behaves like it
was just some number and you can factor it out. This is the important point - all of the stuff
that's involving s is now entirely separate from the integrated variable. This remaining
integral is a little bit tricky. I did a whole video on it. It's actually quite famous but you
almost don't really care. The point is that it's just some number. That number happens to be the
square root of pie but what really matters is that it's something with no dependence on s.
And Essentially, this is our answer. We were looking for an expression for the area of
these slices as a function of s and now we have it. It looks like e to the negative s squared
divided by two, scaled by some constant. In other words, it's also a bell curve,
another Gaussian just stretched out a little bit because of these two in the exponent.
As I said earlier, the convolution evaluated at s is not quite this area. Technically,
it's this area divided by the square root of two. We talked about it in the last video but
it does really matter because it just gets baked into the constant. What really matters
is the conclusion that a convolution between two Gaussians is itself another Gaussian.
If you were to go back and reintroduce all of the constants for a normal distribution with a
mean zero and an arbitrary standard deviation, sigma, essentially identical reasoning will
lead to the same square root of two factor that shows up in the exponent and out front and it
leads to the conclusion that the convolution between two such normal distributions is
another normal distribution with a standard deviation square root of two times sigma.
If you haven't computed a lot of convolutions before, it's worth emphasizing this is a very
special result. Almost always, you end up with a completely different kind of function. But here,
there's a sort of stability to the process. Also, for those of you who enjoy exercises, I'll leave
one up on the screen for how you would handle the case of two different standard deviations.
Still, some of you might be raising your hands and saying,
"What's the big deal?" I mean, when you first heard the question "What do you get
when you add two normally distributed random variables?" you probably even
guessed that the answer should be another normally distributed variable. After all,
what else is it going to be? Normal distributions are supposedly quite common, so, why not?
And you could even say that this should follow from the Central Limit Theorem. But that would
have it all backwards. First of all, the supposed ubiquity of normal distributions
is often a little exaggerated. But to the extent that they do come up,
it is because of the Central Limit Theorem. But it would be cheating to say the Central Limit Theorem
implies this result because this computation we just did is the reason that the function at the
heart of the Central Limit Theorem is a Gaussian in the first place and not some other function.
We've talked all about the Central Limit Theorem before but essentially it says if you repeatedly
add copies of a random variable to itself, which mathematically looks like repeatedly computing
convolutions against a given distribution, then after appropriate shifting and rescaling,
the tendency is always to approach a normal distribution. Technically,
there's a small assumption - the distribution you
start with can't have infinite variants but it's a relatively soft assumption.
The magic is that for a huge category of initial distributions,
this process of adding a whole bunch of random variables drawn from that
distribution always tends towards this one universal shape - a Gaussian. One
common approach to proving this theorem involves two separate steps. The first step
is to show that for all the different finite variants distributions you might start with,
there exists a single universal shape that this process of repeated convolutions tends towards.
This step is actually pretty technical. It goes a little beyond what I want to
talk about here. You often use objects called Moment-generating function that
gives you a very abstract argument that there must be some universal shape but
it doesn't make any claim about what that particular shape is, just that everything
in this big family is tending towards a single point in the space of distributions.
So then step number two is what we just showed in this video - prove that the convolution of
two Gaussians gives another Gaussian. What that means is that as you apply this process
of repeated convolutions, a Gaussian doesn't change. It's a fixed point. So, the only thing
it can approach is itself and since it's one member in this big family of distributions
all of which must be tending towards a single universal shape, it must be that universal shape.
I mentioned at the start how this calculation, step two, is something that you can do directly
just symbolically with the definitions but one of the reasons I'm so charmed by a geometric argument
that leverages the rotational symmetry of this graph is that it directly connects to a few things
that we've talked about on this channel before. For example, the Herschel-Maxwell derivation of
a Gaussian, which essentially says that you can view this rotational symmetry as the defining
feature of the distribution. That it locks you into this e to the negative x squared form.
And also as an added bonus, it connects to the classic proof for why pi shows
up in the formula. Meaning we now have a direct line between the presence and
mystery of that pi and the Central Limit Theorem. Also on a recent patreon post,
the channel supporter Daksha Vaid-Kwinter brought my attention to a completely different
approach I hadn't seen before which leverages the use of entropy and again,
for the theoretically curious among you, I'll leave some links in the description.
By the way, if you want to stay up to date with new videos and also any other projects
that I put out there like the summer of math exposition, there is a mailing list. It's
relatively new and I'm pretty sparing about only posting what I think people will enjoy. Usually,
I try not to be too promotional at the end of videos these days but if you are interested in
following the work that I do, this is probably one of the most enduring ways to do so.