Let X be a random variable, and let g be a function.
We know that if g is linear, then the expected value
of the function is the same as that linear function
of the expected value.
On the other hand, we know that when g is nonlinear,
in general, these two quantities will not be related.
But there is a special case in which
we can establish some relation between these two quantities
in the form of an inequality.
This is Jensen's inequality, which we're going to develop.
Jensen's inequality applies to the special case
where g is a convex function.
I'm going to define more precisely what it
means to be convex shortly.
But in terms of a picture, it's a function
that has a shape of this kind.
So it tends to curve upwards.
So let us look at a simple example.
Suppose that X is a random variable that
can take two values with equal probability.
So these two values have both probability 1/2.
And this is our function, g of x.
Since they take the two values with equal probability,
the expected value is going to be in the middle.
So this here is the expected value of X.
And in particular, this value here
is going to be g of the expected value of X. Now,
the random variable g of X will take this value
with probability 1/2, and it's going
to take that value with probability 1/2.
What is the average of g of X, the expected value of g of X?
It's going to be 1/2 of this value
plus 1/2 of this value, which you can find as follows.
If you draw a straight line, it's going to be this much.
So this quantity here is the expected value of g of X.
And we see that in this example, the expected value of g of X
is above the value of g evaluated
at the expected value of X.
So this is what Jensen's inequality says,
but for a more general distribution
for the random variable X. Let us now step back and define
more precisely what it means for a function to be convex.
The most general definition is the following.
If I take any two points, x and y,
and I take some number p between 0 and 1.
So in that case, the number px plus 1 minus py
is a weighted average of x and y--
so it's somewhere in the middle.
And if I look at the value of my function
at that particular point-- so this value here
corresponds to this-- this is less
than or equal to the weighted average of the values of g
of x and g of y.
So this is g of x.
This is g of y.
This value here is the weighted average of the two values.
So this quantity here is p times g of x, which is this value,
plus 1 minus p, g of y.
Convexity means that this value is below that value.
Or in other words, whenever I take two points on this curve
and join them by a segment, then the function
lies underneath that segment.
This is one possible definition.
Now, in terms of a picture, we see
that a convex function tends to curve upwards.
This means that the derivative or the slope of the function
keeps increasing.
An increasing slope means that the second derivative
of that function is non-negative.
And that could be an alternative definition of convexity.
It turns out that if you have a function that's
twice differentiable, these two definitions are equivalent.
On the other hand, the first definition
is a little more general, because it also
applies to functions that are not smooth.
So for example, the function absolute value of x
is a convex one.
But it's not differentiable at zero.
Finally, there's another way of defining convexity,
and it is the following property,
again for differentiable functions.
What it says is the following.
If I fix a certain point, c-- to use the same diagram
let's say that this is c-- this value here
is going to be g of c.
I look at the derivative of that function and take
this quantity, which is the derivative times how far I
am going.
This quantity here is a first-order Taylor series
approximation of my function.
So it corresponds to this black line.
What this condition says is that the function
lies on top of that tangent line to my function.
It is not too difficult to show that this condition,
non-negative second derivatives, implies this condition.
What you do is that you write down the second order Taylor
approximation of your function g.
And then because the second order term
is going to be non-negative because of that condition,
you're going to get an inequality in this form.
But in any case, this inequality is pretty intuitive.
And we could take this one just as our definition
of convexity-- that is, a function is convex
if it has the property that whenever I draw
a tangent to my curve, the function lies
on top of this linear function.
So now let's move back to probability.
Suppose that g is convex.
This is true for every x.
So in particular, if I have my random variable,
capital X, no matter what my random variable is,
we're going to get this kind of inequality, no matter what.
And here what I they left blank is c.
Now, c is a number.
This is true for any number.
So in particular, it's true if I use as my number
the expected value of X. So we have this inequality
that's true now in terms of random variables.
No matter what capital X happens to be, this will be valid.
And now let us take expectations of both sides.
What we obtain is that the expected value of g of X
is larger than-- now, this is a number,
so the expected value of that number
is the number itself plus the expected value of this term.
This quantity is a number.
The expected value of X minus the expected value of X
is equal to 0.
And we have established this particular fact,
which is true for any convex function.
So this is Jensen's inequality.
Let's apply it to some examples.
Let's consider the function g, which
is the quadratic function.
Clearly, this is a convex function.
It has this kind of shape.
And the second derivative of this function is positive.
That's another way of verifying it.
Jensen's inequality is going to tell us
something about the expected value of X squared.
Now, for this expectation, we already
know that this is equal to the variance of X plus the square
of the expected value.
And since the variance is always non-negative,
we obtain this inequality.
This is consistent with Jensen's inequality.
Jensen's inequality tells us that E of g of X,
with g the quadratic function, is
larger than or equal to the square-- that
is, g of the expected value.
So for the case of the square function,
Jensen's inequality did not tell us anything
that we didn't know.
But it's nice to confirm that it is consistent.
But we could use Jensen's inequality
in another setting where the answer might not be as obvious.
For example, take the function X to the fourth.
This is also a convex function.
And Jensen's inequality is going to tell us
that the fourth power of the expected value is less than
or equal to the expected value of X to the fourth.
Another case of a convex function
is the negative logarithm.
Remember that the logarithmic function
has a shape of this kind, which curves the opposite way.
So it's called a concave function.
But if you take the negative of this function,
then you're going to get something that is convex.
So by applying Jensen's inequality
to this setting, what we obtain is
that g, which is minus log of the expected value of X,
is less than or equal to the expected value of minus log X.
And then we can remove the minus signs from both sides.
And that is going to reverse the inequality.
And we will obtain that the logarithm of the expected value
of X is larger than or equal to the expected value of log X.
So in this case for the logarithmic function,
the inequality goes in the opposite direction.
The reason is that the logarithmic function
is a concave function, not a convex one.
And by arguing similar to this example,
a concave function is the negative of a convex function.
And for concave functions, Jensen's inequality still
holds, but the inequality goes the opposite way.
Jensen's inequality turns out to be quite useful.
In many cases, we want to say something
about the expected value of g of X,
and Jensen's inequality allows us to do that.