>> All right, Michael. So I, all I have written up
here for you is, are a maximum a posteriori equation, right?
So the best hypothesis is the one that maximizes this expression.
Nothing new, right? So I want to do a little trick, the
same trick that you did before. So, you noticed that when
we had E to the something, that we could use the
natural log on E to get rid of everything. So I
am going to try to do the same thing here. In the
nat, why did the natural log work again?
>> Well, it's the inverse of the E, but it let us turn products into sums.
>> Right. And the other reason it worked is because it's a.
>> Oh, it's monotonic.
>> Right, it's a monotonic function and
so it doesn't change the argmax. So, I'm
going to do the log of both sides here. But this time I'm going to do
log base 2, for no particular reason other
than it'll turn out to help later. So,
I'm just going to take the log of
this entire expression, which, because it turns products
into sums, gives me this. And by the way for
those of you who haven't noticed, I drew in a little
bit of notation here. When you write just LG, it's just
log base 2. Okay, so, we agree that the answer to
this equation and the answer to this equation is the same.
And now I'm going to do one other little trick, exactly
the trick that you used before. I'm going to change my
max into a min, by simply multiplying everything by minus 1.
>> Okay, I don't quite see where you're going here.
>> But you agree that
we haven't changed the answer.
>> I agree that we haven't changed the answer.
>> Okay. Do a log in there, do a minus
sign in there that took us from a max to a
min, but I haven't changed the answer. Now, do you recognize anything
about these expressions? I'll give you a hint. Information theory.
>> Okay. So, information theory is usually entropy,
which is like sum of P log P stuff.
>> Right.
>> I'm not seeing that.
>> Well, there you, there's your log and there's your P.
>> Sure. [LAUGH]
>> [LAUGH]
>> It's not P times that though.
>> That's true. But we know from information theory, based exactly on this
notion of entropy, that the optimal code for some event with probability P has
length minus log base 2 of P. So, that just comes straight out of information
theory. That's where all the entropy stuff comes from. Okay. So, if we
have some event that has some particular probability P of happening,
the best code for it has this structure, minus log of P.
>> Okay.
>> So, if we take this fact that we know, and
we apply it to here, what is this actually saying? This is
saying that, in order to find the maximum a posteriori hypothesis,
we want to some how minimize two terms that can be described
as lengths.
>> Okay. I can see that.
>> So my question to you is, given that this definition over here, that an
event with probability P has some length minus
log P, what is this the length of?
>> So that would be the length of
the probability of the data given the hypothesis.
>> Mm-hm.
>> And the length of the hypothesis, or the probability of the hypothesis.
>> Well no, it's just the length of that hypothesis.
>> Oh, because the event is what has the length.
Oh, I see. So it's the length of the data,
given the hypothesis, and the length of the hypothesis.
>> Right. So let's write that out.
>> But I was just doing, like, pattern matching
there. It's not clear to me what a length
of a hypothesis is. Hypotheses are functions. I don't
know how to take a tape measure to a function.
>> That's fair. So this is the length of the hypothesis. Right?
>> Yep.
>> So, you said you don't know what that means. But, let's think about
that out loud for a moment. What does it mean to have a length of
a hypothesis? That's really sort of the number of
bits you need to describe a particular hypothesis, right?
>> Okay.
>> Okay. And in fact, that's exactly what it means.
That's why we use log base 2. So, if we want
to minimize the length of a hypothesis, what does that mean,
the number of bits that we need to represent the hypothesis?
>> The number of bits that we need to represent the hypothesis is, I guess,
in some representation, or, so in this case
I guess it would be some optimal representation.
We are taking all the different hypotheses and writing them out. The ones
that are more likely have a higher P of H, because that's the
prior. And those are going to have smaller lengths than the optimal code.
And the ones that are less common are going to have longer codes.
>> Well, let's make it more concrete.