Okay, Michael, so we've gotten through that quiz and you see
that Bayes' rule actually gives you some information. It actually helps
you make a decision. So I'm going to suggest that, that whole
exercise we went through was actually our way of walking through
an algorithm. So here's a particular algorithm that follows from what
we just did. And let me just write that down for
you. All right, so here's the algorithm, Michael, so it's very
simple. For each H in H, that is, each candidate hypothesis
in our in our hypothesis space, simply
calculate the probability of that hypothesis given
the data W which we know is equal to the probability of the data given
that hypothesis times the prior probability of
the hypothesis, divided by the probability of
the data. And then simply output whichever
hypothesis has maximum probability. Does that make sense?
>> Yeah.
>> Okay, so I do want to point out that since
all we care about is computing the argmax, as before,
we don't actually ever have to compute that little bit
so, and that's a good thing because we don't always know
what the prior probability on the data is, so we
can ignore it for the purposes of finding the maximal hypothesis.
>> So the place you removed it from, it seems like that's not actually valid,
because it's not the case that the probability of h given d equals, it's the
probability of d given h times the probability of h. It just means that we
don't care what the probability is when
we go to compute the argmax. That's right,
so, in fact, it's probably better to say that I'm going to approximate
the probability hypothesis given the data
by just calculating the probability of the
data given the hypothesis times the probability of the hypothesis and just go
ahead and ignore the denominator. Precissely
because it doesn't change hte maximal age.
>> Yeah, so it's, it's nice that that goes away.
>> Right, because it's hard to know, often what
the prior, what the prior probability over the data is.
>> It would
be nice if we didn't have to worry about the other one, either.
>> Which other one?
>> The probability of h, where's that coming from?
>> right, so where does that come from? So that's
a deep philosophical question. Sometimes it's just something you believe, and
you can write down. And sometimes it's a little harder. And
that's actually good that you bring that up. When we compute,
our probabilities this way so it's actually got a name,
it's the MAP or the maximum a posteriori hypothesis and that
makes sense, it's the biggest posterior given all of your
priors. But you're right Michael that often it's just as hard
to say anything particular about your prior over the hypothesis
as it is to say something about your prior of the
data and, so it is very common to drop that. And,
in dropping that, we're actually computing the argmax over the probability
of the data given the hypothesis. And,
that is known as the maximum likelihood hypothesis.
>> I guess you can't call it the maximum
A priori hypothesis, because then it would also be MAP.
>> Exactly, although I've never thought about that before.
By the way, just just to be clear, we're
not really dropping this, in this case, what we
really said, is that, our prior belief is that
all hypotheses are equally likely. So we
have a uniform prior that is, the probability
of any given hypothesis is exactly the same
as the probability as any other given hypothesis.
>> I see, so you're saying if, if we assume
that they all are equally likely, then, the choice of
hypothesis doesn't change that term at all, the p of
h term, so it really is equivalent to just ignoring it.
>> Exactly, in some constant, we don't even have
to know what the constant is. But whatever it is,
it's the same everywhere and therefor it doesn't affect
the other terms or, in particular, affect the argmax computation.
>> So that's actually pretty cool right?
Once you think about what we just did.
We just took something that was very
hard. Computing the probability of a hypothesis given
the data and turned it into something
much easier that is... Computing the probability of
you seeing the data labels given a particular
hypothesis and it turns out that those are
effectively the same thing if you don't have
a strong prior. So that's really cool, so we're
done right? We now know how to find
the best hypothesis You're just finding the most likely
hypothesis or the most probable one and that
turns out to be the same thing as just
simply finding the hypothesis that best matches the
data. We're done its all, its easy. Everythings good.
>> So,the math seems very nice and pretty and easy
but is isn't it hiding a lot of work to actually
do these computations?
>> Well, sure well well look you know
how to do multiplication that's pretty easy right?
>> [LAUGH].
>> So I guess the only hard part
is we have to look at every single hypothesis.
>> Yeah, that's just a slight, little, you know, issue.
>> So, mathematically meaningful, but computationally questionable.
>> Hm.
>> So, the big point there, is that it's not practical. Well, unless the
number of hypotheses is really, really small. But as we know, a lot of
the hypotheses spaces that we care about, like, for example, linear separators,
are actually infinite. And so it's going to be very difficult to use
this algorithm directly. But despite all that, I think that there's still
something important that we get out of thinking about it this way
in just the same way that we get something important out of
thinking about vc dimension. Even if we're not entirely sure how to
compute it in some particular case. This really gives us a gold
standard, right? We have an algorithm,
at least a conceptual algorithm, that tells
us what the right thing to do would be if we're
capable of computing it directly. So, that's good because we can
maybe prove things about this and compare results that we get
from some Real live algorithms to what we might expect to
get but also it turns out it's pretty cute because it
helps us to say other things about what it is we
actually expect to learn. And I'm going to give you a couple
examples of those just to sort of prove my point, sound good?
>> Yeah.
>> Okay.