So this idea of Naive Bayes, where you have
a network that has a label producing or, or
conditionally producing a bunch of attribute values, is just
a really cool and powerful idea. So one of the,
one of the issues is that, even though inference
in general is, is is a very difficult problem it's
NP hard. To work out what these probabilities are,
when you have a naive Bayes structure, it's cheap. It's,
it's the formula that we had on the previous slide. The
number of parameters that you need to write down, again even if
you have a very large number of variables, it's not exponential
in the number of variables, it's just linear. There's, two probabilities for
each of the attributes and one probability for the class. We
can actually estimate these probabilities. So so far, we've only been talking
about Bayes Nets in, in not in a learning setting, but in
a setting where we just write down what all the numbers are.
We can actually very easily estimate these parameters. How would we
do that? Well the odd, the easy way to do it, is
you count. When you're trying to estimate the probability of a particular
attribute value given a class, it's really just in your, in your
labeled data. How often do you have an example that has an
attribute value in that class, and then divide by the number of
times you had that class at all, and that gives you the
conditional probability. So this is, you know in, in the case of
infinite data this is actually going to give you exactly the right
number. It also connects this notion of inference that we've been
talking about with classification. Which is mostly what this, this mini
course has been about. So, that's really great to have a connection,
it actually allows us to do all kinds of interesting things
like instead of only generating what the labels are, we can actually
generate what attributes are. We can do inference on, in, in
any of these directions. And it turns out it's wildly successful empirically.
So, my understanding is that Google uses a tremendous amount of Naive Bayes
classification in what they do. If you have enough data you can estimate
these values really well, and Naive Bayes is just remarkably good. So yeah
so it's like unclear why we'd even have any other algorithms, right Charles?
>> Well, there's no free lunch. But I, I gotta say I, I you know
there's this as a famous man once said it works in practice but doesn't work
in theory. And I'm trying to figure out how this can possibly work.
So I noticed it's called Naive Bayes. And, I think I know why now.
>> Alright.
>> One is that it's well it's naive and
in fact painfully ridiculous to believe that the bayesian
net that you wrote up there in the upper
right-hand corner represents the real world most of the time.
>> Hm, I see, and why is that?
>> Well because
what the, what the network says is that all
of the attributes are conditionally independent giving that you know
the label, that just can't be true. We talked
about this before where we were using Bayesian inference to,
to derive the sum of squared errors that it
makes a very strong assumption about where your errors come
from and an even stronger assumption about where your errors
don't come from. So you're not modeling any of the
interrelationships, between, the different attributes and
that just doesn't seem right. So, one
question I have. I have two, we'll save the second one though. One question
I have is, how in the world can it possibly be the case
that this works in practice? Hm, that's a good question. It does. Moving on.
>> [LAUGH] No, that's not satisfying.
>> No?
>> How about, how about I give it a guess? Okay?
>> Alright.
>> Now,
now that I yelled at you, why don't I, why don't I give it a guess.
>> [LAUGH]
>> I think it comes back to one of
the conversation we had in the previous slide. When
I was saying well we don't have to care.
We don't care about probabilities. And you said we
do care about probabilities because of the question your
asking and that was fair. But once were down
to classification. The probabilities really don't matter. Right all
that matters is that you get the right answers.
So its okay I guess if the probabilities you
get are long. So long as they're sort, sort
of in the right direction right. That you end
up getting the, the right label as a result.
>> Yeah, that's a good point. That in fact
we're introducing this idea in the context of, of
Bayesian Inference it might actually not be so good
at that even if it is particularly good at classification.
>> Oh, oh actually I think I have a good example so,
so here, here write this down. So let's imagine there are four
actually you can use the network that you have up there okay
>> Good.
>> So let's say that the first attribute, I'm just going to call it A
and the second attribute I'm going to call B, and let's say we're really, we're
really lucky and our naïve assumption is
right and they really are conditionally independent. But
let's say the third attribute, is actually
just another way of writing down A, and
the fourth attribute is just another way of writing down
B. So, clearly there are interrelationships between the attributes, right?
>> The third attiribute is the first one, the fourth attribute is
the second one. There's not way around that. And so you'd think
Naive Bayes would fail. But, actually, looking at your equation right below
there where you're doing counting, I actually think, it'll work just fine.
>> Why?
>> Because all you're really doing
is double counting the sort of weight of
attribute A, but you're also double counting the
weight of attribute B and they'll cancel each
other out. And you'll get the right answer.
>> When you do the arg max, but these
>> When you do the arg max
>> You get bad probabilities. The probabilities
end up being kind of squared of what
they should, what they're supposed to be.
But that's okay because the ordering is preserved.
>> Right, exactly. And so, even if you're unlucky and
the fourth attribute wasn't B but it was something else, C.
It doesn't matter if you double count A as
long as it still gives you the right label.
And you can imagine that if you have weak
inner relationships or, you know, you have enough attributes and,
and so on that you would still get the
right, you know, yes this is the correct label, even
if you've got the probabilities wildly wrong. Okay, so
I'm willing to believe that that could happen in practice.
>> Okay.
>> So in fact, my guess is that Naive Bayes believes
it's answer too much. But it doesn't matter if it happens to be right.
>> All right and did you have other issues with it?
>> So the second problem I have actually boils down to that
equation you wrote there. So it's really nice and neat that you
can compute the probabilities of seeing an attribute, given a value by
just doing counting. But, I don't have an infinite amount of data, right?
>> Not on a bad day, no.
>> No. Or even on a good day I usually
don't have an infinite amount of data. So what if
I'm unlucky enough that for some particular attribute value,
I have never seen it paired with that label, V.
>> Right. So then, that means this numerator will be zero
>> Right.
>> So.
>> Well that numerator is zero, but since
the computation involves a product by just having
one attribute value that I've never seen before.
I'm going to end up saying well the probability
of that entire product of seeing that value given
a set of attributes is also going to be zero. So
one unseen attribute, basically says it doesn't matter what else
is going on. Which seems a little weird, right? You,
you, you'd think that you, if all the other
attributes are screaming yes, yes, yes, yes, it should be
positive. But just because you haven't happened to have seen
any examples of some other one single attribute, that shouldn't
be enough to do veto.
>> Good point, so in fact that's not what
people often do. People will often, what they call smooth
the probabilities, by essentially initializing the count, so that
nothing is zero, everything has a tiny little non-zero value
in it. And there's, there's smarter and less smart
ways of doing that, but no, you're absolutely right. That,
that is, that zeroing out problem is a real
thing and you have to be a little bit careful.
>> Hey, hey I just had a thought. So,
if you, you have to do that, because if you don't do
that, then you're believing your data too much. You're kind of over fitting.
>> Ooh. Over fitting comes up again.
>> Oh, oh, it's okay, okay so, so, so, so, so bear with
me on this Michael. So if you're over fitting by believing the data,
and you're fixing it by smooth, I usually spell it with a V,
but whatever. If you, you'd think that by being smooth, then you're making
an assumption. There's a kind of inductive
bias, right? Your'e, you're saying that I go
in with the assumption that they're sort
of all things are at least mildly possible.
>> Good.
>> Huh.
>> Yea, that's, that's right.
>> Okay, Naive Bayes is cool, you've convinced me.
>> Nice.