Hi, In this module, I'm going to talk about Laplace Smoothing
for guarding against overfitting.
So let's review maximum likelihood estimation
in Bayesian networks.
Remember last time, we had an example
of a two variable network, it's genre of a movie
and the rating of a movie, where their joint distribution
is given by probability of a genre times
probability of rating given.
And now, we don't know these parameters,
but we want to estimate them from data.
Suppose we gather five data points here.
And the way that maximum likelihood estimation works
is by counting and normalizing.
Parameters here are probability of g.
And for that, I'm going to count the number of times g goes up,
and normalize.
And for probability of r given g,
I'm going to look at the number of times
each of these configurations shows up.
And then I'm going to normalize each one, conditioned
on the value of g.
So if you look at these estimates,
you might notice that there's something funny going on here.
So the probability that these parameters
assigned to a rating of 2, given that there's a comedy, is 0.
It doesn't show up in this row at this table, which
means that it's 0.
So do we really believe this?
Just because we didn't see an example of a comedy being
rated as a 2, are we licensed to just
give it a probability of 0?
Well, that would be very closed minded.
So this is a case where maximum likelihood has [INAUDIBLE]
overfit to [INAUDIBLE].
There's a very simple way to fix this, called Laplace smoothing.
And the idea is that we're just going
to add a lambda, which is some positive value,
let's say 1, to each count.
Let's do a maximum likelihood with Laplace smoothing.
So the training data is the same as before.
And what we're going to do is, for each
of these local distributions, we're going to preset,
preload a 1, lambda, more generally, into this position.
And now I'm going to go through the training
and count as usual.
So I had 3.
And I had 2.
And then I'm going to normalize over these combined counts.
And same with the probability of r given g, for each
of these configurations.
So now I have to actually instantiate
all possible configurations.
I'm going to load a 1 into each of these counts.
And then I'm going to look at my training data.
And I'm going to add 2.
There's two d4's, 1 d5, 1 c1, and 1 c5.
Now, given these counts, I'm going
to normalize to get my probability emit,
look at all of the d's, add them up, normalize.
I get some d's here.
And look at all the case rows where
[INAUDIBLE],, I'm going to come [INAUDIBLE]..
So now, while we revisit our probability
estimate of r equals 2, given g equals c, this was 0 before.
But now, it's here, that's 1 over 7,
which is greater than 0.
So now, because we smooth these estimates,
now we have a little bit more probability
on even those outcomes that we've never
seen during training.
So the key idea behind maximum likelihood with Laplace
smoothing is as follows.
So we're going to go through each distribution
and partial assignment to the parents of a node
and the node itself.
And we're simply going to add lambda to the count.
Now we do maximum likelihood estimation as usual,
though we're going to go through the training data
and increment those counts, based on what we saw.
And then recount and normalize.
And that's it.
So the interpretation that we can place on Laplace smoothing
is it's like we hallucinated lambda occurrences
of each local assignment.
So sometimes, these lambda counts
are called pseudo counts.
Because they're not based on the data,
they're kind of made up or virtual counts.
So you can think about pretending
you saw some examples before you saw data,
and then doing maximum likelihood estimation.
So how much should lambda be?
How much smoothing should we have?
And how does it interact with the data?
There's two observations that I want to make.
First is that the more you smooth,
which means that the more, the bigger the lambda is,
the closer you're going to push the probability
estimates closer to the uniform distribution.
So for example, if I just smooth with lambda equals 1/2,
and I observe only a d here, then the probability estimates
are going to be 3/4 and 1/4.
Whereas, if I smooth with 1, then the probabilities
are going to be 2/3 and 1/3, which is closer to half half.
The second observation I want to make
is that no matter what you set lambda to,
the data wins out in the end.
So suppose we only see examples of dramas, g.
So suppose that we're smoothing with lambda equals 1,
and we saw only one example of g equals 1.
Then, again, the probability estimates are 2/3 and 1/3.
But suppose we keep on seeing dramas over and over again,
so we see 998 of them.
And now, if we account and normalize,
then we get the probability estimate
is 0.999 of drama, which is much closer to seeing only dramas.
So to summarize, we looked at Laplace
smoothing for avoiding overfitting and estimating
Bayesian networks.
And the key idea is that we preload counts with a lambda.
And then we're going to go to training data
and we add counts based on our data.
And then we normalize.
So the smoothing pulls us away from zeros
to the uniform distribution.
But in the end, all the smoothing
gets washed out with more data.
That's the end.