So that's sort of EM at the level of
the equations, but I thought it would be helpful
to actually implement it and kind of mess around
a little bit. So do you want to see what happens?
>> I would love to see what happens.
>> So I generated some data here,
which comes actually from two different Gaussian
clusters. One that's centered over hereish, and
one that's centered over hereish. [LAUGH] Just not
to be too specific about it, but you can sort of see that there's two
clouds of points, right? One in the upper left and one in the lower right.
Can you see it?
>> The one in the lower right looks like
a clown. I think it's a little bit more of
a superhero but that's that's not the point. The point
is, is that it's just a bunch of random points.
>> Hm.
>> That's the point. It's a random point, but it is a point.
>> So, your point is that the points are randomly pointed.
>> Yeah, I just want to point out that at random.
>> I see your point.
>> And, so, what I want to do now is run
EM and the first thing I need to do is pick
the initial centers. Right, so I need to pick the mu's
that we think these were generated by. And so a common
thing for people to do is just to take two of
the points from the data set at random. So I took
two points at random and I'll mark them with an x.
And now what I'm going to do is run an iteration of
EM. And what that means is, I'm going to first do
expectation, which is going to assign every one of these points to
one of these two clusters. And I'll color them as to
which cluster they belong to. And then I will move the centers,
re-estimate the means based on that soft assignment. Makes sense.
>> Alright, now so what you can see is something kind
of interesting happened. Because these centers started off in the same
cluster together, many of them were assigned to one cluster, which
was the one that was a little bit more above. And
then very few of them ended up getting assigned to the
lower cluster because very few points were essentially closer to that
one than the other one. When I colored the lower ones
red and the upper ones blue. And there's this sort of band
of 1s in the middle that had intermediate probabilities,
they weren't really deeply one cluster or the other.
>> They were near 0.5?
>> Yeah, exactly, specifically between, including 0.4 to including 0.6.
>> Okay that make sense.
>> Alright, and so those are the green ones. And
then based on the cluster assignments, we estimated the means. So
this xis now at the, you know, the mean of
the red points. And the green points, because these they're actually
shared and this acts as kind of the rest. And you can
see, it doesn't really capture yet what the true clustering is because a
huge number of the lower right cluster points are blue. They're kind of
grouped in with the upper left cluster. But we can run another iteration
of the app. And now things have shifted, right? So now it
looks like this lower right cluster has claimed a good amount of the
points in the lower right and the blue ones are the blue ones.
There's just a few little green scattered between them at the boundary and
you always expect some of that to happen. Now, the centers have
moved and now they are starting to take on their clusters, but we
just run this a couple more iterations until we see the x is
not really moving any where. Alright, seems to have settled down and you
can see it really did pull out the lower right cluster, the upper
left cluster as a cluster and just a few points at the boundary
where it said well I can sort of believe that that's part of the
red cluster, I can sort of believe it's also part of the blue
cluster, I really can't decide. And you know what? That's
probably right too. I mean they could go either way.
>> Yeah, exactly. I can sort of hallucinate this green
point here as being part of the upper cluster. Just
kind of on the fringe. But, you know, it works
just as well as being part of the lower cluster.
>> Actually Michael I'm kind of curious. Can you, use
your magical computer powers to, you know, click on
one of those green points and see what the
probability is? I could. Yeah, so this, in particular,
it's 55% in the first cluster and 45% in the other cluster.
>> Hm.
>> So, it really is. It's hovering near that boundary.
>> That makes sense. What about that red point all the
way in to the right? The ones that are all by itself.
>> Excellent choice. So, it is actually. 0.9999996
in one cluster. And 1 minus
that in the other cluster. This one
is pretty certain coming from the second cluster.
>> I see two things that
came out of this. One is EM is much nicer in
that it's not forced to make a decision about where those green
points go. So that's sort of soft clustering does that. And
that's a good thing. Because we don't have that problem that we
had before. But one of the consequences of that, and this
is not a bad thing, but it's a thing, is that even
points that pretty clearly belong to one cluster or another, given that
we're staring at them. They do have a really high probability, 0.9999999996,
but they all have some non-zero probability. Of
ending up, of belonging to the other cluster.
>> And is that, you think that's a good thing or a bad thing?
>> I think it's a good thing. I
think it makes sense because Gaussians have infinite
extent and even if a point is very, very far away from the center, it has
some chance of having been generated from that
Gaussian and so, this just tells us what
that chance is, it is very, very unlikely.
But it is still as non-zero probability match.
>> In, in some sense, it's acknowledging truth.
Right? Which is that you can't be sure
which of the two clusters, this came form,
even if I tell you where these clusters are.
>> Right?
>> Yeah, okay. I, I agree that.
>> Okay, this is cool. So, EM is nice. So,
does EM work for anything other than Gaussians? It does, it
actually can be applied in lots of different settings. Let's
flip over and talk a little about some properties of EM.
>> Okay.