Recall that, if we have a random variable x, and
we want to understand its relationship with another random variable, v.
The way we do that is we examined the expected loss and
entropy due to that variable.
Specifically, to understand the relationship between x and
y, we examine the expected loss and entropy due to y.
Similarly, for z, we do exactly the same thing.
We examine the expected loss and entropy due to z.
After we know these two quantities the feature that we
want is the one that results in the biggest drop from h of x.
So as you can see in order to do this we'll have to calculate three quantities.
h of x, the conditional entropy, the entropy of x.
The conditional entropy of x given y and the conditional entropy of x given z.
I've written them here, over off to the side, so
that we can keep track of them as we do our calculations.
Let's go ahead and find them.
So h of x is just this expression
minus one times the probability of x times log of probability of x.
Minus the probability of x2 times the log of probability of x2.
Now, remember the base of the logarithm doesn't really matter.
Traditionally, either base two or base e is used.
Let's assume for these calculations we're using base e.
So, we can just pull the values for
the probability of x1 and x2 straight from the joint probability table.
And if we do that, we find that we get minus one times 0.472
times log of 0.472 minus 0.528 times log of 0.528 which is approximately 0.691.
So let's just keep track of that by writing it up here.
So one of three down.
Now we just have to do h of x given y and h of x given z.
Recall that the conditional entropy of h of
x given y is just a conditional expectation.
It is the average of how much we expect h of x to change due to y1 and
due to y2.
So, the first term is just this expression below.
So first, so first, let's tackle h of x given y1.
H of x given y1 is just minus 1 times the probability of x1 given y1,
times the log of probability x1 given y1,
minus probability of x2 given y one times the log of probability of x2 given y1.
And again we can just pull these numbers straight out of the joint probability
table above and
if we do that we find we get minus one times .0318 times log .0318.
Minus 0.779 times log 0.779 and that's approximately 0.3043.
Let's just make a note of that here.
The calculation for h of x given y2 proceeds in exactly the same way.
And if you evaluate that, you get approximately 0.6888.
And let's just make a note of that here.
So if we put it all together, h of x given y is .32075
times .3043 plus .67925 times .6888, which is approximately .566.
Let's make a note of our second term above.
Now the calculation for
the conditional entropy of x given z proceeds in exactly the same way.
I'm not going to step through the calculation for
this one as well, I'll leave that to you.
But you'll find that if you do, you'll get approximately 0.5787.
So now we have all three of our terms.
Before knowing anything about y or z, h of x is .691.
It drops to .566 when we condition on y and
it drops to .5787 when we condition on z.
So recall that the feature that we want is the one that results in
the biggest change in entropy.
Which is the arg max of h of x minus h of x given a variable.
Where in this case the variables range over y and z.
So you can see that in this case the answer is y.
Although not by that much.
This difference for y is 0.125.
Whereas for z it's just slightly less at 0.1123.
So although y is the winner here, z doesn't do that much worse.
And so in practice, if I were going to build a model in order to understand x,
I might consider including both y and z.
But if I had to choose one, y is the winner.