BILL MACCARTNEY: It's good methodological practice
whenever you're starting to build new models to start
by evaluating very simple models, which establish
baselines to which you can then compare
the more sophisticated models that you're
going to build later on.
So to do that, we're going to start by looking
at three simple models.
A random guesser, a very simple phrase matching strategy,
and then, our first machine learning
based approach, which will be a simple bag-of-words classifier.
Just about the simplest possible model
is one that doesn't even look at the input,
but just flips a coin.
And I strongly encourage you, whenever
you're embarking on a model building
adventure in your final project, wherever, you start
by evaluating a random guesser.
It's a snap to implement.
It can help to work out the kinks in your test harness.
And it's often very informative to put a floor under what
good scores look like.
Now, we've written an evaluation method for you.
It's in the RelExt module and it's just called evaluate.
You invoke it with your splits, your classifier,
and the name of the split that you want to evaluate on,
which defaults to dev.
When we evaluate our random guesser,
we have some interesting results,
so we have results separated for each of the relations.
And for each one, we have precision recall, F-score.
Remember, that's F0.5, which gives more weight to precision
than to recall.
We have the support, which is the number of instances
whose actual label is true.
And we have size, which is just the total number of instances.
We find that recall is generally right around 0.5.
And this makes sense, because recall
says of the instances which are actually true,
what proportion do we predict true?
Well, we predict true about half the time
because we're just flipping a coin.
Precision on the other hand, is generally quite poor,
because precision says of the instances
where we predict true which are basically
a random sample, because we're just flipping a coin.
How many are actually true?
Well, relatively few.
And actually, you can tell that by looking
at the ratio between support and size.
The ratio between support and size
is how many of the instances are actually true.
So when we're tossing a coin, the precision
should be right around the ratio between support and size.
Our F-score is also generally poor,
it stays close to precision, for two reasons.
Number one, because the harmonics--
the harmonic mean stays closer to the lower number,
and number 2, because we're using F0.5,
which gives more weight to precision than to recall.
And the bottom line our macro-average F-score is 9.7%,
so that's the number to beat.
It's a pretty low bar, but this is a random guesser, after all.
OK.
So let's look at another approach, which
is very simple, but smarter than random guessing.
And it's a simple pattern matching strategy.
And the idea is for each relation,
let's go through the corpus and find the most
common phrases that connect to entities
that stand in that relation.
The most common middles, in our terminology.
So here's some code that does that.
I won't go through it in detail, but one thing to note
is that it counts separately the middles that connect subject
with object.
So here, it gets all the examples
and counts the middles.
It tallies up the middles.
And it does that separately from the examples that
connect object with subject.
And it stores them in separate dictionaries under the keys
forward and reverse.
So we're going to have forward middles
and reverse middles stored separately, stored
and counted separately.
If we run that code, here's what we get.
I'm showing results.
I'm only going to show results for 3 of the relations
here, not all 16.
All 16 are in the Python notebook
if you want to take a look.
But even from this sample, there's
a few things that jump out.
First, some of the most frequent middles
are really natural and intuitive.
For example, comma, "starring" indicates a reverse film
performance relation.
So that would be one where the film comes first
and the actor comes second.
And I think that makes perfect sense.
Star Wars, comma, "starring" Mark Hamill.
Similarly, comma, "son of" indicates a forward parents
relation.
So this would be one where the son comes first.
The child comes first and the parent comes second.
So those are extremely intuitive and it's reassuring
to see them near the top of the list of most common middles.
Another observation is that punctuation and stop
words, like comma and "and" are extremely common.
Unlike some other NLP applications,
it's probably a bad idea to throw these away.
They carry lots of useful information.
On the other hand, punctuation and stop words
tend to be highly ambiguous.
For example, if you look across the full range of all 16
relations, you'll see that a bare comma
is a likely middle for almost every relation in at least
one direction.
So that comma does very often indicate a relation,
but it's a really ambiguous indicator.
OK.
Now that we've identified the most common middles
for each relation, it's straightforward to build
a classifier based on that information, a classifier that
predicts true for a candidate KB triple, just in case
the two entities in the triple appear in the corpus connected
by one of the phrases that we just discovered.
I don't show the code for that here,
but it's in the Python notebook for this unit.
And when we evaluate this approach,
we see some really interesting results.
First, recall is much worse across the board.
And that makes sense because we're no longer just
guessing randomly.
Before, we were saying true half the time.
Now, we're going to be a lot more selective about what
we say true to.
But precision and F-score have improved dramatically
for several relations, especially
for adjoins and author and has_sibling and has_spouse.
Then again, there are many other relations
where precision and F-score are still quite poor, including
this one genre where we get straight zeros
across the board.
I'm not quite sure what happened there.
But it indicates that although things
have improved a lot in some places,
they're still rather poor in others.
And our macro-average F-score has improved only modestly,
so it improved from 9.7% to 11.1%.
We're heading in the right direction,
but you'd have to say that's still pretty unimpressive.
To make significant gains, we're going
to need to apply machine learning.
So let's get started on that.
We're going to build a very simple classifier using
an approach that should be familiar from our look
at sentiment analysis last week.
And we're going to start that by defining a very simple
bag-of-words feature function.
So here's the code for that, and let
me briefly walk you through it.
What we're going to do is to get the features for a KB triple.
That's the kbt here.
We're going to find all of the corpus examples containing
the two entities in the KB triple, the subject
and the object.
And note that we do that in both directions, subject and object,
and then, also object and subject.
For each example, we look at the middle.
We break it into words.
And then, we count up all the words.
So a couple of things to note here.
One is that the feature representation for one KB
triple can be derived from many corpus examples.
And this is the point that I was trying to make last time,
that we're using the corpus to generate features
for a candidate KB triple.
And the role of the corpus is to provide the feature
representation.
And the feature representation for a KB
triple will be based on all of the examples
in the corpus that contain those two entities.
The other observation to make here
is that we make no distinction between what
you might call forward examples, which have subject first
and then object, and reverse examples, which have object
and then subject.
We're lumping them all together.
The words that come from the middles of examples
in either direction all get lumped together
into one feature counter.
And you might have qualms about whether that's really
the smartest thing to do.
So let's get a sense of what this looks like in action.
First, let's print out the very first KB triple in our KB.
We actually looked at this last time.
It's a KB triple that says--
that contains relation, holds between Brickfields and Kuala
Lumpur Sentral railway station.
And now, let's look up the first example
containing these two entities.
I'm just going to look them up in the forward direction,
subject and object, and get all the examples.
I look at the first one, and let me just point out the middle.
The middle says, "It was just a quick 10-minute walk to."
So I guess the full example probably said something
like, "From Brickfields, it was just
a quick 10-minute walk to Kuala Lumpur Sentral railway
station."
And maybe there was more.
Now, let's run our featurizer on this KB triple
and see what features we get.
So we get a counter that contains "It was just
a quick 10-minute walk to the--" So it
looks like it's counted up the words in that middle, which
is just what we expected.
But if you look closely, there's something unexpected here
because the word "two" has a count of 2,
even though it appears only once in that middle.
And also, the word "the" has a count of 1,
even though it didn't appear in that middle at all.
So where did those come from?
Well, remember that the featurizer counts words
from the middles of all examples containing those entities
in either direction.
And it turns out that the corpus contains
another example containing those two entities
and that other example has--
there's just one other example, but that other example
has middle "to the."
And so, that's where these counts come from.
So all is well.
We did the right thing.
OK.
We have our simple bag-of-words featurizer.
Now, we need a way to train models to make predictions
and to evaluate the results.
The RelExt module contains functions for each of those.
And so, I just want to give you a quick tour
of what those functions are.
But you'll definitely want to go read the code for this,
so that you're more familiar with how it can be used.
And a lot of this code appears in a file called RelExt,
rel_ext.py.
So we'll start with a function called train models.
This takes as arguments the dictionary of data
splits, a list of featurizers.
And here, we have a list consisting
of just our simple bag-of-words featurizer.
The name of the split on which to train,
which defaults to train.
And a model factory, which is a function that returns
a classifier and it's--
sorry, a function which initializes an Sk classifier.
And by default, it's a logistic regression classifier,
as shown here.
But you could easily substitute this with some other Sklearn
classifier.
It returns this thing called train_result
which is a dictionary holding the featurizers, the vectorizer
that was used to generate the training matrix,
and most importantly, a dictionary holding the trained
models, one per relation.
So it's a dictionary which maps from relation names to models.
So that's train models.
Next comes predict, this is a function
that takes his arguments a dictionary of data splits.
The output of train models, that train results thing,
and the name of the split on which to make predictions.
And by default that's dev.
And it returns two parallel dictionaries,
one holds the predictions grouped by relation
and the other holds the true labels grouped by relation.
And our third building block is evaluate predictions.
So this is a function that takes as arguments the two
parallel dictionaries of predictions
and true labels produced by predict,
and it prints evaluation metrics for each relation,
like we saw earlier.
Now, before we dwell on these results,
I want to show one more function, which is
a function called experiment.
And experiment simply chains together the three functions
that I just showed you.
It chains together training, prediction, and evaluation.
So that's very convenient for running end to end experiments.
I haven't shown all the parameters here,
but if you go look at the source code,
you'll see that it actually takes
a lot of optional parameters.
And those parameters let you specify everything about how
to run the experiment.
It lets you specify your featurizers, your model
factory, which splits to train and test on, and more.
So for example, earlier, I mentioned that the tiny split
is really useful for running fast experiments to work out
the kinks.
If you wanted to do that, it's very easy
using the experiment function just
to set the training split and the test split
to tiny to run a very quick experiment.
Now, here are the results of evaluating
our simple bag-of-words logistic regression classifier.
And let's take a closer look, because this is quite stunning.
Even though this is just about the simplest
possible classifier, we've achieved huge gains
over the phrase matching approach.
The first thing that jumps out is
that our macro-averaged F-score has jumped from 11.1 to 56.7.
And we see big gains in precision
for almost every single relation and correspondingly,
big gains in F-score.
On the other hand, there's still plenty of room for improvement.
I mean this is much, much more impressive than where
we were before, but we're very far from perfection.
There's abundant headroom and opportunity
to continue to improve.