BILL AMCCARTNEY: So I now want to turn
to the question of how to formulate our prediction
problem precisely.
I want to be precise about how we're defining the inputs
and outputs of our predictions, and that, in turn,
is going to have consequences for how
we join the corpus and the KB, how we construct
negative examples for our learning algorithms,
and how we handle multi-label classification.
So first what is the input to our prediction problem?
In the supervised learning paradigm,
the input is a pair of entity mentions
in the context of a specific sentence.
We're trying to label a sentence just
like we do in part of speech tagging or sentiment analysis.
But in the distance supervision paradigm,
we'll do things differently.
The input will be a pair of entities full stop, independent
of any specific context.
We're trying to determine the relation between this entity
and that entity, and that's it.
The other question I want to look at
is, what's the output of the prediction problem?
Are we trying to assign a pair of entities
to a single relation?
That's called multi-class classification.
Or are we trying to assign a pair of entities
to multiple relations?
That's called multi-label classification,
and it's a different beast.
So over the next couple of slides,
I want to explore the consequences of these choices.
The difference between these two ways
of thinking about the input becomes really important
when we talk about how we're going
to join the corpus and the KB.
In order to leverage the distance supervision paradigm,
we need to connect those two.
We need to connect information in the corpus with information
in the KB.
And there's two different possibilities
depending on how we formulate the prediction
problem, depending on how we define
the input to the problem.
If our problem is to classify a pair of entity mentions
in a specific example in the corpus, in a specific sentence,
then we can use the KB to provide the label,
and this is what it looks like.
We have a corpus example like this.
We're trying to label this specific example.
And to do it, we can check to see if these two entities are
related in the KB.
Yep, they are.
And we can use that to generate a label for this example.
Labeling specific examples is how the fully supervised
paradigm works.
So it's an obvious way to think about leveraging
distance supervision as well.
It can be made to work, but it's not actually
the preferred approach.
If we do it this way, we'll be doing things
exactly as they're done in the supervised paradigm.
It does work, but it's not the best way
to take advantage of the opportunity
that distance supervision creates.
There's another way of doing things,
and the other way is where, instead, we
define our problem as classifying a pair of entities.
Not entity mentions in a specific sentence, but just
entities.
Elon_Musk and Tesla, period, apart from any sentence.
And if that's how we define the input to our problem,
then we can use the corpus to provide
a feature representation that will
be the input to the prediction.
So if we have an entity pair like Elon_Musk and SpaceX
that we're considering adding to a relation in the KB,
we can find all sentences in the corpus containing
this pair of entities, and then we
can use all of those sentences to generate a feature
representation for this pair.
So in this example, and I'm imagining--
it doesn't have to be this way, but I'm
imagining that we're using a simple bag of words feature
representation.
The bag of words has come from the middle, that
is, the phrase between the two entity
mentions, the blue phrases here.
And all I've done is counted up the words
in all of these blue phrases across all
of the examples in the corpus where
these two entities co-occur.
Yeah.
Well, you can see here in the token counts
that they include tokens from the various examples.
All of these examples together are
used to generate a single feature representation.
This is a feature representation for this pair,
and it's this feature representation
that my learned model will use to make
a prediction about this pair.
So this is a very interesting way of reversing things.
Instead of using the KB to generate a label
to make a prediction about a specific pair of entity
mentions in a specific sentence, I'm turning things around.
I'm using the corpus to generate a feature representation that I
will use to make a prediction about an entity
pair in abstraction, an entity pair
considered just as a pair of entities.
Just one more thought on this.
This is still kind of about the topic of joining
the corpus and the KB.
We've created a data set class which
does that, which combines a corpus and a KB.
Just kind of staples them together
and provides a variety of convenience methods
for the dataset.
And one of those convenience methods
is this one, count_examples, which
shows, for each relation, how many examples we have
in a corpus, how many triples we have in a KB,
and the ratio, so the total number
of examples, the average number of examples per triple.
For most relations, the total number of examples
is fairly large, so we can be optimistic about learning
which linguistic patterns express a given relation.
I mean, even the smallest one has at least 1,500 examples.
That's not really industrial grade data,
but it's certainly enough for the kind of exploration
that we're doing here.
However, for individual entity pairs, the number of examples
is often quite low.
So some of these relations are betwee--
some of these ratios are between 1 and 2.
Of course, more data would be better, much better,
but more data could quickly become
unwieldy to work with in a notebook like this, especially
if you're running on an ordinary laptop.
And this data is going to be enough to allow us to have
a fruitful investigation.
First, I want to talk about negative examples.
So by joining the corpus to the KB,
we can get lots of positive examples for each relation,
but we can't train a classifier on positive examples alone.
We're also going to need some negative examples,
negative instances.
So that is entity pairs that don't belong to any relation.
We can find such pairs by searching
the corpus for examples which contain
two entities which don't belong to any relation in the KB.
So we wrote some code to do this,
and there's a method on the dataset class called
find_unrelated_pairs.
And when we run it, wow.
It found almost 250,000 unrelated pairs,
so 250,000 negative instances for our prediction problem.
And that's way more than the number of positive instances.
If you remember, the KB has 46,000 triples.
Each of those is basically a positive instance.
It's something that we know is definitely a positive example
of the relation.
Here, we have 250,000 negative examples.
It's so many more that when we train models,
we'll wind up downsampling the negative instances
substantially so that we have a somewhat
more balanced distribution.
A reminder, though.
Some of these supposedly negative instances
may be false negatives.
They may be entity pairs that don't appear to be related,
but in the real world, actually are.
Our KB is not complete.
A pair of entities might be related in real life,
even if they don't appear together in a KB.
And as I said earlier, after all, that's the whole point.
That's the whole reason we're doing relation extraction is
to find things that are true in real life
and true according to some text that somebody wrote, but aren't
yet in our KB.
OK, now I'm going to come to the question that
was asked about pairs that belong to multiple relations.
And this is related to the question of the outputs
of our prediction problem.
We wrote some code to check the KB for entity pairs
that belong to more than one relation.
So that's this method count_relation_combinations.
And it turns out, this is a really common phenomenon
in the KB.
There are lots of pairs that belong to multiple relations.
For example, I won't even mention the most common one,
but there are 143 people in the KB whose place of birth
is the same as their place of death.
And actually, that's not that surprising.
That makes perfect sense.
It even turns out that there's no fewer than seven people who
married a sibling.
Well, since lots of entity pairs belong
to more than one relation, we probably
don't want to be forced to predict a single relation.
So this suggests formulating our problem
as multi-label classification.
We want our models to be able to predict multiple relations
for any given entity pair.
There are a number of ways to approach
multi-label classification, but the most obvious
is the binary relevance method, which
just factors multi-label classification over n labels
into n independent binary classification
problems, one for each label.
So if you have a pair like Pericles and Athens,
you want to be able to predict any combination
of these labels.
You just train a separate model, a separate binary classifier,
for each of the labels independently.
Each of them generates a prediction independently.
And in this example, we've predicted
that the place of birth relation applies,
the place of death relation applies, but not
the has_sibling relation.
A disadvantage of this approach is that it fails to--
because it treats the binary classification problems
as independent, it fails to exploit
correlations between labels.
For example, there may well be a correlation
between the place of birth label and the place of death label.
And if you already have evidence that the place of birth label
applies, that might tilt you, at least a little bit,
toward saying yes for place of death.
This approach of factoring them into independent binary
classification problems is not able to take
advantage of that information.
But it has the great virtue of simplicity.
It's incredibly straightforward, incredibly easy to think about
and to implement.
And it'll suffice for our purposes.
It's going to make the investigation move forward
very smoothly.
So I want to sum up a little bit.
We set out to establish a precise formulation
of our prediction problem.
And when we put all the pieces together,
here's the problem formulation we've arrived at.
The input to the prediction will be an entity pair
and a candidate relation.
The output will be a Boolean indicating whether the entity
pair belongs to the relation.
Since a KB triple is precisely a relation and a pair
of entities, we could say equivalently
that our prediction problem amounts to binary
classification of KB triples.
Given a candidate KB triple like worked_at,
Elon_Musk, SpaceX, do we predict that it's valid?
This is really nice because it's a very simple way
of thinking about what problem we're taking on.
We have a bunch of positive examples,
which come from our KB.
We have a bunch of negative examples, which we synthesize
from the corpus using pairs which co-occur in the corpus
but don't occur in the KB.
Now we have lots of data consisting of candidate KB
triples, including positive examples and negative examples.
We can use that data both for training and for evaluation.
And once we've trained a model to do
this binary classification, we can now
consider novel KB triples which don't appear anywhere
in our data, and ask whether the model will
predict them to be true.
And by doing that, we may discover new relations that
are not currently part of the KB that
could be candidates for adding.