BILL MACCARTNEY: So that's the end of the introduction.
Let's now begin to drill down on the data resources
that we'll need to launch our investigation.
And there are two different kinds
of data we need to talk about, the corpus and the KB.
Just like any other NLP problem, we
need to start with a corpus, a large collection
of natural language text.
And for relation extraction, we need sentences containing
two or more entities.
And because our goal is to do relation extraction
with distant supervision, we need
to be able to connect the entities to a KB.
So we need a corpus in which the entity mentions
are annotated with entity resolutions, which
map them to unique unambiguous identifiers.
The same identifiers that are used in the KB.
So in this representation, I've got the string Elon
Musk, which is just an English language string.
That's what we call an entity mention.
And then I've got Elon_Musk, which is an entity ID.
It's a unique unambiguous identifier
for this entity in some predefined dictionary of entity
IDs.
And it's very common for this purpose
to use something like Wikipedia, which
has one Wikipedia page for almost any entity
that you can think of.
For our investigation, we're going
to use an adaptation of the Wikilinks corpus, which
was produced by Google and UMass in 2013.
The full corpus contains 40 million entity mentions
from 10 million web pages.
And each entity mentioned is annotated with a Wikipedia URL.
But we're going to use just a subset of the full corpus
in order to make things manageable.
So let's start to look at some of the code we'll
use in the Python notebooks for this topic.
The data assets that we'll use live
in a subdirectory of our data directory called rel_ext_data
and we've defined a class called corpus, which
holds the examples and which lets you quickly look up
examples containing specific entities.
So if we load our corpus, we find
that it contains more than 330,000 examples.
Pretty good size.
It's small enough that we can work with it easily
on an ordinary laptop, but it's big enough
to support effective machine learning.
And we can print out a representative example
from the corpus.
Actually, this is a bit hard to read.
So let me give you a different view of the same example.
We represent examples using the example
class, which is a namedtuple with 12 fields listed here.
The first two fields entity_1 and entity_2
contain unique identifiers for the two entities mentioned.
We name identities using Wiki IDs,
which you can think of as the last portion of a Wikipedia
URL.
The next five fields represent the text
surrounding the two mentions, divided into five chunks.
So left contains the text before the first mention.
mention_1 is the first mention itself.
middle contains the text between the two mentions.
mention_2 is the second mention and right contains the text
after the second mention.
And the last five fields contain the same five chunks of text,
but this time, annotated with part
of speech tags, which may turn out
to be useful when we start building models for relation
extraction.
Now, whenever you start to work with a new dataset
it's good practice to do some data exploration
to get familiar with the data.
A big part of this is getting a sense of the high level
characteristics of the data, summary statistics,
distributions and so on.
For example, how many entities are there
and what are the most common ones?
Here's some code that computes that and here are the results.
So there are more than 95,000 unique entities,
and it looks like the most common entities are dominated
by geographic locations.
Now the main benefit we get from the corpus class
is the ability to retrieve the examples containing
specific entities.
So let's find examples containing
Elon Musk and Tesla Motors.
There are five such examples, and here's the first one.
Actually, this might not be all of the examples containing
Elon Musk and Tesla Motors.
It's only the examples where Elon Musk was mentioned first
and Tesla Motors was mentioned second.
There may be additional examples that
have them in the reverse order.
So let's check, look for Tesla Motors, Elon Musk.
Sure enough, two more examples in reverse order.
So going forward, we'll have to remember
to check both directions when we're
looking for examples containing a specific pair of entities.
OK, a few last observations on the corpus.
First, this corpus is not without flaws.
As you get more familiar with it,
you'll probably discover that it contains
many examples that are nearly, but not exactly duplicates.
This seems to be an artifact of the web document sampling
methodology that was used in the construction of the Wikilinks
data set.
And it winds up creating a few distortions
and we may see some examples of this later.
But even though the corpus has a few warts,
it will serve our purposes just fine.
One thing that this corpus does not include
is any annotation about relations.
So it could not be used for the fully supervised approach
to relation extraction, because that
requires a relation label on each pair of entity mentions.
And we don't have any such annotation here.
The only annotations that we have in this corpus
are entity resolutions.
Mapping an entity mention to an entity ID.
That means that in order to make headway,
we'll need to connect the corpus with an external source
of knowledge about relations.
We need a KB.
Happily our data distribution does include a KB,
which is derived from Freebase.
Freebase has an interesting history.
It was created in the late 2000s by a company
called Metaweb led by John Giannandrea, who
later became my boss.
Google acquired Metaweb in 2010, and Freebase
became the foundation of Google's knowledge graph.
Unfortunately, Google shut Freebase
down in 2016, which was tragic.
But the Freebase data is still available from various sources.
So our KB is a collection of relational triples each
consisting of a relation, a subject, and an object.
So for example, place_of_birth, Barack_Obama, Honolulu.
has_spouse, Barack_Obama, Michelle_Obama.
Author, The Audacity of Hope, Barack_Obama.
So as you might guess, the relation
is one of a handful of predefined constants
like place_of_birth birth or has_spouse.
The subject and the object are entities
represented by Wiki IDs.
It's the same ID space used in the corpus.
Wiki IDs are basically the last part of a Wikipedia URL.
Now just like we did for the corpus,
we've created a KB class to store the KB triples
and some associated indexes.
This class makes it easy and efficient
to look up KB triples both by relation and by entities.
So here, we're just loading the data
and printing a count of the KB triples.
There are 45,000 KB triples, so this is quite
a bit smaller than the corpus.
If you remember the corpus has 330 examples
and we can print out the first KB triple.
So this is a KB triple that says that the contains relation
holds between Brickfields and Kuala Lumpur Sentral railway
station which I did not know.
Just like we do with the corpus, let's do some data exploration
to get a sense of the high level characteristics of the KB.
So first, how many relations are there?
The all_relations attribute of the KB
contains a list of its relations.
And it seems that there are 16 of them.
Well, what are the relations and how big are they?
This code prints out a list with sizes.
Note the get_triples_for_relation
method, which returns a list of the KB triples
for a given relation.
You begin to get a sense of what kind of stuff is in this KB.
It looks like the contains relation
is really big with more than 18,000 triples.
And there are a few relations that
are pretty small with fewer than 1,000 triples.
Here's some code, that prints one example from each relations
so that we can form a better sense of what they mean.
Some of these are familiar facts like adjoins, France, Spain.
Others might refer to unfamiliar entities.
So for example, I've never heard of Sheridan Le Fanu.
But I think you can quickly form an intuitive sense of what
each relation is about.
Now one of the most important message in a KB class
is get_triples_for_entities which lets us look up triples
by the entities they contain.
So let's use it to see what triples
contain France and Germany.
OK, sure, they belong to the adjoins relation.
That makes sense.
Now relations like adjoins are intuitively symmetric.
So we'd expect to find the inverse triple in the KB
as well, and yep, it's there.
But note that there's no guarantee
that such inverse triples actually appear in the KB.
There's no guarantee that the KB is complete.
And you could easily write some code to find missing inverses.
Now, that relation adjoins is symmetric,
but most relations are intuitively asymmetric.
So let's see what triples we have
for Tesla_Motors and Elon_Musk.
OK, they belong to the founders relation.
Good.
That's expected.
That's an asymmetric relation.
What about the inverse, Elon Musk and Tesla Motors?
OK, they belong to the worked_at relation.
Seems like a funny way to describe Elon's role at Tesla,
but OK.
So this shows that you can have one relation
between x and y and a different relation that
holds between y and x.
One more observation.
There may be more than one relation
that holds between a given pair of entities,
even in one direction.
So for example, let's see what triples hold--
what triples contain Cleopatra and
Ptolemy_XIII_Theos_Philopator.
Oh, my goodness.
This pair belongs to both the has_sibling
relation and the has_spouse relation to which I can only
say, oh.
Moving right along, let's look at the distribution
of entities in a KB.
How many entities are there and what are the most common ones?
Well, here's some code that computes that.
There are 40,000 entities in the KB.
So that's fewer than half as many entities as in the corpus.
If you remember, the corpus has 95,000 unique entities.
So there are lots of entities in the corpus that
don't appear in the KB at all.
But just like the corpus, the most common entities
are dominated by geographic locations, England, India,
Italy and so on.
Note that there's no promise or expectation
that this KB is complete.
For one thing, the KB doesn't even
contain many of the entities from the corpus.
And even for the entities it does include,
there may be possible triples which are true in the world
but are missing from the KB.
So as an example, these triples are
in the KB, founders, Tesla_Motors, Elon_Musk,
worked_at, Elon_Musk, Tesla_Motors, Founders, SpaceX,
Elon_Musk.
You might expect to find worked_at, at, Elon_Musk,
SpaceX.
But nope, that triple is not in the KB.
That's weird.
Well, in fact, the whole point of relation extraction
is to identify new relational triples from natural language
text so that we can add them to a KB.
If our KB's were complete, we wouldn't have anything to do.
Now actually, in this case, you might
object that we don't need to do relation extraction to make
that completion.
We could write some logic that recognizes
that founders, xy entails worked at yx
and apply that rule systematically
across the KB and use that to fill in the missing
triple in this case.
But the general point still stands
that there may be lots of triples
that are true in the world but missing from the KB
where that strategy is not going to allow us to add the missing
information.