CHRISTOPHER POTTS: Hello everyone.
Welcome to part 6 in our series on distributed word
representations.
This can be considered an optional part
but it's on the irresistibly cool idea of retrofitting
vectors to knowledge graphs.
Here are the central goals.
On the one hand, as we've seen, distributional representations
are powerful and also easy to obtain,
but they tend to reflect only relatively primitive semantic
notions like similarity, or synonymy, or connotation,
or relatedness, so that might feel limiting.
On the other hand, structured resources
like knowledge graphs, while sparse and kind of hard
to obtain, support really rich learning
of very diverse semantic distinctions.
So the question naturally arises,
can we have the best aspects of both of these?
And the inspiring answer given by retrofitting is yes.
We can combine them.
The original method for doing this
is due to this lovely paper Faruqui et al., 2015,
which I'm going to be giving a brief summary of in the screen
cast.
So here is the retrofitting model.
It consists of two sums and they constitute
kind of opposing forces.
Imagine that we have an existing embedding space, like GloVe,
or some embedding space that you built yourself-- that's q-hat.
And we're learning these qi's and qj's.
The term on the left is basically
saying, remain faithful to those original vectors
as you learn these new vectors qi.
Try not to be too dissimilar from where you started.
That pressure is balanced against the pressure
on the right, which is saying, make representations
that look more like the neighbors for the current node
in the knowledge graph, which is defined
by this set of relations E. So two
opposing pressures, on the one hand,
we're saying to be faithful to the original,
on the other hand, we're saying look more like your neighbors
in the knowledge graph.
If we set alpha to 1 and beta to 1
over the out-degree for the node that we're targeting,
then we have basically balanced these two pressures.
If we set alpha really large, we'll
mostly want to stay faithful to the original vectors.
If we set beta comparatively very large,
then we'll mostly want to look like the neighbors
in the knowledge graph, and we won't
remain so tethered to the original embedding space
that we started with.
This illustration kind of nicely depicts
what happens in the model--
the gray vectors of the original embedding space.
We have these knowledge graphs that
connect the associated nodes.
And because they're connected in the retrofitting space which
is given in white, these nodes are kind of pulled together
and look more similar.
There's a bunch of code for doing retrofitting
in the course repository, and I'll just
show you a few quick illustrations using that code.
Let's start with a simple case, we have a very simple knowledge
graph where node 0 is connected to node 1 and node 0
is connected to node 2, just directionally.
What happens when we run the retrofitting model is
that 0 is called equally close to 1 and to 2, kind
of equidistant between them and closer to both
than it was in the original embedded space.
Here's a situation in which every node
is connected to every other node that's represented on the left
here.
That's where we start, and as a result
of running the retrofitting model with alpha and beta set
in their default parameters, what happens
is that triangle just gets smaller
in kind of fully symmetric way as the nodes become
more similar to each other because of the graph structure.
Here's a kind of degenerate solution.
If I set alpha to 0, I have no pressure
to be faithful to the original vectors.
All I care about is looking like my neighbors
from the term on the right.
And as a result, all these vectors shrink down
to be the same point after the models run
for a few iterations.
If instead I had done the opposite,
I've made alpha really large comparative to beta,
then basically nothing would have
happened in the learning of the triangle.
It would remain its original size.
It's worth considering some extension.
So I think the fundamental limitation of this model
is that it is kind of assuming, right there in its objective,
that to have an edge between nodes
is to say that they are similar.
But of course, the whole point might
be that your knowledge graph has very rich edge
relations corresponding to different linguistic notions
like antonymy.
And we certainly wouldn't want to treat synonymy and antonymy
as the same relation and just assume that it meant similarity
in our model.
So there are various extensions.
I think the most general extension that I've seen
is from a paper that I was involved
with led by Ben Lengerich, which is called
functional retrofitting, which allows you
to very flexibly learn different retrofitting
modes for different edge semantics.
And once you start down that road,
you have a really natural connection with the literature
on graph embedding, that is learning
distributional representations for nodes and knowledge graphs.
And this paper led by Will Hamilton
is an outstanding overview of methods in that space.
And then you have this nice synergy between NLP methods
and methods that are more associated
with work on knowledge graphs and social networks
and so forth.
And finally, here are some code snippets just showing
some simple illustrations of the sort
that I showed you earlier in the screen cast.
And I would just mention at the end
here, if you would like to apply these methods to WordNet, which
could be a powerful ingredient for the first assignment
and bake off, I would encourage you to check out this notebook
bsm_03_retrofitting, because it walks through all
the steps for doing that.