WILLIAM COHEN: So I've been working here
at Google for about a year and a few months.
I started last summer.
Yeah, I'm not quite that tall.
So I came over here as a professor at Carnegie Mellon
University.
So I'm now starting up this small research
group doing machine learning, of course,
and NLP sorts of things.
And I'm just going to give you a little flavor of one
of the projects going on in my group.
I won't be around too much this afternoon.
I'll be back for a little while around happy hour
if you want to talk about it.
But here's a problem.
OK.
So this actually looks a little bit different
than I thought it would, but this project
is called Neural Query Language, NQL.
And it's all about queries on Knowledge Graph
but doing them in a way that's completely differentiable
and doing it inside TensorFlow.
So here's a motivating task.
So here are a couple of examples of this from a paper.
So here's a question.
Which states does the Mississippi run through?
And the way to answer that is with some sort
of query against a database or a knowledge base.
Or another one of these.
In Denver, what kind of ground transportation
is there from the airport to downtown?
Again, sort of the same kind of thing--
there's a natural language question,
and you want to have some sort of structured query language.
OK.
So this is a problem.
People don't know what's in the databases.
They don't know how to ask questions,
so using natural language to access
that kind of makes sense.
A slightly harder problem, as a learning task,
is if you're given the question and the correct answer
without being given the underlying database query.
So here, x, the input, is a natural language
question. y is a Knowledge Graph query.
Here, y is just the answers you want to get.
So there are many cases where this sort of data
is a little bit easier to obtain, but it's harder to use.
So the problem here is there are some
that are missing a step here, which
is what's the query going to be against the Knowledge Graph,
and that's hidden.
That's a latent variable.
So somehow, you have to deal with that.
So how do you deal with that?
So the idea behind NQL is the following.
What we're going to do is we're going
to have a structured query that produces a set of Knowledge
Graph entities.
And we're going to do this with an encoder-decoder strategy.
So you're going to start out by encoding the natural language
query, and then you run it through a decoder process,
which gives you something like a structured query.
And then you execute it on the Knowledge Graph.
So this is kind of the standard thing.
How do you deal with the fact that this is latent?
So you don't have any explicit training data for that.
So the basic idea is what we're going to do
is we're going to turn this execution against the Knowledge
Graph into something that's also part of the neural network.
So the execution will be something that's also
done in the neural network.
And then when you look at your target answers,
you can look at those answers, and you
can compare them with what the neural network actually
provided.
So here, there's a mistake.
It actually gave you the wrong relations, like defines_border
instead runs through.
OK, so this is the answers.
They're not right.
I highlighting the differences here.
So there's going to be some loss on those, that answer.
And what you want is you want to be
able to propagate that loss all the way back to the things that
actually made the mistake.
So here, we're going to propagate the loss back
through the execution of the query against the Knowledge
Graph and figure out what went wrong, what sort of errors
were made in those decoder-encoder steps.
So we can use this indirect data that's
just the answers to train the encoder-decoder pipeline.
So this is the idea behind NQL.
We want to be able to execute a structured query,
generate a set of knowledge base entities.
We want to do it in a way that's completely differentiable
so you can take the loss, and you could push it
through the Knowledge Graph, back to whatever components
that you need to use that semantic parsing task.
So there's a whole system that is a query language
for Knowledge Graphs.
It's part of TensorFlow.
It's now been open sourced, so you
can sort of try it out and use it, no matter where you are.
And here are some examples.
So these are basically the simple type
of-- the simplest type of expression
is you're just basically saying this is a singleton set.
So this is a set of one element.
Here's the name of the element, and here's the type.
So Queen Vic and Henry VIII are expressions now,
in this language.
And when you evaluate them--
and I guess in TensorFlow 2.0, you
can do this a little bit more easily using eager execution.
But when you evaluate them, you'll
get back the set of things.
So you notice the set here actually-- every element
of the set has a number.
So all the things in the set are soft.
That's kind of important.
So here are a few more expressions.
So we've got Henry VIII.
That's a singleton set.
And then I apply a relation.
So this is basically finding all the entities in the Knowledge
Graph that are related to Henry VIII by the wife property.
So if you know your history, he had six wives.
And Henry VI, of course, had eight wives.
I think that's right.
Here's another more complicated expression.
We've got the wives of Henry VIII,
and I want to intersect those with the daughters of Ferdinand
V. OK?
So if he was married to one of Ferdinand's daughters,
then that would be like a non-empty set.
Otherwise, it would be an empty set-- turns out
there's a singleton set.
It's one person.
Or here's another one.
These are all the sons and daughters of Queen Victoria.
So these are the sorts of operations you can do.
You can basically sort of follow chains of properties.
You can do unions and intersections.
And you can compose those so you can construct
relatively complex queries.
And the important things about these
is each one of these expressions is actually
a wrapper around a TensorFlow compute graph.
OK?
And it's fully differentiable.
So you can send a [? law ?] through it.
You can drop them anywhere into a neural model.
So you can use Knowledge Graph information wherever you want.
OK.
And everything's also extended to sort
of a soft probabilistic setting.
So all the sets of entities are weighted sets.
Here, the weights are all 1.0, but that
doesn't have to be the case.
All right.
And what's really constructed here
is a vector representation of those sets.
So when I'm printing things out, like here,
I'm giving you strings that are associated with numbers.
Under the hood, it's really just a kind
of TensorFlow representation.
So the eval is just a little bit of syntactic sugar.
OK.
So here's another thing you can do,
which is less common in a query language.
So you can also define a set of relations.
So here, I've defined a set of relations which
are nuclear family relations.
And I've got two singleton elements, the wife relation,
and the husband relation.
And I put them together, and I call it the spouse relation.
So I can follow any one of those spouse relations.
So I can find out whoever you're married to.
So this is a little bit more, you know--
I could say less patriarchal, like gender normative.
You can just say find someone's spouse, rather than specifying
husband or wife.
So that's kind of nice from a diversity point of view,
but it's also really practical from a machine
learning point of view because a very typical thing is
for to do this semantic parsing thing, you've got to guess.
The machine learning system has got
to guess not only what the structure of the query is.
So in this case, the query is basically
starting with some object, the Mississippi River,
and following one relation.
So it's a simple kind of query, but it's also
got to figure out what are the constants in that query.
And those constants can be relations.
So here, we've got the relation, traverse_state.
So that's something you've got to predict.
So the way you would actually do that in NQL,
what the semantic parser would look like for this--
if all you cared about was like these single relation queries--
is you'd first construct your seed set, which would
be Mississippi in this case.
And that's some function of the query.
And you could use any differentiable function
you want, so whatever you want.
You could use BERT.
You could use encoder-decoder strategies
with Transformers or LSTM, whatever you want to do.
And you'd also pull out the relations,
and then you'd construct a query where all those things are
potentially variables.
So you'd execute that and get the answer.
And if you were working in a setting where you also had more
complex queries-- so here is one where you're following two
relations--
then you've got to construct a few more constants.
You have to construct the constant for the seed set here.
And here, we got to construct two relations,
and then you got to choose which one of those types of things
to follow, which one of those patterns to use,
but you can just sort of just join them and let the learning
system sort out what you want to do.
So it's relatively simple to do this kind of semantic parsing
in an encoder-decoder framework.
Here are some numbers.
So these are on the left-hand side.
This is varying the number of entities.
So we're going to--
from a small knowledge [? base ?] to one
that has about a million entities.
So you see things get slower.
That's queries per second, so it's getting slower.
The other one is varying the number of relations.
So it's going up between a handful of relations
and about 1,000 relations.
And that straight line on the top
is sort of the implementation of NQL.
These other ones are kind of baseline methods.
And I guess the punch line is the graph on the right, where
this is the speed up over of the baseline methods.
So depending on which baseline you use,
it's either 50x or about 12,000x--
so like four orders of magnitude faster than that.
So this is basically just doing kind of clever tricks
with sparse matrices.
And here are some other numbers.
So this is looking at accuracy of systems
that do this end-to-end semantic parsing
on a few different problems.
So the big one is this WebQSP thing, which is about--
if I remember correctly, about 40-something million triples
in the Knowledge Graph and on the order of like 10 or 15
million entities.
The one that has more complex questions is this MetaQA,
and then there's this synthetic one we developed.
And the models for all these, the decoder models,
look like this.
So that's one for WebQuestionsSP.
That's the one for MetaQA.
That's the one for the grid.
So they're very simple.
And because this model is relatively straightforward
to modify, this is like one extra line of code
to handle one additional property
that this very simple model didn't have.
So that's basically doing exclusions.
And that gets you to the state of the art for these MetaQA
tasks.
So this is basically all I wanted to say.
So just to give you a quick wrap up--
so what NQL is is it's a language
that lets you query a Knowledge Graph in a differentiable way.
It's relatively expressive.
You can drop it into a model anywhere you want.
So the model, the implementation,
we're using now currently scales to around 10 million triples,
one million entities.
And that's running on a set of multiple GPUs.
Those were experiments with using four GPUs on one machine.
So you can go a little bit further than that.
We're working on ways to scale it up even more.
And this makes these very, very simple models
competitive on some pretty hard knowledge-base-type tasks.
We talked about semantic parsing from these sets,
these denotations, and also the knowledge base completion--
something else we've looked at.
And here some more follow-up points.
All right.
So I've got a couple minutes for questions.
SPEAKER: Actually, we're going to bring our next Lightning
Talk speaker up, and then we'll have both of you
up for questions.
WILLIAM COHEN: OK.
SPEAKER: Cool.
WILLIAM COHEN: All right.
SPEAKER: All right, thank you.
[APPLAUSE]