Hi, I'm Barak Michener, a software engineer
on Google's Knowledge Team.
One of my favorite additions to Google search
has been the integration of Knowledge Graph.
We've been working to add great features, based
on facts and relationships, to more directly answer
your questions every day.
From charting and comparing public data about countries,
to relating facts and entities that allow our users
to more easily explore what they're interested in.
All possible because these facts are
related inside Google's Knowledge Graph,
based on publicly accessible data from Freebase, Wikipedia,
and the web.
But why shouldn't you, as a developer,
be able to build your own smaller Knowledge Graph?
Today, I'm excited to show you around my open source
project, Cayley, an open-source graph database.
Written in Go and inspired by the graph database
behind freebase.com, it stores structured data
and retrieves deeper connections,
much like Knowledge Graph does for Google, itself.
So, here's what I'll cover.
First, how Cayley fits into a development stack.
Then, I'll demo Cayley and step through writing
a few simple queries.
Next, if you're coming to this video from Part 1
of this series, we'll then use Cayley to load and query
the music data that Jarek generated.
And finally, we get to have a little fun
and show off visualizations that come built in.
So, where does a graph database fit within your data stack?
A graph isn't meant to be a replacement
for traditional data stores.
SQL is still great for tabular data,
and Key Value stores are great for scale and speed.
But graph stores are meant to provide developers
with another tool, to query graph-shaped data.
What form does graph-shaped data take, and why is it useful?
Graph data comes in the form of triples.
These are the edges of our graph, where
some entity relates to another entity
through some type of relationship.
For instance, Arthur Cayley, the mathematician
for whom the project is named, was born in the city of London.
A triple is simply these three strings in our graph.
Further, the relationships you use, or as it's called,
your schema, depends on the specifics of your data set.
These are sort of the phrases you might use,
converted into a machine readable form.
There are many kinds.
I use a lot of Freebase schema, but Jarek based his music store
on schema from schema.org.
By thinking in terms of entities, the nodes,
and relationships, the edges, we can define a complete schema,
a way for graph databases to represent connections
that humans understand intuitively.
But why use another type of data store?
Graphs afford you a few key things.
One, the ability to simply represent more complex data
structures, such as hierarchies and lattices.
It's also great for social graphs.
Two, the ability to more flexibly change
your relationships.
Where SQL has traditional schema-first decisions
up front, and key value stores are fast and schema free,
a graph straddles the line by allowing schema
to be more easily changed.
What one might call "schema-last".
Three, graphs have the ability to have
self-defining relationships that can form a greater
ontology toward greater abstraction and composability.
Which is really just a fancy way of talking
about combining relationships to form new relationships.
Like inferring that my father's father is my grandfather.
And finally, the ability to use a broader mathematical theory.
Graph theory provides many algorithms and metrics,
like distance topologies, cliques, and neighborhoods,
all fairly intuitively derived.
So that's a bit about where and why graph data can be useful.
Next, let's dive right in and take a look at Cayley
and run a few queries against the music graph.
Here's the Query Editor for Cayley.
The editor comes with the server and uses the same HTTP API
that your applications might use.
It's a way of exploring your data
and testing your queries on the fly.
In this instance, we've loaded a data
set on music, sourced from the Freebase data
dumps provided by Google.
Because Freebase is open to the community,
its data-- about 2 billion triples in total--
is licensed in the Creative Commons.
For this data set, I literally ran grep across the data dumps
and pulled out anything related to music
and all of the name relationships.
Which happens to be 134 million triples, altogether.
Back to the Query Editor.
First, you already know the query language.
It's just JavaScript, the lingua franca of the web,
executed by the graph server, itself.
There's a special variable called Graph, or just g,
with some special methods.
Emit just adds the value that's passed to the results set.
To access the data in the graph, we start by defining a path.
Graph dot vertex, or equivalently,
g.v as is very common, defines where we start.
So, let's find the name of our artist, Dual Core.
I add "all" to actually run the query
and emit all the nodes that it finds.
And sure enough, that name is in our graph.
This however, is just the name, a literal string.
There are many Dual Cores in Freebase, and in the world.
The processor type has the same name,
as does, to some engineers, a type of electrical wiring.
So we need to disambiguate.
If we follow all links that point
into this string over the name relationship in Freebase,
we should see all the things named Dual Core.
And indeed, we see there are a couple.
Well, we know that the Dual Core we're looking for
is a musical artist.
So quickly, let's take a look at a selection
of artists from the graph.
Starting from the entire graph, by passing no arguments
to Vertex, find all things that have a type of music artist.
But let's only get 20 of these.
Well, that query seems to work.
We get a set of IDs from musical artists.
You may have already noticed the way
a query is generally structured.
Start somewhere, follow some path,
and then tell the database to run the query.
That can be all, for everything, get limit,
for only a handful, or a couple other options,
covered in the documentation.
So, we know how to get artists, and we
know how to get things named Dual Core.
So let's combine these two paths to try
to find the one Dual Core we're talking about.
We add all to make sure we're getting all of them.
And in fact, we find we get exactly one.
Here's the identifier for the band, Dual Core.
It may look odd, but it's all from machines, right down
to the namespace prefix.
Machine identifiers are the key to graph databases.
Think of them as primary keys, a way
of representing a single concept in unique way.
Like a barcode does for products in the store.
Well, let's find out where they're from.
I've refactored the query slightly--
it's just JavaScript-- to make a function, getArtist.
From there, let's follow the relationship between a band
and its home city, music artist origin.
This gets us to the machine identifier
for the city they're from.
But there are two of them.
If we follow the name relationship again,
we can find out exactly where they're from, just by the name.
And it turns out they're from Cincinnati in the United States
and greater Manchester in the UK.
Since they collaborate online, that make sense.
If you are worried about the efficiency of JavaScript,
a cool feature of Cayley is that when you call a running method,
like all or get limit, the query at that point
is compiled into a go structure and run natively.
So what more can we do, now that we have the band?
Since graph data is flexible, we can
ask for any property we have for artists.
If we change the last query just slightly
and ask for genre instead, we get the genres, nerdcore
and hip hop, that they're associated with in Freebase.
Now, if we can do that for Freebase data,
let's talk about how we can combine that with the data
that Jarek generated and crawled from his music site.
The next step is easy.
We just take the data that Jarek provided and load it.
Cayley provides a generic writer interface,
but again, it could as easily be done
from an external process over HTTP.
We get the message that we've loaded 113 triples.
In the context of a music site, that
might be writing multiple facts when the artist page is edited,
for example.
We have two graphs, and loading both is easy.
But how do we know that the band on Jarek's site
is the same as a band in Freebase?
It's similar to the name problem earlier, but in reverse.
We're trying to determine if one band in one data set
is the same band in another.
This is called reconciliation, and it's an art
as much as it is a computer science problem.
It happens even in conversation.
I'll mention Sean, and you'll ask, "Which Sean?"
And I'll reconcile, "Oh, Jarek's friend who deejays."
To help do the same conversational trick
for machines, there's a special link
called sameAs, which will tell us
that a machine identifier in one graph
is the exact same concept in another.
And fortunately, Jarek has already
added these to the link data on his site,
so it's just another triple.
Let's use that.
Here's the URI that Jarek's site uses, straight from the web.
It doesn't have a Freebase name, either.
I would try that, but again, it has information
that isn't in Freebase schema yet.
However, we know it has a sameAs link,
so we can follow that link first,
which takes us to the Freebase identifier.
Exactly the one we already found.
Now that we've managed Jarek's data into our graph,
it's just the same as before.
We have a Freebase entity.
Let's go ahead and find the genres and hometown
for the band.
We copy and paste our thing.
We change the strings in music artist origin, and we run it.
We get exactly the same result.
We could obviously do the reverse.
We could take the genre and find the entities
on Jarek's site that have that genre.
Or any combination of fun connections.
That should give you a sense of what
traversing a graph database feels like.
Hopefully, it wasn't too unfamiliar.
There's a lot more that this talk is too short to cover,
but it's all on the project site.
So, now that we've done the hard work,
next comes my favorite part.
We get to play with the data and look
at the graph in interesting ways.
Here's the query from earlier, Dual Core genres and location.
With a few extra tricks for readability,
we can ask about artists that are similar to Dual Core.
That is, have the same genre and the same location.
If that seems a little confusing,
that's what query shape helps you look at,
the overall structure of your traversals.
Until now, we've had straight lines.
Dual Core through the name, through the genre, to the name,
again.
But here, we see something that branches off.
The top node shows a relationship
to the band on one side, through their genre,
and to the band on the other, through their city of origin.
In this way, queries are composable,
making it easy to build complicated relationships out
of simpler parts.
But what about the shape of the graph, itself?
Can we generate insight into the music influence
of an entire city?
So, this query may look daunting,
but it uses exactly the same schema we've used today.
Find 1,000 artists, make sure they're
from a place named Brooklyn, and get their genres.
With the extra source and target tags in place,
we can switch to the Visualize function, which is a simple web
app using sigma.js on the client side.
And when it runs, it gives us a picture
of the music graphs in Brooklyn.
You might notice a significant East Coast hip hop hub,
with smaller hubs for rock and indie rock in the center.
All of the bands seem fairly well connected,
and they cross genres quite often.
It's different across the country, however.
So, let's take a moment and re-run the query
and look into Oakland, California.
As it runs, we start to see there's a great hip hop scene,
but other bands don't crossover so much.
It's a little more stratified, it's a completely different
shape, and it's cool and informative.
That's just a sampling, but hopefully it
sparked your imagination on how graphs and graph
data are useful for developers and can create fascinating apps
for your users, much as Google does using the Knowledge Graph.
Thank you so much for watching.
Check out the Cayley project page,
or play with a smaller live version on App Engine.
Or join Jarek for the next video in this series.
Have a great day.