- Hello everyone.
I'm really excited to be here because just last week
we launched our official connector for Apache Spark
for Neo4j and Apache Spark.
So it's a fairly good fit and timely.
And I quickly want to show you how this would look like
and how it would work.
So, basically I'm leading the Developer Relations Team
at Neo4j and if you want to find more about me,
you can look at this contact details.
Neo4j is a graph database, and why are graphs important?
Because oftentimes there's much more value in the
relationships between entities than in the entities
themselves.
And if you can take the topology and the context into
account, then you oftentimes get much more value out of
your data.
So that's just a little bit often
baseline.
And actually I have a really cool data sets.
So it's the FinCEN files from the ICIJ,
which is the Investigative Consortium,
International Consortium of Investigative Journalists.
And they got access to a leak of defense and authority
which is part of the US Treasury where banks need to
report their
suspicious activities to that authority.
And they got an access to part of that,
and published some of the data that they got.
So there's basically the suspicious activity reports
which contain information about which bank filed a report,
which bank kind of vetted where did the money come from,
where did the money go to?
And what's really interesting is that even in
that tiny, tiny, tiny sample, it was already like
2 trillion dollars of money that kind of flew around
the globe from what they got.
And as they published the data, I thought it would
be a really cool dataset to look at from this perspective.
So in graph model,
I created a very simplified version of the graph model.
So you basically have just one bank which is the sender
of the money,
one bank which was the receiver of your money or
the recipient of money,
and then it transferred relationship with amount and date
on the relationship.
So kind of how the two banks interact with each other
and what kind of volume of money did flow between
the two banks, so it's clearly simplistic cause you only
have 10 minutes to go into this.
And what I can do now is kind of,
I can make use basically of Spark or Databricks and Neo4j
for each of them to do what they are meant to do best.
So basically do all your data processing and pre-processing
in Spark, ingest it into Neo4j for advanced graph analytics
and then get the results back for my integrating it
into my MLflow, for instance, as such.
So in this kind of lower box of data integration is where
the, so this, the data science aspect.
At very Neo4j connector for Apache Spark comes in
which is a bi-directional connector between the two.
So wherever Spark runs, you can connect it to Neo4j,
send data to the graph, get the data back,
and I just want to show you how this looks like.
Basically as you build this new version on the data source
API, so it's available through Python, through Scala
or R so you can pick your choice of language.
These examples on Python here.
And on left side you see kind of examples for writing data
and reading data from the Neo4j into Sparks.
So you basically get a data frame back based
on the data that you request from the graph database.
On the right side, you see kind of how you would connect
to the graph you have in store right from spark into Neo4j .
For practical example, I kind of have this kind
of tiny pipeline and a nodebook, very busy, get the data
from the ICIJ CSV file or as table.
Either way as possible.
Of course, we load in data frame and then used to connector
here to create the graph database from its nodes own
and relationships.
And then you run a graph compute and we call
the graph compute again from Spark to get the results back
into Spark, into a data frame.
And then I could use ML-Compute I could train a model
for prediction or classification or other things based on
for instance embeddings that I couldn't compute in
the graph as such.
So that's on a high level.
Now, I want to kind of show you what this looks like.
So I set this up as a nodebook in, in Databricks.
Basically having all these steps.
So I initially got the CSV from GitHub.
Here is the CSV data of the banks,
source and target bank and the volume of money that flew
between these banks.
And so we
see this here
and then I can use this as an input data frame
for what I want to do.
As the first step, I create the nodes, the attitude nodes,
bank nodes basically from that
and then just create an union.
The source and target into one column.
ID column.
And I just create entities.
So it's called an entity label that identifies the type
here based on the ID.
So it's really anemic entities.
So I didn't want to add more detail that we don't need
for the demo here.
So it's all the banks that we have in here.
And then as the next thing, the relationships.
And I connected it to a Neo4j Sandbox instance, which are
on the fly playgrounds for Neo4j that you can use
and it has connection details and you can open the Sandbox
with an Neo4j browser or Bloom visualization tool.
And if I sent the data to Neo4j I'll show you how this works
What it looks like.
So I initially start with the entities
and then
do the same with relationships.
So I basically gave a source and target ID for the banks
and the sum optimum money that flows between the banks.
And as you can see, it's quite a lot of money.
So these are up to $2.7 billion that flow between two banks
which is really scary in some way.
So, and then I use that to kind of map these
ID's data frame
to relationships.
Meaning I have a source entity and target entity
and create and transferred relationship by
the amount properties
stored on the node.
And after I've run this, which just takes a few seconds
I get my data in a graph.
So if I look...Want to look at this
I can look at my data in a graph.
So I see I have a few dollars in banks and a few dollars
in transfer relationships, which looks like this.
Or if I want to go at a larger scale I can go into
Neo4j Bloom and then do like graph based pattern searches.
I don't need to know the clear language
and then kind of zoom in, and then look at
are there certain structures that I want to look at
or details of any of these things.
Okay.
And that's kind of the first step so we can
of course use standard connector in the other opposite
direction as well.
So we can also use this to read data from the graph.
So this is matching source entity transferred money
to target entity and just get the amounts.
And as you can see, that's between the Amsterdam Trade Bank
and Rosbank in Russia, there's 2.7 million, billion dollars
that went from.. into this direction.
So does this all nice and well, but of course this is not
not really a variant from what I can do in Spark.
What I can do now is I can use the graph data science tools
of Neo4j to do more advanced computation.
So these examples here are just simple picturing
and clustering, but you can use the same for computing
graph embeddings with random projections,
graph embedding algorithms that are also part of the
Neo4j Graph Data Science Library.
So here's basically, if you have to creative it's called
picturing, search in the Neo4j I can say I want to kind of
have... take relationship properties in the columns
like rates and do the search.
Project the graph into a
projected graph on ML compute.
And then I just find this query and then I get
the results back as a data frame which I can ofcourse
can also visualize.
So picturing is interesting because it kind of shows
the trends that money follows.
So if a bank transfers money from A to B to C to D
then the bank D which received all the money basically
has the highest picturing because it's kind of transitive
importance.
And of course we can also do other things.
So yeah, I put in clustering algorithm
which kind of returns errors of data.
So it would look similar if you copied
basically the embeddings for nodes
just then you have like 1024 size vectors
or 10 source of data that you can then use.
So it stops here because I didn't have much more time
as such.
So I have in the slides
I also have some like examples
what I can do next is kind of actually use the graph
embeddings so I don't kind of of encode
what's the meaning of each entity in my graph
based on topology and attributes.
And then kind of develop KNN similarity networks
which then show how similar are entities based
on embeddings or based on other attributes.
And I can also, as I mentioned, train ML Models.
If you wants to learn more about this just graph data
science link here and we also have O'Reilly book
which I also got here.
So when you, if you want to get a free ebook for this
which has examples for Spark and Neo4j on graph algorithms
you can find this here.
And
that's...
And if you want to do it more about the connector, there's
a learning page for that.
Neo4j.com/developer/spark
or if it's Sandbox under try-neo4j.
So, and I hope I didn't run over in terms of time.
- No, that was great.
Perfect timing
- I also
- Just one last thing.
I also have an get up repository with the stuff I also added
to the slides and also add the nodebook as well.
So you can run it in your own environment as well.
Cool. That's it for me. - Awesome