>> Hey, friends. I'm Scott Hansen,
and it is Azure Friday.
I'm here with Emily Lawton,
and we're talking about Cosmos DB again,
except this time, the Spark Connector.
Two things I haven't plugged together in my brain yet,
you got Cosmos DB and Apache Spark.
>> So, as we all know,
real time data is incredibly prevalent around us today,
and people love to use Cosmos DB as the way to query and
persist that real time data for
Web, Mobile, IoT, Applications.
And Cosmos also has support for a number
of aggregation functions such as sum,
min, average, count.
But if you're a data scientist,
some of these functions aren't
sufficient for the types of
complex querying and analytics
that you want to do over your data.
So, I'm excited to be here today to talk about
this new connector we have to bring
together Spark and Cosmos DB.
And essentially, what this connector allows is Cosmos DB
can act as an input source or
an output sync for Apache Spark jobs.
>> Okay. So, if I've got a huge amount of data in
Cosmos or I want to
use Cosmos to a store huge amount of data,
I'm already doing Spark, it just plugs right in?
>> Yes.
>> It just works?
>> Yes.
>> That's pretty cool. Is it an open source connector?
>> This connector is open source.
You can go to our Github and see, well,
the connector itself is not in source
but if you go to our Github,
you can see all the samples and how
to get started and ramped up on it.
>> And do you have some actual live
demos where you plug this right now?
>> We do and I'll go through one shortly.
But I also want to acknowledge the kind of
the two advantages of
bringing these two services together now.
With Spark, you can do
very complex distributed aggregations over your data,
and Cosmos, what that brings the table
is your access to the real time data itself.
And everything that comes along with
a fully managed database service like we have,
automatic indexing, turnkey global distribution,
all those great features you get as well.
>> Do you think that Cosmos is like
the best place to put Spark data.
Like there's lots of places you could
put data that comes from Spark, right?
>> Yeah. Well, the fact that Cosmos DB is just fully
managed and with such great SLAs,
it is a really great place to keep your data.
>> And you use that term aggregations.
I'm not super familiar with that,
like the sum function is a kind of aggregate.
>> So, what Spark actually
does is it distributes out this kind of
functions to a bunch of worker nodes
and then do all the work
on smaller amounts of
the data and then it's aggregated back together.
That's where the power of Spark really lies.
>> Okay. So, it takes
account or sum or min or match, whatever,
and it says, do it across all of these.
Spark is smart.
Does all the math, the mathy stuff.
And then, of course,
combine with another smart global database
which is the Cosmos DB,
you get the best of all worlds.
>> Exactly.
>> That's cool.
>> So, let's jump over to
this Python Notebook here that we have.
Python notebooks are what a lot of people,
data scientists who use Spark,
are most familiar with.
So, before we actually jump into a demo,
let's look at this diagram of
the architecture of how this connector is actually built.
So, when a user makes
a connection between Spark and Cosmos,
what's actually happening under the hood is that
a Spark master node is connected to
Cosmos DB gateway node
to get the partition information from Cosmos.
That information is translated back to Spark and is then
distributed among the worker nodes
which can then interact when a query comes in,
they know which partition in Cosmos to
access and can bring the data back to Spark,
the worker nodes, to kind of do their work on it.
>> And is this largely transparent?
>> The user is not aware
of what's actually happening here.
>> Cool.
>> So, these first cells,
I already pre-baked them.
I'm going to skip over them because
they are just specifying the jars,
the Spark jars, and defining
how the connector should actually talk to Cosmos.
And this next cell creates the flights,
so I should give some context on this demo first.
This is three months of on time flight
performance data from the United States
Department of Transportation Statistics.
>> So, hundreds of thousands, millions of flights?
>> There's a lot of flights here.
I don't think millions but hundreds of thousands.
And so, we create
a Spark data frame here which is essentially
just a distributed collection
of the flight data that's organized into named columns.
It's almost like a relational table, in fact.
Okay , so, the first query
we're actually going to do here is
calculate the top 10 delayed destinations
originating from Seattle.
So, if we run this here.
>> This would be the places not to go.
>> Yes. And if we look at the bar graph,
you can see that the flights to San Francisco from
Seattle are most often delayed or are most delayed.
And then something that another query we can
do that has a function that is
definitely not built-in in Cosmos is we can calculate
the median delays by
destination cities departing from Seattle.
And this uses Spark's percentile
approximation function which, again,
if you're interested in math or algorithms,
I think it's doing some sort
of algorithm similar to a HyperLogLog.
So, it's in an estimation
function but it's still very accurate.
>> So, it's quite a sophisticated algorithm
across a huge amount of data.
>> Exactly. And here,
we see that the city,
the flight from Seattle to Cleveland actually has
the greatest median delay of 12 minutes.
>> That's not too bad.
>> It is not terrible.
Other interesting things that you can do with
the connector or actually do graph queries as well.
And so, what these cells actually do is create
a graph frame which is composed
of Vertex and Edge data frames.
A Vertex data frame contains a special column named ID,
which contains the ID of the Vertex.
And the Edge data frames contain
a source and input column with
the source Vertex and destination Vertex ID.
And so, we can actually figure
out the most important airport
in terms of like the number of
connections going through that airport.
>> Oh, I see. So, we walk the whole graph.
This is, again, a quite
sophisticated aggregate that has a large amount of data.
>> So, this would take like a lot of grouping
and it would take some time to
compose a SQL query that could do this.
But here, we see with Spark,
we can just use
the graph degree method and quickly find that,
as expected, you know a lot about flights.
Atlanta has the greatest number of
connections going through it followed by
Dallas and Chicago.
>> And it's so fast.
>> It's very fast.
That's largely the power of
Spark combined with Cosmos. But they work great together.
Another interesting query that we can do is see if there
are any direct flights between two cities.
So, I pitched San Jose and Buffalo here.
So, this is using Sparks Breadth-first search function,
and if we set the max path length to one,
that means there is no stop in between those two points.
And this query returns no results,
meaning that there are no paths
of length one between these two cities.
>> So, let's try to actually increase
the max length to two.
>> So, this is seeing if there
are flights between Seattle and Buffalo.
>> With one stop.
>> One stop to a location. Okay.
>> And this one has taken a little longer.
>> There's a ton.
>> There are a lot. And so,
we can see that of the top 20 that are shown here.
They all stop in Boston in between.
So, there are two legs of a flight,
that's why the path length is two.
And you can see that this took
only very simple query to do.
>> This might be a dumb question
but if I were going to try
to do these kinds of queries with Cosmos directly,
would I have trouble because I don't have
the aggregates available that I want to have?
>> We don't have this BFS function.
In SQL syntax, this would be very difficult to do.
But if you're not already aware,
we actually do have an API for Gremlin which
is also an Apache graph,
sort of graph way of query and documents.
You could also do something similar using that.
But I think using Spark will be much faster.
>> So, when I make a decision about
to include Apache Spark
in my Cosmos ecosystem of data,
do I need to be thinking about the kinds
of data that I store and the shape of the data?
>> Yeah.
>> Like if I just had a table the books,
I probably don't need this.
>> I think that's very important.
And the cool goal of a roadmap for Cosmos is
being able to have multiple API's so users can,
depending on their dataset, query
with different API's and
use the API that will give them
the best performance. Makes most sense.
>> And it's one of the cool things about
Cosmos is that it supports those multi-way.
>> Yeah.
>> Okay, cool.
So, I can talk to this with Jupyter. What else can I do?
>> So, this is the main way.
We also have support for Scala.
If you like programming and Scala,
we also have some tools for that.
I think Python is the main language
that people who are using Spark
use so this is generally what
we direct people to use and we have most support for it.
If you go to our Github, it's github.com/azure,
I can show you in a moment, /AzureCosmosDBSpark,
and we have a bunch of performance tips,
tools, and how to just get ramped up,
a number of samples,
other Python Notebooks of other sorts of
scenarios that you can go through yourself and to unload.
>> Can I do ML algorithms with Spark?
>> Yeah, I was actually just going to,
I don't think we have time to go
through this whole demo here.
But you can also,
using like a data set that you have in Cosmos,
you can split it in two with
Spark training set in a test set.
You can train a model in your training data.
You can make predictions.
So, what this demo actually
does is predicts whether a flight will be delayed or not.
And so, you can create your ML pipeline.
And in this example,
we ran a logistic regression model.
And I actually did pre-bake this and
this is a confusion matrix that shows the results.
>> What is a confusion matrix?
>> Basically, it shows like the number it splits up.
If you're testing,
evaluating whether an aisle model is good or not,
you can see what the percentages
of times it guessed whether a flight would be delayed,
or if the flight is delayed,
how many times it actually predict that that
flight would be delayed.
>> How often it was delayed.
>> Exactly. So, if we sum
up the true negatives and the true positives here,
meaning, when the algorithm was correct,
we see that we can predict with
over 70 percent accuracy whether a flight
was delayed or not, which is not bad.
And this whole model,
you can train and run in about five minutes.
We don't have time now.
>> That's amazing.
Can people use the Spark Connector for Cosmos today?
>> They can. If you go to our Wiki,
it says, all in public preview.
You can download the tools, you can get started,
run these samples on your own,
and experiment, and do whatever your heart desires.
>> Very cool. Well, thank you so
much for getting me up to speed.
>> Yes, no problem.
>> All right. I learned all about the Spark Connector for
Azure Cosmos DB here on Azure Friday.