>> Data Accelerator for Apache Spark
simplifies the streaming of Big Data within Azure.
It offers a rich,
easy-to-use experience to create, edit,
and manage Apache Spark jobs on
Azure HDInsight or enabling
the full power of the Apache Spark engine.
Encoding is optional.
Jeff Stana is here to show us how it works today on Azure Friday.
[MUSIC]
>> Hey everyone, Donovan Brown here with
another episode of Azure Friday.
I'm here with Jeff and we're going to learn about
data accelerator for Apache Spark.
So before we get too far,
I don't even know what Apache Spark is.
>> Apache Spark is just like plumbing for big data.
It's a cluster for executing data in the Cloud,
it runs in memory, it's five years old.
So it's fun, people are using it,
and it's really great for performance.
>> Got it. So anyone who has seen the big data arena
is going to know what Apache Spark, right?
>> Yeah.
>> Now what we're talking about is
that Data Accelerator for that particular service.
>> Right. So when we started out a couple
of years ago building a production system.
We didn't have a lot of
tools that worked at the scale where we're looking for.
So we moved into Apache Spark
because it was fast and could keep up with our demands,
but a lot of the quality of life pieces
around it just didn't exist yet.
So we spent a lot of our time
building these components so that you have
a rich environment for developing and managing your pipeline.
>> Okay. Got it. So show us what you're going to show us today.
So this is called the Data Accelerator?
>> Yeah. We opened source in April.
So the community can get it and use it themselves.
It get started pretty quick.
We have two ways to get started.
The first is you have an ARM template,
so you can deploy it straight into Azure with a default setting.
The other is we have a Docker image so you can put
it on your own desktop machine,
and do the experimentation without working in Azure directly.
So when you first do a deployment you're going
to see this right here.
Is our portal, it has all the data,
all the tutorials that we have in our GitHub repo here as well.
There's about 20 of them for your on
box scenarios and 25 of them for the Cloud.
So we get here right through all the capabilities straight away.
It's easy to get started and get going.
>> So when I deploy this to Azure,
this is what I'll see or if I run
the container locally on my desktop,
and I go to the right port on my boy browser,
this is the experience I'm going to get immediately?
>> Exactly. You're going to be here.
>> All right. Perfect.
>> Right. The first place you want to go is straight to Flows.
Flows are basically pipelines.
So anytime we want to take data from
one place to another in Azure, that's a pipeline.
We are going to go right into the IoT sample.
The IoT sample here is pre-configured.
It's one of our tutorials.
We have the tabs here in order on the left so that you go
down the steps that you would do
when you're configuring your own pipeline.
So first you'll name, not a lots to do there.
Now we get to how you're going to define it.
A data Accelerator provides streaming right now,
we're working on batching that should be
out in the next few weeks,
we have a number of data sources you can
connect into the streaming pipeline,
so we can source some of these various places.
We have the, naming your hub for this case,
your connection information, we have your subscription ID,
and a bunch of data around,
all the configuration options you need in order to connect
Data Accelerator to your data source
which may be in a different subscription,
and your output destinations which
may also be in a different subscription.
It's a full many-to-many situation here.
Once you've done this configuration,
your pipeline will deploy.
Then we move on to the rules.
The things you're going to do on the data as it comes through.
For this, we have a few by default in this scenario.
But as you can see here we have a Query Builder Interface.
You can construct your rules on
your dataset without writing any code right here.
>> So does the data have to come in and JSON or XML?
>> That's a great question.
Because in the next step we get right
to an example of what we call Implicit Schema Detection.
That was actually right here.
So we have a Get Schema button right here and what it does is,
it samples the data flowing through your Event Hub,
flowing 15 seconds in this case,
and it identifies all the JSON it finds,
and it groups it together and loads it into
the object model for the rest of the tool views.
>> You said JSON and I just want to be very specific if
my data source is generating XML, it won't work?
>> It reads everything that makes it into an Event Hub.
So once it's in an Event Hub then we read it
and we store the schema as JSON.
>> Got it.
>> Yeah. The rules know about that object model.
So when we say pick,
what kind of event type it is. It already pre-loads.
But we can go a step deeper.
If you want to get in and actually write the queries yourself.
We have that ability right here.
Demos ships with a large number of
basic queries which are showing you, what it all does.
Some of the interesting things we have here is we have
an output macro that we've added to Spark SQL.
It basically wraps up how you put
your data in a particular location after you're done.
Because there's a lot of ways you can fat finger
in and mix up your output,
and this just macros it for
you so you have to write that code and maintain it.
>> Okay.
>> We also have things like Windows,
windowing, sliding windows, tumbling windows,
so that you can aggregate your data and wait for
late events or pull
in the things that you're missing before you move on.
The last piece of this is
motivated by a lot of our experience starting up.
When I would work and write a script,
it would be submitted to the cluster.
The queue would be about two hours long.
The provisioning and execution is about 15 minutes,
and my job would take two minutes.
So I went in two hours just to see whether
or not I wrote something wrong.
So I'm going to go ahead and do something wrong here.
What we're going to do is we're
going to see a real time running experience.
What we've done is, we've sampled your kernel.
Now instead of waiting two hours to go
through that queue and fight for everyone else for resources,
right there, I got the answer.
So I can go and I can fix my code.
>> So this really helps your inter loop?
>> Yes it does. It really, really tightens it up.
Oh, it doesn't handle the spaces.
I'll run it again and it selects a query.
>> There we go.
>> There we go. So I get
my feedback straight away and now it's
saying it doesn't recognize the type.
It's like of course,
I have a tuna sandwich right here
and if I go down here, I have a status.
>> This would be really painful if I had to wait some hours,
two hours for a typo.
>> For a typo. Your spending a lot
of time not getting a lot of value.
So we bring that up so that we could get the,
right here and now we've got it all.
>> When I heard the title
Data Accelerator for some reason in my mind I immediately thought
about just speeding up
the ingestive data but
this is actually providing a lot more value.
This is actually, I'm going like in
a development environment which allows me to
have a much tighter and quicker inter-loop
of my development process.
>> Yeah, that's a big focus.
We are the Developer Tools organization who built this.
So for us this is the VS Code Editor that we have.
You can change themes, we have IntelliSense,
it's all right there in the code
>> It's completely web-based so you can access it from
anywhere once you have deployed in Azure or [inaudible]
>> Exactly, once you have access to your subscription in
your portal that thing that you can just get what you need here.
So if we pop out and I can do it.
>> You're just trying to get IntelliSense as well?
>> Yeah.
>> So I just might get rid of
status and then dot after details is close.
>> That's not.
>> There we go.
>> There we go. So it
knows everything about the schema we detected earlier.
This is great when you're dealing with
real systems because it will have thousands of versions
of the product in market at
the same time or the piece of hardware.
So you'll have small variations version to version.
The schema is going to say, "These all look the same."
So it's going to help you paste over,
how to get from one to another in the same document model.
Which is usually like an eight hour operation for just
onboarding your schema types.
>> What I think you showed this one is an IoT example.
So this could be data coming from
an automobile or a factory or from anywhere.
We have all these small device sitting tons of data.
We now have a very easy way for us to be able
to create the rules in the interact sack,
you talking about the inner loop because we have it
as even non bigoted developers right there.
There's that time that you spend
on your machine trying to build that and you want to
make that as efficient as you possibly can so that you
can find those errors before you push it out to be deployed,
which is what you are able to do now
>> Right. Exactly. You could be
production ready before you hit go,
before you push it at all. Which is awesome.
It's super critical because
you've got 20 years of legacy to work with if you're in IoT.
That's a production piece of hardware deprecations cycle.
If you've got a product that's been out for a long time,
Visual Studio IDE is the variable product of this point,
data over many, many versions.
So pulling it all together and being able to work across it
very quickly has been a great win for us.
Of course, we've got scheming as well, is right there.
>> You got to have the dark thing
>> That's right. So we can get our rules
and executed very quickly.
We can also do more, we can add in reference data.
A lot of people will have
fact tables that they just know about their environment.
You want to be able to say,
"This temperature sensor is located on this road,
in this warehouse, in this city, in this the state."
That's where you pull this in and you can find it anywhere.
We also pulled in functions.
So you can come in and add
User-Defined Functions, Azure Functions,
anything you need, you can write your code,
you can have it there, we can just
reference it while we're executing in the pipeline.
>> Got you. So this is a way for me to
almost have like a filter or some type of
data manipulation as part of my pipeline that I've
written myself not relying upon the information provided?
>> Exactly. You can leverage the work
your company or your team has done over the years,
and put it right into your pipeline today.
Last piece we have here are outputs where you send it.
We support Blobs, Cosmos DB, Event Hubs,
so you can send it from Event Hub to Event Hub and SQL Database.
Also by default we have a metrics dashboard.
For scale there's a couple sliders symbol
for hoping up and down scaling for you.
The metrics board here that comes with it,
is going to give us summaries of what's going on on your pipeline.
Each of the rules you set up shows up as a metric.
So you build your rules and it's right here.
The samples are powered by a simulator for events.
So you don't need to plug it into a live Event Hub to get started.
You can see it all going from the start. That's great.
So for this case we've already ingested 70,000 events
today and we have
our closed alerts have gone off the number of times.
These alerts now paste in webpage but you can
also plug in like have it texts you or have it send an e-mail.
So you can get your learning right here out of your pipeline.
>> Awesome.
>> Yeah.
>> Then you said this is all open source, right?
>> Yeah, we've open source this entire pipeline.
For us we've been using it in production for two years.
We have very large deployment,
we're handling terabytes a day. That's big.
>> Yeah.
>> The open source project we've put out there we're maintaining,
we're running it right off of that repo.
So we've gone ahead and we've added support for Kafka.
We were adding support now for Databricks
and we're listening to feedback.
So if people are taking this in a direction that we
hadn't thought of or see something else to bring together,
we are interested in that.
>> Where do they go to contribute that?
>> So to get to that we have our GitHub repo right here.
GitHub.commicrosoft/dataaccelerator. As you can
see we have our code and issues and straight here.
This is very similar to the portal we saw.
Again, all the information that you need to get
started is right here in the GitHub page.
>> Awesome. Thank you so much.
We're learning all about Data Accelerator
for Apache Spark here on Azure Friday.
[MUSIC]