>> Not only can you run Databricks notebooks
in Azure Data Factory,
now you can submit JARs and Python scripts to
Azure Databricks clusters using
new activities and Data Factory.
Gaurav Malhotra is here to tell us
all about it today in Azure Friday.
Hello! This is Lara Rubbelke
and today on Azure Friday I have
Gaurav who is a senior program manager
with Azure Data Factory.
He is going to walk through
some great new updates that we have with
our integration between Azure Databricks
and Azure Data Factory.
So, what are you going to show us today?
>> Yeah, thanks Lara for having me here.
So, last time when I was here we talked about that,
we have Databricks as
a compute engine supported in Data Factory.
Where you can use Databricks notebook
as an activity which is a step in your pipeline and
they're typical pattern looks like what you
see on your screen where you have data
coming in from On-Prem
Cloud and you're ingesting using Data Factory into
the Lake and then followed by that you
are prepping and training the data using Databricks.
So, last time we said that you can use
the Databricks notebooks where you've
authored your Python, Java, Scala code.
You can use that as a step in
your pipeline and then load the data into DW.
So, what we're seeing today is
that in addition to those capabilities,
we are announcing new capabilities
where not only notebooks,
but you can also submit a JAR.
So, if you have a Java code
and you have combined as a JAR,
you can now submit the JAR to
a Databricks cluster and if you had a Python script,
let's say you have a Python script
for retraining your model,
then you can also submit
a Python script to the Databricks cluster.
Both JAR and Python are supported as
first class activities in Data Factory now,
which you can submit over
Azure Databricks compute clusters.
>> That's fantastic. So, you
have something you'd like to show us?
How this works together?
>> Yes. So, I have a very interesting demo,
but before we dive into the demo,
I wanted to show you a pattern on
how typically this is used.
So, as you see on your screen,
we have data coming in from Amazon S3 to Blob and we have
some data in On Prem sequel server or
some reference data that we
have that we are moving to Blob.
You could have a notebook where you have written
your scholar code maybe to cleanse
some data and this is
the Databricks notebook that's
running that cleansing process.
Once that completes on success,
you could have a JAR
which does your classification of the data.
So, here in this case I found some JAR,
maybe a third party JAR that I want to use
as part of the step in my pipeline.
So, I have a machine learning
library classification step that
submits a JAR to the Databricks cluster.
Also on success, I'm trying to train my model,
which I'm using and here for training my model,
I have a Python script that
I'm submitting to do third database cluster.
If my classification succeeds then I'm
loading the data to the data warehouse.
So, let's dive deeper into how you can submit
the JAR and Python scripts to your database cluster.
>> Yeah. I'd like to see that. Great.
>> So, what I have are two examples.
The first one is that I have over here
Pipeline where you have a JAR activities,
so again it's simple drag and drop.
If you see I have a pipeline and there is this,
the way it's organized is
activities run on what computes,
so these notebook JAR
and Python activity runs on Databricks.
So, you can just drag and drop
a JAR activity onto the canvas.
In this case I already have one
and what you need to specify the configuration.
So, you specify what Databricks
cluster you want to run it on,
which I've already selected over here.
If you want to take a look at how it looks
it's cluster that I have in the East US Region.
I could have an access to get specified over here,
or I could pull it from a keyboard.
I specify that it's a new job cluster that I want to use.
I had the option to use
an existing interactive cluster as well.
Again, I specify the cluster version,
the number of working nodes,
if I wanted to enable or to scaling.
So, minimum and maximum number of nodes that
you want to your Databricks cluster to spinoff.
If it start with the minimum and
as the load increases it will keep
going up to the maximum count that you specify over here,
and what type of
nodes do you want in the Databricks cluster.
Then we have some advanced settings if you want to pass
some conflagration or tags, you can.
As you can see in this particular case,
I'm bossing a custom tag,
which is business group,
it's a finance group.
The value for that. So.
>> You have actually a lot of flexibility
then in how you want to configure a cluster
within certain parameters or
ranges and that allows you then
to balance the cost and
the speed in which you want these JARs to execute.
>> Yes. So, we're giving full flexibility to the user
in terms of configuration on what they want to select.
Because in some cases if it's
a test environment you might do
with it to work a node cluster.
But if it's a production environment,
you might have 15 nodes
and there you might want to enable
a autoscaling to start with
a minimum node count and
go up till the maximum node count.
>> Right.
>> So, once you specify your Azure Databricks cluster,
which is a link service in Data Factory.
It's just a connection to your Databricks cluster,
then you specify the settings.
So here, you can see
that for the JAR that I'm going to run,
this is the class which is the entry point for that JAR.
Here I'm just calculating the value of pi.
So, this is a example on
Databricks documentation that you can
see how it works in terms of calculating the value of pi,
it's a mathematical symbol.
Here, you could blossom and
parameters and for the JAR that I'm running,
it is already in their DBFS and you
can see the path that I'm specifying over here.
So, you just mentioned that
bot and Data Factory will automatically pick it up.
So, if I were to just remove this and again
all the features that we had with Data Factory
in terms for you to just do a test run,
as right now I'm doing a test run and it's
going to start an execution.
At this point it's going to spin up
a job cluster and do the execution
which will take around about four to
five minutes and this will run.
Or if you are ready to trigger this Azure schedule,
you can always board a new trigger on
top and say that I want to run it on the schedule.
>> Your standard, ADF, capabilities,
are all integrated together. Great.
>> Yes, all of those are available.
For this one I already did a run.
So, you can see that
I'm just going to discard any changes that are there.
So, we have this monitoring interface and I already
did a bricks JAR pipelined run and
you can see that the activity run inside that and you can
see the inputs that were passed to this the JAR.
So, all the values that are resolved
for the parameters that we are
calling the Spark by class and the outputs that produce.
If we want to look, look at the run logs,
you can look at it and all of
those standard monitoring characteristics
for a particular run are
available for you to look at right here.
>> Great. What if something goes wrong?
>> Yes so, if something goes wrong,
then we also provide you
all the information in the monitoring.
I would use the other example here to show that.
Which is the spark,
like I said that not only you can submit a JAR,
but you also have
a Python code that if
you want somewhere to
your Databricks cluster you could do that.
In this case I have a Python Spark activity,
again the standard thing you select
the Databricks cluster and you specify the file.
So, what I'm doing separately or here
is that I'm constructing the file path.
So, I have defined two parameters,
which is my folder path and filename.
These parameters get resolved to put some default values,
but once you run these,
you can always change the values for these and do that.
>> Okay.
>> So, let's say in this particular case if
I- this is a script
which again calculates the value of pi,
but I have a buggy script over there,
where I'm dividing it by zero.
>> Okay.
>> Which is an error. So, if I run that and trigger
this particular process on
the cluster, it's going to run.
Again, you can look at the monitoring view
as you can see this one is progress,
but I already did this run earlier as well.
So, you can see that once you look at the activity runs,
the entered information is right here and not only that,
you can essentially click
that to directly go to your Databricks cluster.
>> So, that's the tight integration
that you were talking about.
So, you're still in Azure Data Factory,
but you can launch and get right into
the error information that is sitting
in your Azure Databricks.
>> Yes. You can see that right now I am on
the Azure Databricks Ux and you can look
at the logs right here and you
can see that I divided it by zero,
so, it is throwing that particular error.
So, is way easy for our customers to use these as
a step in their pipeline and then
navigate to Databricks to look at the logs,
and we are working to make this even tighter,
to even display these errors inside Data Factory.
>> Fantastic. All of this is available today?
>> Yes. All of this is available today.
You can go to Data Factory documentation.
Actually I'm going to show you right now.
So, all of this is available on our public documentation.
I should have opened this up earlier.
So, it's in Azure Data Factory.
If you go to how to guides,
we have transformed data,
and then we have the Databricks JAR activity,
all the documentation here.
>> Great. So, you can go and walk straight through it.
>> Yes, and you can look at the Python and all of
the examples are there for you to try it right now.
>> That's great. That's great.
These are really great updates,
and I love that tight integration that you have
both with Python as well as the JAR.
In addition to what you had spoken
about last week which was
tight integration with the notebook still.
It's that Databricks it's
getting better and better with that Azure Data Factory.
>> Yes. Absolutely.
>> Great. Okay. Well, thank you so much for joining
us today and showing off some of the great updates,
and we look forward to hearing more and more.
>> My pleasure. Thank you.
>> Azure Data Factory. Once again
this is Lara Rubbelke with
Azure Friday and we've been learning all about
Azure Data Factory and Databricks.