>> Hey friends, I am Scott Hanselman.
It's another episode of Azure Friday.
I'm here with Gaurav Malhotra and we are
talking about Azure Databricks.
>> Yes we're talking about
Azure Databricks integration with Azure Data Factory.
And how you can operationalize
your pipelines that are using
Databricks notebooks as a step in
your entire ETL or ELT workflow.
>> Okay so ETL, Extract?
>> Transform and Load or Extract Load or Transform.
So it depends on your scenario,
data integration scenario that you are targeting.
So, Databricks GA was announced in March,
and with Databricks GA comes with an ability to do ETL,
ELT with Azure Data Factory.
Now, Azure Databricks is a fast,
easy, and collaborative
Apache's Spark-based analytics platform.
And you can build your notebooks,
and you can write code in Python, Scala in that.
And essentially you could shape your data,
transform your data using
the notebooks and the code
that you have written in your notebooks.
Now what we're saying with this integration with
Data Factory is that not only you can ingest
data from 60-plus sources
that Data Factory support study into Azure,
but you can also now prep and shape the data
with this Databricks integration using Data Factory.
And you can essentially operationalize
your pipeline end-to-end using Data Factory.
>> Maybe you can expand a little bit more
on what that means operationalize,
like formalize and make it
such that by the time it gets into the data warehouse.
It's correctly shaped,
appropriately joined, it's cleaned,
the closer and closer it gets to the dashboard,
the better looking that data has to be.
The less raw the data is.
>> Yes exactly.
So, essentially after you ingest the data
using the Data Factory Connectors.
The data is still in its raw shape right?
So you essentially want to shape the data into a format
that essentially your downstream applications can accept.
Now Databricks would allow you to
do those transformation in the form
of notebooks that you can write Python,
Scala code in that.
And what we are saying is that with Data Factory,
once you ingest the data,
you can essentially operationalize
your notebooks as a step in your pipelines as well.
And not only you can
operationalize but you can monitor and manage it
from a single pane of
glass using your Data Factory pipelines.
>> Nice.
>> Yes. So let's see how this can be done.
>> Okay.
>> So essentially once you provision a Data Factory,
you land onto a blade like this.
And you click "Author and Monitor" and it
launches a brand new UX that we announced,
public preview in January of this year.
Now, once you go click "Create Pipeline" you
essentially get a blank canvas with Data Factory.
So this is our Author tab,
and you can see over here that in
this bank pipeline Databricks is
one of the entity and we
have a notebook activity under Databricks.
So you can essentially drag and drop
this notebook activity onto the canvas.
You can start to provide all of the settings,
so you need to first give
the connection to your Azure Databricks account.
So in Data Factory terms it's called the linked service.
So either you can create a new one and
here usability is again the key.
So, as you select options over
here so let's say you want to use
the Databricks region in East
U.S. For you to generate another token we
essentially have made this
super easy for you to click these links and will
automatically take you to
your Databricks account where you can
generate the token and put it over there.
As you can see that I'm on
my Databricks workspace right now
and I can generate a token and just
put it on my linked service over here.
So, for the interest of time
I've just created a linked service and you can
see over here that in this case I'm
using a Databricks account,
my access token I can
provide whether I want to use a new cluster,
or an existing cluster.
So in this case, I'm using
a new cluster and all the cluster versions,
that today Databricks support is
available or here for you to select from here.
You can essentially tell us the node types as
well whether you want a standard D-series,
or whichever version you want to use,
and number of nodes that you want
us to spin up for your Databricks cluster.
You can also pass configurations
to your Azure Databricks clusters.
In this case I'm passing
some Spark configuration as you can see over here.
And once I define my linked service then all I need to
provide is the path to
my notebook in my Databricks account.
And this notebook I've just created
a demo notebook over there so I'm
just going to provide this path.
>> Are those specific to a user or an organization?
Can I can share them with my co-workers?
>> Yes. So essentially all these notebooks are
there in your Databricks workspaces right?
So, once you provision
your Databricks account and
you essentially launch your workspace.
Every user who has access to
your workspace can look at these notebooks and you
can essentially control that access as well.
For the users who can or cannot access these notebooks.
So once these opens up you can see
that I have all my recent notebooks over here.
In this case what I'm using
is this demo notebook and I'm not
doing anything fancy over here
just grabbing some context,
Spark context and creating a data frame out of it,
and just displaying that data frame.
So, I have this demo notebook over here,
and I have just referred to
that demo notebook in
my notebook activity in Data Factory.
And at this point as you can see
that my check box is clean.
I've provided all the required settings
for this notebook activity.
So, I can now start to debug this.
I can look at this and see that
whether this notebook activities is correctly configured.
I can essentially I'm doing a test run to
validate my notebook operationalization.
So at this point the notebook activities in progress
and if I were to go here and look at the clusters,
you will see that a job cluster is
being created in Databricks right at that point.
So it takes around about four to five minutes for
this cluster to come up then your activity would execute.
And how long this cluster would stay
alive depends on the configuration
that you have provided.
You can look at your logs,
the Spark UI and all the logs in this interface as well.
So what I have also done for this demo is
that I've also created a pipeline
where Databricks is used as a step.
So you can see in this particular pipeline
that I'm copying data from S3 to blob.
Then I'm processing data using
the notebook that I have defined in Azure Databricks.
And once that processing off my customer records is done.
I'm moving the data to SQL data warehouse.
And in case of my copy fails I'm sending a failure email.
So you can see how this can be
operationalized end-to-end
using our Data Factory Pipeline.
>> How did people do this before?
>> Yeah. So, great question.
So, essentially before this people were to essentially
come to their notebook and they
would have to write code for all of this.
They can schedule it for
a particular notebook to run at a cadence.
But essentially if they wanted to ingest data from
65-plus data sources that we support today,
then they will have to
write custom code in order to do that.
So with Data Factory comes
the ability for you to ingest data at
scale in a severless fashion
from 60-plus data sources that we support.
And then essentially you can operationalize
your notebook as a step in your Data Factory Pipeline.
>> Right.
>> So you can build this complex workflow.
Not only you can do that but you
can also parameterize your workflow.
So, in this case I had
a simple demo notebook where
I was just doing some context.
But I have another notebook where essentially I am
passing a parameter for my pipeline to this notebook.
So, you can imagine that if I'm copying data every day,
and I just want that particular data to be used as
the data for processing
in the notebook in my Azure Databricks.
Then I can essentially pass that as a parameter,
the location of the data as
a parameter to my notebook for it
grab and run for that particular day, hour, or month.
>> It feels like this takes the work that
the data scientist is already doing in their notebooks,
and like you said it operationalizing.
It puts them in the middle of
the flow more than they have ever been before.
>> Yes absolutely.
So this makes their job super simple
because if you think about any data scientist,
and the first step that they do is that they have to
grab data from a bunch of sources.
So Data Factory makes that super simple for them.
And after that if they need to
essentially prep their data, shape their data,
they can essentially build their logic as
Databricks notebooks and operationalize
it as a step in their Data Factory Pipeline.
>> And you said that they can share
it with their co-workers?
>> Yes.
>> They can monitor all of this and make sure that
it's succeeding or failing?
>> Yes. So as you can see over here
that once this Data Factory Pipeline runs,
you can essentially go do
your monitor tab and you can see that you can see
the information about your Data Factory demo pipeline.
And I can see over here that
this notebook was successfully run.
And I can see all the inputs that were
passed through this pipeline all the outputs.
When it run? How much duration it took?
And if there was a failure
the logs will be presented to you as well.
>> And this is available now.
People who've watched this can do this now
as they're watching Azure Friday they can hit
"Pause" and set up a Databrick immediately?
>> Yes. So we actually published a blog,
last week as well on how you can do ingest,
prepare, and transform using Databricks and Data Factory?
And we also have documentation
in our Azure documentation for Data Factory.
In one of the tutorial we talk about how you can
run a Databricks notebook
with the notebook activity in Data Factory?
This is also linked to
our overview page on our user experience.
So we have featured tutorials and you
have your tutorial for how you can
transform through Databricks and
this links you directly to that Azure documentation.
>> Very cool. This sounds like
a great new series of
tools for the data scientist toolbelt.
>> Yes absolutely.
>> All right. I am learning
all about how to ingest and clean
my data and operationalize
my notebooks here on Azure Friday.