>> Hey friends. As organizations modernize their data estate,
they want to explore, prep, transform, train,
predict, and serve their data, to unlock insights,
to drive better outcomes,
more insightful decisions, and business transformations.
Gaurav Malhotra is back once again to
show us how to do that on Azure,
code free, and at scale.
Today on Azure Friday.
>> [MUSIC].
>> Hey friends, I'm Scott Hanselman and it's Azure Friday.
I'm here with Gaurav and we're going to talk more
about Azure Data Factory. Welcome back.
>> Yeah. Thank you Scott.
>> So you've got a pattern here for me,
to make things work at scale with Azure Data Factory dataflow.
>> Yes. So what we are doing is that,
right now more organizations are looking
to modernize their data warehouse, right?
So we see that customers are looking to
ingest data from multiple sources, prep the data,
train the data, serve the data,
so that ultimately they can make
intelligent decisions based on that data.
So what I'm going to do today is,
I'm going to use a retail scenario
of omnichannel customer engagement.
>> Okay.
>> To show you how organizations are
looking to combine data in store,
and in like from online sales,
and provide the best in class customer experience.
So the first step of doing that is ingesting data into the lake.
Okay. So here I am on Azure Data Factory,
and this is the modern web-based ID that you
already know that is used to build your ETL or ELT workflows.
So here I haven't ingest pipeline,
where you can see that I'm getting data from different sources.
So I have the location data coming in,
which includes the in-store beacon data for customers,
I have some customer information
from Dynamics, products information,
point of sale information from SQL,
clickstream data coming in from Amazon S3,
and I'm ingesting all of their data.
These are live data ingestion activities
in Data Factory and to Azure.
Now, if as a customer you want to use any other dataset,
you can do that in few simple clicks,
and here we support 80 plus connectors.
We have support for On-prem connectors,
like if you're looking for Echo,
If you're looking for SaaS connectors,
they're all over here.
So you can choose from any of these and ingest
data from these connectors into the lake.
>> Okay.
>> So now that you've ingested the data into the lake,
the next step for you is to prep and transform the data.
So I have another pipeline over here,
which I call as a transform pipeline,
and here I'm using a data flow activity.
So this is the newly added capability in Data Factory,
which allows you to do quote
free Transformation at Cloud-scale whilst park execution.
>> Code free.
>> Yes. Code free. So if I double-click this,
you will see that this is the dataflow canvas,
and here you can see that I'm doing a bunch of transformations.
If as a customer,
you want to choose any other transformation,
you just click the "Plus," then you can choose any of these.
If you want to do Join Conditional Split,
Lookup, Sort, Merge, Pivot, Window transformation.
>> Those canvas keeps getting better
and better. It's really amazing.
>> These are all codeless transformations that we are giving too.
So for this particular example,
you can see that I am doing a joiner here,
so I want to join my sales transactions with account information,
and here, I've just selected what type of join I want.
I've selected the column on the left and the right,
and you just have to do this.
You can do this in a few simple clicks.
>> Right.
>> Right. So now here as a customer,
via doing all the heavy lifting.
Because all of these transformations are
executed on the big data clusters via Spark.
So the job management and cluster management,
all is taken care by us.
You as a customer don't have to do anything,
because it's all a managed environment.
>> Amazing. Big data as a service.
>> Exactly, right. So if you think about it,
what we are trying to do here is that we're
trying to democratize big data tools.
We're trying to take those to
the masses so that they can do two things.
A, they don't have to focus on learning Spark,
Scala, Java, they can actually focus on the business logic.
>> All right.
>> The second thing is, this gives
them longer shelf-life for their solutions,
because tomorrow if a new Spark version comes,
we will automatically work
to enhance their goal to work with the new version.
They don't have to do anything.
So their solution is just keep on
working once they code data here,
once the buildings data flows and it's always running. All right.
So now that we have done the prep and transformed the data,
the next step is to load the data into the warehouse.
Here I'm going to be using Azure SQL Data Warehouse Gen2,
which as you guys are aware,
is the fastest and
the most performing petabyte
scale Cloud data warehouse available in the market.
I hope everyone read the independent analysis that Gighome.
did, which actually says that data warehouse is 14 times faster,
and 94 times cheaper than it's
other Cloud-scale providers that
provide the Cloud data warehouse story and the market.
>> Well.
>> All right. So here with Azure Data Factory Data Flow,
now we have this ability where you can pipe in
your final data directly into a data warehouse,
so we have a sync transformation over here.
So if you choose "Sync," and here let's say I'm going to
select "SQL_DW," and here now I have a DW available.
I can essentially say,
do upsurge, inserts, and support other patterns.
I can essentially create tables on the fly.
So if my table does not exist in my data warehouse,
it will automatically create that,
it can truncate the table, recreate the table,
as well as it will load the data
parallelly using PolyBase at scale,
into your data warehouse.
So all of that can be done from inside the dataflow now.
So now, what we did was be ingested the data into the lake,
we prepped and transform the data,
and we loaded the data into the data warehouse.
We are ready to now combine all of
this together in a data integration pipeline.
So I have this data integration pipeline over here.
So if I were to just full screen it,
you can see that over here,
what I have is all my data ingestion activities.
If they succeed, then I do a dataprep,
if that runs successfully,
I do the sales and [inaudible] again using a data flow.
If any of it fails,
I send a failure e-mail.
>> [inaudible].
>> So now if you are a traditional ETL guy,
who does not want to write code and wanted to work in
a code free manner that I showed you, you can use that.
But if you are a guy who likes writing code,
you can do this exact thing by writing code as well.
So here in this pipeline,
I'm doing the exact same thing,
but instead of using the visual data flow,
I have a notebook activity where I have written all of
the code to do those transformation in the notebook.
>> Nice.
>> So it's the exact same thing.
So totally as a choice, for the customer,
whether they want to go code free or
they want to write their Python code,
and have more control. So totally up to them.
>> That's nice. This is again,
we've talked about this before when you had an Azure Data,
on Azure Friday talking about Azure Data Factory.
Is being prescriptive but you're
not demanding that people do it a certain way.
So Azure Data guys and girls that want to do it a certain way,
they can bring the tools they want, the connectors that they want,
they can do it with code, they can do with Visual Designer.
It's just going to work and you don't impose upon them,
how they have to do those things.
>> Yes. So it's a choice as a customer that you
make whether you want to work with one versus the other.
>> Yeah.
>> Okay. So now you have done this,
you've build your pipeline,
you can use all the visual capabilities that we have
available for or doing iterative development and debugging.
So you could put breakpoints and
say that I want to only run after this.
If you want to run this on a cadence or an event,
you can put a trigger on top to say,
I want to run this every day,
every 15 minutes, or on an event when my file lands in the Blob.
Once you put the trigger,
now you can monitor these solutions end-to-end as well.
Using our rich monitoring interface,
so here, you're looking at the monitoring tab in your Data Factory.
We have tool. Every tool here is a pipeline run.
You can dive deeper into it,
we'll look in more detail.
So you can see all here that for copy,
you can look at details to see how much throughput you get,
how much time it took for your copy to happen,
how much time the data was in queue.
So all of that details is presented to you.
In addition to that, for
the visual code free capabilities that we added,
we have enhanced monitoring capabilities.
Where we can tell you for every step,
how much data came in,
how much data went out,
partitioning information, code doses,
all of that is available for you to look at.
>> So much more powerful than it was even just a few years ago.
>> Yes. One of the things that we are doing is we are
adding in build patterns to these code free capability.
What that means is that for example, for this one.
Like if I were to show you this particular data flow,
and here if you want to use
complex partitioning schemes like round robin and all of that.
So it's just a choice that you make.
You just do a selection over here, and again,
we're doing all those heavy lifting and you can
do this in few simple clicks.
>> Yeah.
>> Also for data,
we are enabling things like schema drift.
As you can see in data integration rule,
you could have more columns coming in in the source,
but you don't want their data
integration pipeline to get affected.
So all of that is supported out of the box,
with this newly added capability.
>> Fantastic.
>> Yeah.
>> So people just get this.
Once again we've talked about this before when we've had you on,
that they don't have to click anything or have to upgrade,
they just visit Azure Data Factory and it just keeps getting back.
>> Yes. This is available today in public preview.
They can essentially go use it, play with it,
give us feedback, and work with it, right.
So and also with this pattern, just for completeness,
once the data is under DW,
now you could have Power BI Dashboards hooked up to it.
So here I have
a Power BI dashboard where I'm looking at the sales trends,
and this is running on a refresh,
hooked up to that data warehouse which
where my pipeline runs every day,
it help layouts my star schema and the data warehouse,
and this Power BI dashboard is hooked up to it.
So where I can gain those Intelligent Insights,
so that the business decision-makers can take
a look at it and make decisions based on it.
>> Fantastic. Always impressive,
always appreciate you coming on the show.
I'm learning how to ingest, prep, transform,
and load data warehouses code free,
at scale, with Azure Data Factory Data Flow.
I'm learning all about it today, on Azure Friday.
[MUSIC]