>> Learn what's new and some practical tips for working with
Azure Data Factory and
other Azure services this week on Data Exposed.
[MUSIC]
>> Hi, I'm Anna Hoffman and
welcome to this episode of Data Exposed.
Today I'm joined by Wee Hyong,
a Principal Group Program Manager on the Azure Data Factory team.
Wee Hyong, thanks so much for joining us today.
>> Thank you, Anna. Thank you for having us on the show.
>> Yeah, of course,
it's always a pleasure to have the Azure Data Factory on,
because we partner so closely with the Azure SQL team.
Today we want to kick the awesome things and learn
more about some best practices and
tips that you might have for us,
but since we don't have you on the show that often,
I'd love to start by understanding
what's new in Azure Data Factory.
>> Oh yeah, absolutely.
We're super excited to share with you some of
the product innovations around data integration on Azure.
What you see in this slide is really from
guidance that we provide all the way to new regions that
we add and new connectors that add
the Azure improvements to data flow.
Now this is really a packed list of
all the innovations that happened just in the last month alone,
and we're excited to share this with the community.
>> Awesome. That's pretty cool and we'll be sure
to put a link in our description so you-all can
check out that blog and stay up to date
on what's going on in the Azure Data Factory world.
Now, apart from doing all of these updates, Wee Hyong,
your team also meets with a ton of customers,
and so through those customers, of course,
you're learning a lot about some of
the maybe mistakes customers might be
making or just some general tips or
best practices that you've learned from working with them.
Do you have any tips that you could share with our viewers?
>> Oh, yeah, sure. In fact, not just tips,
Anna we have distilled all these tips into
five practical tips that
we hope that will be useful to the community.
These are the five practical tips.
I think the goal here is to make sure that
customers have the best data integration experience on Azure.
We start off with the first tip which
is really about how you could
leverage a lot of the skills and a lot of the learnings,
whether is around performance tuning,
or just connecting things together
in Azure Data Factory pipelines or data flows,
and being able to use this immediately in
Synapse and be productive from day 1.
Today, the data integration experience on Azure,
you could either use Azure Data Factory or if you are looking and
using Cloud scale analytics
and you know whether is of Synapse spot,
Synapse SQL pool, and a whole bunch
of value that Synapse brings to the table.
You cannot move your Data Factory pipelines
and data flows and use them in Synapse.
I think that's one, but two is you get the same experience,
the same set of capabilities,
which hopefully will help you be able to
truly achieve less data integration.
Just as an introduction today,
if you get into Synapse,
you will be able to leverage Synapse pipelines,
and in Synapse pipeline you'll see that you
get all your familiar activities.
On top of that, you also
get other activities as relevant to Synapse.
For example, things like Synapse Notebooks,
the ability to then execute code on Synapse spot,
which is just amazing.
Other things that you'll get out
in the Synapse workspace is Synapse Data Flow.
When you use Synapse Data Flow,
you literally have
the same cooked free data integration experience
that you might already be used to,
and from the moment you start creating
a data flow in Synapse and be able to get from your source,
to your transformation, to your sink,
everything is the same.
But more importantly in this runs of Synapse spot,
where they allows you to run this data flows at amazing skill.
>> Awesome. Oh, I was just going to say,
it seems like a pretty good tip,
especially because you're just taking
pretty simple things like learning
how to use these pipelines and data flows,
and now you're going to be able to be
much more productive, so I love that tip.
I think the second tip has something to do if I recall correctly,
something to do with integration of
your data but also involving governance.
Can you tell us a little bit about that one?
>> Yeah, sounds good Anna. One of the things that comes to
mind whenever you do data integration in general is,
it's not just about data integration,
is about the data assets that you have and how we can leverage
some of this data assets in your data integration.
Whether is a data pipeline or data flow.
We think that bringing data integration,
data governance coming together is
just simply unmatched capabilities
that we hope that will be useful
to a lot of the customers that we work with but more importantly
allowing you to combine the power of Azure Purview,
which is a data governance service to get at Azure Data Factory.
Then be able to leverage all this assets that has been
scanned and is now available in Purview
immediately in your data factories.
I think that's one piece to it when you bring them together.
The second case is very exciting,
especially the fact that in many organizations today,
you often use and you
build data pipelines to move data from your source,
you transform your data.
You land them in the data warehouse that
Synapse SQL pool, for example.
But many a times,
why do we do it?
Because you want to reduce the time
we get from raw data to reports,
and you know that all these partners run 24/7 all the time,
but there will be a time when things break.
You want to know, "Hey, why is this value in
the report not showing up or is computed wrongly,
is there a problem with the whole lineage
all the way to the raw data?"
By enabling Azure Purview and Data Factory to come together,
now you get all your ETL lineage.
Now for every single pipeline run that you have,
this lineage gets extracted,
and then get loaded into Azure Purview if you will.
Now we can do very powerful analysis on ETL lineage.
On what are the data that got transformed,
that lender this in Synapse SQL pool for example,
and then being able to show this in a report,
I think that's one value that you get out of it.
The other value that you get off it is really the fact that,
many of times as a DBA or as a data developer,
you might be wanting to make changes to the schema of the table.
When we make changes to the schema of the table,
what happen if you drop a column
or what happen if you add a column?
What is the impact of the downstream?
You want that visibility into
their whole lineage as well but from the other angle.
Because then that will give you the confidence
that whatever change that you're making,
it propagates through all your ETL pipeline, if you will.
By bringing this together,
it also enable you to do that.
Right now, what you're seeing and let me get to this page
is Azure Data Factory with
the familiar design environment that everyone is used to,
and in this Azure Data Factory design view
or authoring view if you will,
which create a pipeline,
and in this case we call this pipeline SQL Server data ingestion,
and all the things that you see here will be familiar to you,
we're doing a lookup to get the list of tables,
and for each of the table, what are we doing?
If we click on the activities
for each of the table we're copying data.
Where are we copying the data from?
Is we are copying the data from a SQL Server table that
has been parameterized to a sink.
In this case, we are landing it somewhere as a parquet dataset.
Of course, in a typical flow,
after a bit of a pipeline like this,
I would of course then run the pipeline on a day-to-day basis.
But what happens if you need to understand
this lineage and be able
to capture it as part of the assets
in which you can then govern using Purview?
The first thing you'll do in order to integrate
Data Factory and Purview together,
is you're going to click on this "Manage" tab.
To show you this Manage tab,
I'll use another factory that has not been
connected to Purview yet just to show you the flow.
In this factory, I will click on the "Manage" tab.
When I click on "Manage", what happens?
This will be the familiar management experience
that you might have already got used to in Azure Data Factory.
You will see this new entry here under
"Connections" called Azure Purview,
and right now it's in preview.
All you need to be able to connect data integration and
data governance together is just clicking
on the "Connect to a Purview account",
and when I do that, what will happen is
it will do a data registration,
it will enable you to do data discovery,
and later I'll show you the exploration piece of it.
If I click on "Connect to a Purview account",
a person's going to asks you, "Hey,
from which Purview account do you want to connect to?"
Of course, right now we are loading
the list of the Purview account.
Now, once this gets loaded,
and if you click "Apply",
and this is taking longer that usual,
you will have already connected Data Factory to a Purview account.
Of course you see several accounts that I can connect to.
In this case, I might want to connect to maybe purviewtest1,
and the moment I click "Apply",
what is happening underneath the hood is
we are now brought Data Factory and Purview together.
Super exciting. Now let's see what you can
do with it once you have connected it.
Now once you have connected this,
and you will see that it has been connected successfully.
You will see the Purview account
that this Data Factory is connected to.
This case, of course, we have already pre-prepared
this using this factory called Contosodataingestion,
as same here when I go to "Manage",
you'll see that this has been connected to
the Purview account called ContosoPurviewAccount.
What will happen now is, if I click on this,
this will bring you to the Purview studio
that will then show you-all
this lineage that we are talking about earlier.
>> Now once this loads,
you're going to get into the Purview Studio.
In this Purview Studio of course you'll
get things like your data catalog,
a glossary that helps you understand things within the catalog.
You'll get some insights and again of course you
have a familiar management experience.
Right now what I click is I click ''Data
Catalog,'' and you'll see that early on in
that factory that was connected to Purview,
it was a factory called CopyData.
If you looking at it now,
within the copy activity at a glance,
you immediately know all the tables
and all the SQL Server that you're connecting it to.
Of course you could click on one of
these to be able to understand what is inside.
You could search, you could understand for each one of
these tables that you have whether it's a sales reason,
salesperson, what are the different kind of entity.
At a glance, immediately you know what
data sources you're connecting to,
but also get a sense of what is
the schema of the data that you're bringing in
from copy and then
storing them in the [inaudible] file and where you're storing it.
Super powerful. Now of course you could continue and say, hey,
let me open this in Azure Data Factory
which would then allow me to start using this.
It provides immediate links instead of you having to
find a Data Factory in which this Linux is a part of.
Now the other thing that I wanted to show
you before we switch back to the slide is
you could now also when you start authoring
a pipeline if you will, you can start searching.
Because this Data Factory is not connected to Purview,
you can search for any asset.
You could say, hey, find me all assets that's copying data.
This immediately connects to Purview,
gets all this information and it tells
you some suggestions based on what we're seeing in the data.
You get what kinds of things we are coping.
Whether it is a JSON file test called PDF,
or whether it's a demo copy from Blob.
If I click on one of this,
immediately you see the exit and you
can immediately leverage it in Azure Data Factory.
>> Awesome. Thanks Wee Hyong. That was a really cool demo.
I hadn't seen a ton of Purview before,
so it's definitely cool to see it,
but also see it integrated with Azure Data Factory.
If we come back to our tips,
I think we have three more tips.
The third tip is around performance.
>> Yeah. That's right. One of the things we get asked a
lot is that I'm building all this pipeline,
I'm building all this data flows which
is trying to bring data in transforming the data.
How do I know I got it right,
or why is it that my data flows is taking such a long time to run?
What should I be even thinking about?
The other set of questions that we get asked a lot and I love
this customer conversations as well
as the conversations of the community,
is if I'm doing a copy,
and I have a self-hosted integration runtime which allows me to
have access a line of sight to on-premises data sources,
how should I think about sizing
this self-hosted integration runtime
so that it performs optimally,
but at the same time do not jam up
the pipe that I have from on-premises to the Cloud.
With all these kinds of discussion that we have with
customers for the last couple of years,
one of the things that involves
the Cloud solution architects and the
few have been asking us a lot,
is why do you distill this down,
not just in the five tips,
but a performance guide that will help
guide us and leverage and stand on
the shoulders of all this knowledge from
the whole Data Factory community coming together.
This is exactly what we did.
We continuously improve the copy activity performance and
scalability guide to make sure that you have
the best data ingestion experience on Azure.
More importantly, just as a fun fact,
today Data Factory move petabytes of data every single month.
>> Wow.
>> This is not just external,
but also internal, what looks as well.
If you bring this together,
a lot of experience comes from
the community for a lot of the customers that we work with.
Now all this has been distilled into
the copy activity performance and scalability guide.
>> Awesome. We'll put a link to that in
our show notes so people can definitely go check those out.
I think the last two tips are around learning.
Can you take us through those?
>> That's right. Now the last two tips is about if I'm new to
Azure Data Factory or in fact we have
new team members on-boarding to the team,
how do we distill
all this concepts for Azure Data Factory into one page?
That was the challenge that we set to the team just about
a month ago and working with
the Cloud developer advocates
as well as the developer relation team.
This was exactly what we did.
Just a few months ago,
we came together and say, "Well,
let's bring all these docs that we have on
Azure Data Factory from scenarios to concepts,
etc, and let's put all of this
together so that it's in one single page,
you could print it as a big poster, and in fact,
we provide a high resolution version of this."
Both in your office,
at your home office, and at a glance,
you get all the value proposition of Data Factory,
but more importantly, concepts that will help you get started.
We're super excited to work with [inaudible] one
of the [inaudible] developer advocate that's extremely
talented in drawing this kind of
concept and putting all this together as a visual guide.
Together with [inaudible] ,
we have launched this and more importantly we did not just
launched this and make the high resolution version available.
Today if you go to the Azure Data Factory documentation page,
when you go to introduction to
Azure Data Factory that you see on the link there,
immediately you get to download
the high resolution so that you can
find immediately if you're
looking and getting started with Azure Data Factory.
>> Awesome. That's really cool.
>> Now I will cover the last tip which is now that you have
learned all these concepts about data integration on Azure,
how do you then set forward on a learning journey?
How can we help you build up your experience
from getting to know data integration in Azure,
whether it's in Azure Data Factory
or whether it's in Azure Synapse Analytics.
Also set you on a path and a learning journey
that if you choose to take a certification,
it will also increase a knowledge that will help
you acquire that certification.
The last thing I wanted to share on
the show is really your journey to success at
Azure Data Factory and even now today on Azure Learn,
we have mod at 8-10 Azure learning
modules on how you do data integration on Azure,
whether it's of Azure Data Factory or
whether it's of Azure Synapse Analytics.
What the team has done together with
the Azure Learn team is to put this together into four weeks.
Like if you have only four weeks of your time and you need to
get started with all nuts and bolts of Azure Data Factory,
we start off with Week 1: learning the basics,
which includes the visual guide that's going to help you navigate.
It's like a cheat sheet if you will.
Then by Week 2, once you have build up all this confidence,
you are now able to be able to achieve
petabyte-scale data ingestion by
learning around different methods.
What kind of connectors are supported?
How should you be tuning the connectors.
Week 2 Azure Learn module.
By the time you get to Week 3,
we say that, well you have ingested all this data,
this data could be on ADL Gen 2,
it could be on different stores on Azure.
How do you do what
we call quote free data transformation at scale?
By Week 3 we'll teach you how to use the data flows,
and how we could then start building
really complex data flows
that allow you to achieve your business needs.
By Week 4, now that you have build up all this experience,
we are showing you how do you orchestrate.
How do you leverage the rich set of
orchestration capabilities that Azure Data Factory
provides out of the box,
but lastly, if you have got interested in a certification,
the DP 203 exam would then be inside.
Of course there is other kinds of
knowledge that you need in order to
build up in order to achieve that certification,
but we hope that with
this learning journey that were put together,
it will help everyone to be successful doing
data integration on Azure and that gets us excited every day.
If there is anything that the community can share with us,
we'd love to hear any feedback.
>> Awesome. Thanks so much Wee Hyong.
This has been a really informative video.
I liked the five tips.
I also like seeing the cool demo with
Azure Purview and I think our viewers will enjoy that as well.
I want to thank you for coming on the show and to
our viewers I want to thank you for joining us today.
We've put a lot of links in the description
for you to dive deeper into this topic.
If you liked this video,
leave us a comment, like this video,
subscribe to our channel,
and we hope to see you next time on Data Exposed.
>> Thank you Anna.
[MUSIC]