>> Learn what a data supply chain is,
and how you can track all of it and more using Azure Purview,
this week on Data Exposed MVP edition.
[MUSIC]
>> Hi, I'm Anna Hoffman,
and welcome to this episode of Data Exposed MVP edition.
Today I have a special MVP joining us today, Wolfgang.
Wolfgang, before we get into the episode,
can you tell us a little bit about what you do?
>> Yeah. Hi, and welcome from Austria.
My name is Wolfgang, and I'm working as a data consultant,
and what I do on my day to day job is
like helping our customers to get some insights out of their data.
In my spare time I'm the community guy organizing user group,
a local one, I'm a member of
the Azure Data Community Advisory Council.
Spending time at conferences,
what you'll in person.
>> Awesome. Cool. Well, it's great to have you on the show,
I know you're a really busy guy,
and have been traveling around a bunch and doing a lot of things,
so we're excited to have you on.
Today we're going to be talking about
Azure Purview which we haven't talked to a ton about on the show,
and specifically I think today we're going
to be talking about lineage.
I think to kick us off maybe a good starter question is,
why is it important to track data lineage in your data landscape?
>> Data lineage, it's very important because if you know
your data systems but you don't know
what actually happens in between,
it's like you're missing a really important point,
and that is where data lineage kicks in.
Like all those relationships between systems,
those data transfer processes,
and so on so it's really
important to know more than just the systems.
>> You use this term supply chain,
and does that refer to data lineage like the supply chain?
>> Yeah. It's like you can
compare it to the supply chain of products.
The data supply chain I
often compare it to where
does my data come from and where does it go to?
It's really a thing where
the interconnection between system kicks in.
>> Got it. Very cool.
Let's get right into it.
You want to take us through what is data lineage in Purview?
>> Sure. I can show you some slides.
Data lineage in Microsoft Purview,
and maybe you know what Microsoft Purview is,
just a short recap,
it's a data governance suite.
It changed name and it got integrated into a suite of tools,
and today I would like to talk about one piece,
it's a data catalog.
The data catalog,
there are like four or five things
the data catalog should do and Microsoft Purview does.
It's to map your data sources and to help you to
discover trusted data by
classifying data like putting some classification.
It's an email address in there and so on,
or providing business context with a business glossary.
What it also does,
it helps you to supply or to
understand your data supply chain as we already talked about it,
and the data supply chain,
I get that question very often, what is it?
To sum it up it's like data lineage or the data supply chain
it talks about where does
my data come from and where does it go to?
Some of those examples are like,
what are the dependencies between data systems?
Like you've got an ERP system,
a CRM system,
what happens in between?
What are the data transformation processes?
Like for a data warehouse it's quite
obvious that you have some data transformation,
but between those systems it's like there
are some relationships but nobody really knows,
and in our case like data warehousing,
those data sources for reports.
Where does that where you come from?
What happens on the way from the source system into the report?
For Microsoft Purview,
there are some concepts I have
to mention before starting to delineate itself.
It's like you define a data source
like a connection to a database,
your ERP system, your CRM system,
and there is a process which is called
a scan which does the technical inventory,
and what you get out of it are data assets like a table,
a schema in a database,
a file and a data lake and Azure Data Factory copy pipeline,
something like this, so those are data assets.
There are relationships between those assets,
and that is the data lineage.
There are different ways how data lineage comes
into the Purview data catalog.
The first is like the Azure Data Factory copy activity.
If you copy data in that example
from a data lake into a database,
it's like getting the lineage,
those relationship between the file and
the database table in your Purview data catalog.
There is a thing which can be distinguished.
There is asset level lineage like between the file and the table,
and Purview also helps you with the color level lineage.
As you can see in the screenshot there is some renaming.
Like the CRM number it's getting into the customer number,
so everything is tracked on the way of your data.
Another way how lineage can get into Purview
it's using Spark and Databricks connectors.
The Databricks connector it's quite new,
it's like brand new and available to get
all those data transformation on
the way into Purview and your data.
Power BI. I'm working with Power BI since many years now,
and there is a relationship between a Power BI dataflow,
a data set, the report,
and the dashboard, and also that lineage is
imported into the Purview environment.
If there is something missing like Purview
doesn't understand all those data systems out in the wild,
there is an API available where you can put in like
your custom connections using that API.
Now, well,
it's getting into the the really hot stuff,
my favorite feature,
it's the dynamic SQL lineage.
That one is really nice because when you have
a database and Purview does a scan,
it does the static scan like the schema.
You've got the database table, all those columns,
data types and so on,
plus it fetches the runtime information.
What are the store procedures that are executed?
What are the dependencies?
Are there any select into some merges and so on?
Are there any relationships between objects in there?
The operations are recorded in the database.
I just put it here like recorded,
and Purview does two types of scans.
One for the metadata,
the table definitions,
and the other one for the lineage extraction,
all those operations that were performed.
Every six hours,
Purview queries all those operations that
happens and puts them into the data catalog.
When I opened the Purview studio,
that is the detail page of a start procedure.
If you move a little bit further down to the properties, well,
you've got a section where you have
some input relationships like data assets that are getting into or
are used by the start procedure and that are output
like you have some query and put it into a result.
Now on the other side you've got
runtime information like how many executions?
How long did it take?
How many of those CPU milliseconds were used?
Everything is put together into
that lineage trough, that is the output,
and it can get even a little bit more distributed like you
have all those dependencies between those systems.
That is like if you don't know
your dependencies between your systems,
well, you have to know that.
Data assets are the start but what really
matters it's the lineage between those data assets.
>> Awesome. Wow. I have learned a lot in this episode.
At the beginning of this episode I was a little bit hesitant.
I was wondering maybe folks have their own self
documented way of keeping track of these data dependencies,
I think it's pretty clear now why
using a tool would be much better but I love your take
on why you think using a tool like
Azure Purview for this is better than self documenting.
>> If you start with the self documented data catalog,
it's old in the second you publish it.
Because it's not at the latest state of your database.
With Purview, you just start a new scan and you
get the information that is in the database.
>> No, that makes sense. Takes a lot of the room for
error and keeping it up to date automatically in a way.
This has been a really interesting episode,
I personally learned a lot.
I think as we wrap I love your advice,
your insights, like any tips or tricks
for folks who are just getting started with this?
>> What I would like to to tell people,
just try Purview and especially try the dynamic SQL,
so that one it's really amazing.
>> Cool. For our viewers,
go check out your Purview,
go check out the new Dynamic SQL capabilities.
We've put some links in our description for you to go learn more.
Wolfgang thanks so much for coming on the show.
To our viewers if you like this episode,
go ahead to give it a like, leave us a comment,
and let us know what you think of these capabilities,
and we hope to see you next time on Data Exposed.
[MUSIC]