- Coming up, we're joined by Engineering Lead, Mike Flasko
for a deep dive on the new Azure Purview
that automatically discovers and maps the data
sitting across your data sources,
on-premises databases, and SaaS data sources
to help you catalog and understand your data
and classify it all in one unified environment.
So Mike, welcome to Microsoft Mechanics
and congrats on the preview launch of Azure Purview.
- Thank you, Jeremy. It's great to be here.
- So Azure Purview is quite
a significant breakthrough service,
especially if you think about it
from a data management perspective.
It's a struggle today,
really to get a handle on the volumes of data
that sit across your organization,
which can also sit in multiple clouds and on-premises.
So it can take a lot of different forms
and it's really hard to track all of it.
- You know, that's right,
that nobody really gives you a holistic map of the data
across your data landscape today.
Thinking about all the different data sources
and data types that often exist in an organization,
from columnar data through to text files,
BI reports and kind of everything in between.
There's no easy way to understand
or connect all these different data types
without a lot of time consuming or manual work.
And it's even harder to understand
what operations are running against all that data.
- Okay, so how do things change then with Azure Purview?
- We're really solving for data discovery
and understanding and providing the foundations
for effective data governance, because ultimately,
the better you understand the data you have,
the more effectively you can use it
across your organization.
Purview gives you a unified platform
that automatically discovers
and classifies your data without having to move it.
All the metadata discovered about your data is then indexed
and brought together as a unified data map
of your data estate.
Purview also provides rich user experiences,
enabling data, producers, consumers,
and stewards to easily collaborate.
For example, business users and domain experts can interact
with Purview's business glossary to empower all users
to easily understand the business context associated
with the data in their organization.
Finally, a key to understanding you're using the right data
is understanding where your data came from.
With Purview, you can easily track and visualize the lineage
of your data across the data estate,
so you can easily see where data is moved
and how it's been transformed.
- Right, and to be clear,
if you think about all the underlying tech
that makes all this possible,
there's a lot of collective experience at Microsoft
that makes Purview a reality.
- Absolutely, everything from our decades of work
with Bing through to Azure Search for custom search
and indexing was really helpful.
As the example you can see here
with the manufacturing company Howden,
and the data classification technology
that we've developed over the years
for Microsoft Information Protection,
all of this has really provided us a mature foundation
that we built from.
And then, of course, we're also our own customer,
and so the approach we're taking today to scanning
and mapping is highly inspired
by what we do at Microsoft every day at exabyte scale
to understand and govern our data estate.
And lastly, we've also adopted some of the great innovations
in the open source community, such as Apache Atlas.
- Great work.
So since this is Mechanics though,
why don't we make this real
by seeing everything in action?
- Yeah, that sounds great.
Let's take a look at Purview in action.
So here we are looking at the home screen of Azure Purview.
I can search for data using the Purview data catalog.
Under the search bar
I can see links to the knowledge center
and other common functions.
Up on the left-hand bar is where I can easily navigate
to additional experiences
to register more data, set up scanning,
get data insights and manage settings.
- Okay, so the most common thing I'm guessing
you're going to do from here is actually search.
What does the search process then look like?
- That's right. This really is the starting point.
Let me show you what anyone would do
to search and find data.
And then after that,
we can go and look at what an administrator
and a curator will do.
I'll start by searching for sales.
You can see, I get suggestions
and recommendations right here as I'm typing.
When I execute the search,
it's finding matching business glossary terms, data assets,
and it's returning the results intelligently
based on relevance, using all the signals,
derived from scanning and classifying your data
as well as business context from the business glossary.
If I go ahead and select one of the search results,
let's pick the sales order header.
Here, it shows me the operational metadata,
and I can see at a glance,
this data contains sensitive information,
such as credit card numbers and social security numbers.
But we'll talk more on that in just a minute.
On the right side,
we can also see that there's a hierarchy
and where this dataset belongs inside of a table
and schema and database and server.
And from here if I wanted to,
I could even go ahead and open up the data,
right in Power BI Desktop to visualize.
- And the nice thing here is that Purview,
like Azure Synapse, actually automatically creates
that pivots file for Power BI users,
so you can visualize the data automatically
and straight from there.
But what happens when we go
and drill into the tables information?
What information is there?
- Yeah, that's right.
There's a lot of rich information available here.
So if I tour through the other tabs first with the Schema,
I can see the column names from the table, their types,
any classifications that have been applied,
in this case, social security number, bank account,
and we've even got a custom classification for customer ID.
And one thing to note here is that even though in my case,
the column might make it obvious
what is the data contained in these fields,
the system can still scan the content of the columns
to verify the presence of sensitive information.
Next, I'd like to show you one of my favorite areas:
The Lineage view.
You often want to know where a piece of data came from
as well as what data is derived from it.
This helps you assess at glance
if the data comes from an authoritative source.
Here, we can see all the Power BI reports
that are ultimately based on this data from this table,
as well as all the transformations
that the data went through along the way.
- So this is great, it's a lot more than just simple
key pairs that you're actually mapping out here.
This is actually a pipeline view of the table
through to the ETL steps, all the way down to the reports.
Now these views would normally
probably be drawn out manually
with tools like Vizio, for example,
but you're doing that automatically within Purview.
- That's right.
Getting an end-to-end view like this
can be really empowering when working with data
because it's spanning data sources, operations on the data,
as well as in this case, how that data is flowing into BI.
Now there's a few more things I'd like to show you.
First the Contacts tab.
Here we can identify experts and owners of the data asset.
And then finally, here in the Related tab,
we can quickly browse all the other tables
related to this one.
By the way, so far in this example
we've been looking at structured data,
but these same experiences work well for your unstructured
and semi-structured data as well.
- Okay, so there was probably a lot going on
under the covers there to bring these capabilities to life.
Can you explain how everything works?
- So it all starts with the data.
Organizations have many data assets, such as files, tables,
BI reports, ML models, and many other things.
And these things are often residing
across on-prem, cloud and SaaS environments.
You can connect your data systems to Azure Purview
using an ever-growing set of included connectors.
You can then get Purview to scan those data sources,
to extract a wide range of metadata,
things like technical metadata, lineage, classification
and operational metadata,
and doing it all without moving the data itself.
And the cool thing is that the scans operates serverlessly
so you only pay for what you use.
Next, all the metadata found during scanning
is then published to the Azure Purview data map.
The map is an intelligent graph describing all the data
across your data estate.
Additionally, because the data map is exposed
as Apache Atlas open APIs,
you can programmatically push any metadata
in Lineage from any data system,
and this is a great way to expand your data map.
So once your data map is in place,
everyone in your organization can go
to the Azure Purview data catalog experience
and easily search and browse for data.
In addition, your chief data officers
can get end-to-end insights across the data estate
using the insights experiences, which are also provided.
- Good stuff, but I want to switch gears here
and talk about setup.
So what does it take then to connect everything up
and get things like the classifications
that we saw for sensitive information
and everything kind of mapped up and working?
- We wanted to make this part of the process
as easy as possible.
So how about I just show you how Purview enables this
in just a few clicks?
As an administrator,
you click into the Sources area of Azure Purview.
Here, you can see
all the data sources Purview can automatically scan,
a Blob account, Power BI,
Hive Metastore and a few others.
Let me show you how to connect a new data source
to Purview though.
I'll click here on the Register button.
Here, you can see all the sources
that are supported for automatic scanning.
There's a range of Azure sources.
There's even both on-prem and multi-cloud sources
such as Amazon S3, SQL Server, SAP, and Teradata.
And as I explained earlier, under the covers
we've done all the work to deeply integrate
with these data sources, so that as we extract the metadata
it can securely flow to your Azure Purview instance.
And for Azure sources, we made it really easy
to register multiple sources in one step.
As you can see here with the Azure Multiple Sources option,
it'll find and register all the data sources automatically
from a management group or a subscription.
Now it's likely you have many data sources to connect.
The nice thing is that Purview allows you
to organize them into collections
and then visualize that as a tree view.
This also allows you to configure data scanning
and classification settings
at the root level of that collection,
and those settings will then be automatically applied
to everything underneath.
- So once you've registered then all of your data sources,
the scans are going to automatically extract
all the metadata that was required
to power the search that we saw earlier,
and also the lineage experiences that you showed before?
- Right, and of course,
the data's going to continually change.
So the scans need to run periodically
to ensure you have accurate
and up-to-date understanding of your data.
- Okay, so what are the options then
if you want to start scanning your data?
- So there's a number of options available
when you're configuring scanning.
Here in the Management Center,
we can see all the classifications Purview knows
how to automatically detect.
There's currently support
for over 100 sensitive information types.
And as we saw earlier,
these range from things like credit cards, account numbers,
through to a wide range of types,
such as government IDs, location data and more.
- Right, and these really look pretty similar
to the sensitive information types that we see
in things like Microsoft 365 Information Protection
or Data Loss Prevention.
- That's right, it's the same classification taxonomy,
but now available across your other data sources as well.
You can even define your own custom classification rules.
I'll click into Transaction ID.
Here you can define your own data pattern,
like an item inventory number or a transaction ID
or customer ID along with thresholds
that you can set to reduce false positives.
And once you've defined what you want to look for,
you can configure scan rules.
You can choose file types you want to scan,
you can even define your own file types if you've got them.
Next, we'll set up the classification rules
that we want to run.
These are the same hundred plus classifications
we saw earlier, just grouped into categories.
And finally, you can determine whether you want the scan
to run on a reoccurring schedule or just one time.
- Okay, so it's going to run the scan
and when it's finished,
then you'll be able to actually search that data catalog
like we saw with your sales table example?
- That's right, but in addition to searching
for a particular dataset in the catalog,
Purview also provides a bird's eye view
of your data landscape, and this is intended
to help chief data officers quickly understand
their data estate at large and gain key insights
such as where sensitive data resides.
Now, here in the Insights area of Purview
is where we really get that bird's eye view
of the data estate.
Here, it has the insights.
You can quickly see where all your data resides
across a range of data sources.
Next, in Scan Insights,
we can see the number of successful, failed,
and canceled scans over time.
Then over in the Glossary Insights section,
we can quickly understand changes made
to the glossary over time,
and we can assess how much coverage your glossary has
over your data map.
And this is a great view for compliance teams.
Here, thanks to all the classification work done earlier,
you can quickly determine
where different kinds of sensitive data exists
across the data in your data map.
- Good stuff, and having this level of knowledge
about the data is something
that hasn't been easily attainable.
But beyond what's available now in Purview,
is it possible then
to enable things like proactive alerting?
- Yes, we're fully integrated with Azure Monitor.
So you can set up alerts and additional views
to monitor the health of your service and your scans.
- Wow, this really changes the game,
I think, for data insights and management.
So what's next on the horizon then for you and your team?
- We just launched Purview and of course,
we're eager to get your feedback as you try it out.
But as I mentioned earlier,
understanding your data is one of the most important steps
in efficient data governance.
So watch this space as we're working hard
to support more data sources
and develop more capabilities in this area.
- Thanks Mike, really great intro,
a lot more coming as well.
So what do you recommend people do then
if they want to get started using Azure Purview?
- Please go ahead and sign up for the preview.
And you can do that at aka.ms/TryAzurePurview.
And to learn more about Purview,
check out all the online resources at aka.ms/AzurePurview.
- Of course, also keep checking back
to Microsoft Mechanics for the latest updates,
and subscribe if you haven't already yet.
Thanks for watching. We'll see you next time.
(upbeat music)