learning
[Applause]
hello
this is amit carr with databricks
in this brief demonstration i'll show
you how databricks's unity catalog a
unified governance solution for the lake
house is used by data teams to centrally
manage govern search discover and
securely access data for analytics or
machine learning initialives
so let's get started as organizations
implement data lake houses managing
access to the data and objects becomes
increasingly important
with unity catalog administrators can
centrally manage and govern access from
a single location and administer access
policies to all databricks workspaces
and personas from the account console
the account administrator can view
configuration information for each
workspace including region credentials
storage and the meta store
now the meta store is what enables unity
catalog
this metastore is a top level container
that stores metadata of the data objects
such as tables views and the permissions
that govern their access in the lake
house
when an administrator configures a meta
store they will specify the storage
bucket where the meta store is metadata
and managed tables will be stored and
the role that will have access to that
metastore bucket
for this particular meta store we've
also enabled delta sharing allowing
users to securely share data outside of
their organization
now each metastore that is configured
is assigned to your databricks workspace
by the account administrator
the workspace url and the statuses are
shown on the right along with other
workspace deployment information
observe that identity federation is
enabled for this workspace
unity catalog uses a federated identity
model where the administrator can manage
all users service principles and groups
at the account level and assign those
account level identities workspace
access an administrator
can configure users and groups directly
in the databricks account console
or by automatically syncing them from an
identity provider
with unity catalog enabled
let's look at how users can securely
interact with data in the lake house
so i'll go ahead and open the databricks
workspace which serves as a repository
for
all lake house assets such as tables
files notebooks machine learning models
and dashboards that are used for data
engineering data science data
warehousing and machine learning
a catalog the first layer of unity
catalog's three-level namespace is used
to organize data assets and is created
by the administrator
now a catalog is a collection of
databases that contain tables and views
unity catalog security model is based on
standard ansi sql allowing
administrators to create and grant
permissions to catalogs databases tables
views and files using familiar sql
syntax
users who have been given access to
catalogs can browse and view data
objects using the data explorer without
spinning up compute clusters a data lake
house may contain thousands of data
objects such as tables or views
to quickly find data users can leverage
unity catalogs building such capability
in this case we're searching for
hypothetical patient visits data
and results are returned based on the
associated search term
users will only see results for data
that they have access to because unity
catalog automatically leverages the
governance model that's in place
by selecting the all visits table
additional details can be viewed
so from the top
with the additional layer of data
segregation
unity catalog's three-level namespace
helps organize data in the lake house
so this table all visits is in the hls
database which is part of the demo
catalog that's owned by the data team
the table schema shown below provides
details about the table structure such
as the column name
data type and any comments
you may add a comment if you have
permission and the knowledge about the
column
users can also preview sample data
this allows them to visually inspect the
contents of the table
the details tab contains all of the
metadata information about the all
visits table this includes the storage
location properties and when and by whom
it was created
a key pillar for united catalog is
centralized access controls
in this view access can be restricted or
granted to the table to the users or
groups authorizing them to read or
modify the all visits table
alternatively access controls can also
be managed by a familiar sql syntax
finally within this view
is lineage
catalog automatically captures runtime
data lineage allowing users to see how
data flows upstream or downstream in the
lake house
in this current view users can see
tables used to derive all visits or
tables that consume data from all visits
notebooks
workflows and any dashboards associated
with the all visits table
alternatively
the lineage graph displays a
data flow graphically
users can
see that all visits is derived from the
encounters table and an ops visit table
downstream usage and all other assets in
its path
so
not only do users have visibility into
the data flow lineage
the unity catalog also captures column
level lineage
so for example patient id is derived
from two tables the encounters table and
the ops visit table
you can see where the column is used for
use cases such as impact analysis
debugging or for compliance
lastly from the data explorer users can
share data
depending on the use case users can
securely share live data with partners
customers or even internal line of
business partners from the data lake
house to any platform without
replicating or moving data using delta
sharing
once the user has understood the data
context the table's relationships
they can create queries or dashboards to
visualize data for analytics
finally with all the activity taking
place within the data lake house
unity catalog provides access to
centralized fine-grained audit logs of
user activities
enabling the organization to monitor
detailed data usage for data compliance
and security requirements unity catalog
automatically captures user level audit
logs that track data access where
administrators can obtain detailed audit
log data on how data is accessed and by
whom
in this example an administrator queries
the autolog tables to find out who
accessed the tables
when they were accessed and whether or
not access was granted
to summarize unity catalog which is
natively built into the databricks lake
house platform is a unified governance
solution for all data and ai assets
unity catalog centralizes data and ai
governance with audit and access control
capabilities
offers building data search and data
discovery generates automated lineage
for all workloads and integrates with
your existing governance tools
to learn more about unity catalog please
visit databricks.com thank you
[Applause]