Hi, everyone.
I'm Matthew Hicks,
a Senior Product Manager for OneLake at Microsoft.
Today, we'll learn about how OneLake’s One Copy approach
lets multiple teams in your organization
access the same data from different compute engines
without any data movement or duplication.
Compute engines power
all the analytical experiences in Microsoft Fabric.
In Fabric, the compute is completely separate from storage.
But what does this mean?
Fabric doesn't require you to have separate storage
for each compute engine,
and it doesn't offer just one multipurpose
compute engine for all your analytics.
Fabric provides multiple compute engines,
which can all access the same copy of data
without having to duplicate or move the data.
This means you will always be able
to use the best engine
for the job that you're trying to do.
For example,
if you have a data warehousing team
building a fully transactional data warehouse,
they can use the T-SQL engine to create tables,
transform and load data in OneLake.
Now, if a data scientist wanted to make use of this data,
previously they would have had to use a connector
which goes through the T-SQL engine,
or they would have had to copy
the data out of SQL and into the lake.
But with Fabric, the T-SQL
engine is natively storing its data
in OneLake in delta parquet format.
This means that data
scientists can use the full power of Spark and open
source libraries to read the data
directly from the data warehouse.
The same goes for business users
trying to view reports in Power BI.
Power BI reports
use the analysis services engine to query data.
And with the new Direct Lake mode,
the analysis services compute
engine can read the delta
parquet files into memory
without making another copy,
combining the best of import and direct query.
If your data engineering teams are more familiar
with Apache Spark rather than SQL,
they can use the full power of Spark and notebooks
to transform and load data into a lakehouse.
The T-SQL engine can still be used to create views
and serve the data to business
analysts running SQL queries.
Business users can view their Power
BI reports using the same copy of data
with Direct Lake mode in the analysis services engine.
When defining the data strategy for your organization,
you no longer need to optimize for different teams
with different skill sets and preferences.
Teams that want to work with SQL can work with SQL.
Teams that want to work with Spark can work with Spark.
Everyone builds the same data lake.
There are no silos.
And with open access for OneLake,
the same is true
even for teams using compute engines outside of Fabric.
Your data engineering
team could use Databricks notebooks and use the ADL's DFS
APIs to land the data directly in the lakehouse.
They could also create shortcuts to existing ADLS Gen2
or Amazon S3 accounts
to virtualize existing
external data into a Fabric lakehouse.
All engines would still work over the same copy of data.
We're doing a lot of work
to optimize our engines, to work
directly with delta parquet
as their native format for tabular data,
as you can see T-SQL for data warehousing and Direct
Lake mode in analysis services for Power BI.
Let's jump in and see one copy in action.
These are a few tables that I've made available
in my lakehouse using different Shortcuts.
The order table from Microsoft OneLake,
the customer data set from Amazon S3,
and the product data set from Azure Data Lake Storage.
I can see these tables are backed by delta parquet
files in OneLake.
With OneLake’s One Copy approach,
the same data can be accessed by multiple compute engines
regardless of whether the data is stored
natively in OneLake
or logically available through shortcuts.
A data scientist can train their models
directly over this data using a Spark notebook.
Here we can see a query running on those three data sets.
Data warehouse professionals can query and analyze
that same data, joining across those three data sets.
And business analysts
can simply navigate to the modeling view
and start developing their data model.
They can create rich BI reports immediately
with great performance using Direct Lake.
With OneLake,
I can easily organize my company's data
into a single unified logical lake
with one copy of data
that can be used across domains and projects
and by multiple compute engines.
OneLake empowers data engineers,
data scientists, as well as SQL and BI analysts
to all collaborate in a common data mesh
on the same copy of data
without any data movement or duplication.
Thanks for joining.
Let us know what features you'd like
to see next in the comments below.
To learn more and get started with OneLake today,
head to aka.ms/OneLake.