we can now kick it off with our first
talk of the day by Michael armburst
Michael is a distinguished engineer at
databricks and he is the creator of
Delta Lake and Delta live tables Michael
will walk us through the lighthouse
platform and how it is the best platform
for data engineering take it away
Michael
thanks Ari I'm really excited to be here
today to talk about the databricks lake
house platform before we get started on
that why should we even care about the
lake house we've been seeing this trend
for many years where these big tech
companies have been causing all kinds of
disruptive innovation through their use
of data analytics and AI basically their
main competitive Advantage is their
ability to acquire huge amounts of data
store it and process it so they can
understand how their business is working
and also predict the future
but I don't think this is actually
unique to these large tech companies and
in fact I would say in five years every
company is going to be defined by data
analytics and Ai and you might ask how
is that possible well let's take some
examples of the data defined Enterprise
and see how analytics and AI are
essential to their business
one area is customer engagement you can
bring relevance and Hyper
personalization to every touch point you
can predict which customers are going to
churn and take action before that
happens
you can also use data and Ai and product
development you can understand how
people are using your product and use it
to change to decide how you're going to
change that in the future
you can use it to optimize your
Workforce and then unlock productivity
through real-time Insight so people are
no longer waiting for data they just
have it at their fingertips as soon as
they need it and finally you can use it
to optimize the efficiency of how your
business operates and maximize
productivity
when we see customers start on their
data Journey they often go through this
maturity curve and we see this kind of
broken into kind of two key sections in
the beginning you're mostly looking at
the past you're trying to understand
what happened you start by just trying
to take all of that data you've
collected and clean it so that it's
ready for consumption once you've got
clean data you can start building
reports that analyze how things have
been happening in the past
as people get more advanced they'll
start asking questions and using the
data to answer ad hoc queries and
they'll start exploring it to discover
new insights
once you've got all of that mastered
though you can start looking forward
into the future you can start asking
questions like what will happen and how
is the best way to respond to that and
when you get really Advanced you can
finally start automating it so that
those decisions are automatically made
by the system with no human intervention
and so kind of in summary what we see is
there's kind of two classes here you
have business intelligence where you're
looking at the past and artificial
intelligence where you're starting to
model and predict the future
now this sounds great but the problem
here is most organizations struggle
because their systems aren't able to
support their goals
and the reason for this comes down to
this dichotomy between the traditional
data warehouse and the data Lake
we have the familiar data warehouse you
know or traditional relational database
which is highly reliable and very
efficient it's been used for many years
to do business intelligence and SQL
Analytics but then came along the data
Lake and this unlocked a whole new type
of data we could now store vast amounts
of raw data in the form of logs text
audio video or images and the problem is
having these two different worlds you
end up spending a lot of time copying
data in between them and this disjointed
and duplicative data silos lead to
problems
additionally these two different worlds
had incompatible security and governance
models so you're spending all of your
time trying to reconcile who has access
to what data and who shouldn't have
access to what data
and the reason you end up kind of
Bridging the the gap between these two
worlds is neither World supports all the
different use cases you want the data
warehouse was great for doing business
intelligence and SQL analytics but only
the data Lake could do data science ML
and low latency cases like data
streaming but we think that there's no
reason to have these two disparate
platforms instead we propose the lake
house paradigm the idea here is you have
a single reliable data platform that can
efficiently handle all data types
whether it's structured data or
unstructured files
and the foundation for this is Delta
Lake which I'll talk about in a little
bit
on top of that you have one security and
governance approach for all of your data
assets that works on any cloud
and we call this Unity catalog
and then finally you can do all of your
ml SQL bi and streaming use cases on top
of this platform these are all of your
different data applications
so now we have the NATO lake house one
place for all of your data analytics and
AI unified into a single platform
and this is something that not just
databricks has noticed in fact we've
seen it across all of the different
Cloud vendors
you can see that Azure has been talking
quite a bit about the lake house
you know Google has also been talking
about their their open data Lake
AWS has been you know one of the places
where the the lake house was born and
even traditional data warehouses like
snowflake are taking note
but one of the things you'll notice is
do these offerings really stack up and
are they a true lake house and one of
the key problems here is you know while
it's great to have open support for open
formats you really have to architect
your system from the beginning in order
for it to be cost effective to operate
in this way here you see a comparison of
the total cost to run a standard
decision support Benchmark called tpcds
on three terabytes of data and so what
we did was we basically took this
Benchmark and we tried to see how all of
these different cloud data warehouses
would perform when they were using open
lake house formats and what you can see
is databrick SQL is dramatically cheaper
while these other engines are able to
operate on lake house floor mats they
aren't optimized for it and so you end
up paying for significant inefficiencies
and we wanted to drive a little bit
deeper and understand exactly where was
that cost coming from and you can see
it's not just running the queries in
fact a significant part of the expense
is just loading the data into the system
and allowing it to do auto clusterings
that you can get good performance in
databricks since we've kind of optimized
for the lake house architecture from the
beginning you can see all of the steps
of the data Journey are dramatically
more efficient and therefore
dramatically cheaper for you
and it's not just benchmarks that are
showing this we've seen performance and
efficiency from many of our different
customers for example Johnson and
Johnson used the lake house architecture
to replace 35 different data sources
streamline their supply chain management
and dramatically improve their inventory
predictability and what this meant was
they were able to get the data
significantly faster and they were able
to cut their costs for data engineering
by up to 50 percent
so what does it actually take to build
the lake house now I want to take some
time to go through some of the different
layers and talk about why they're
important
so at the bottom of the stack we have
Delta Lake Delta lake is the system that
allowed us to rather than store a bunch
of parquet files out uh you know on
cloud storage it actually took all of
the nice things of cloud storage meaning
that it's very cheap to operate and
Incredibly elastic so it can scale to
you know any amount of data that you
want to store in it and it changed it
from just being a dumb collection of
files to a full relational table with
all of the benefits that that means
so what Delta Lake brings is both data
reliability and performance to your data
lake so it's designed from the beginning
for high performance transactions at
massive scale it can handle you know
millions of partitions and millions of
files and it brings reliable table
operations by bringing acid transactions
to these operations so you no longer
need to worry what happens when there's
a failure assets will automatically take
care of it for you and clean up it was
built for the beginning from streaming
because we knew that low latency was
becoming increasingly important it's
also the only industry standard
completely open way to do data sharing
and a key benefit here is it's
completely open in computer Community
Driven so it's not uh you know owned by
one single vendor and it has inputs for
many different people around the space
the community is pretty big and you know
this is one of I think it's key uh you
know enabling parts of Delta all of the
source code is available on GitHub
governance is managed as part of a Linux
Foundation project we talk about the
public roadmap you know is all available
out there on the internet and finally we
have community events and office hours
where you can meet other people who are
trying to solve similar problems and
what this means for you is you never
need to worry about being locked in by
your data format
another key benefit of being an open
ecosystem is it means that you aren't
stuck using one tool for one job you can
use whatever the best tool that you
prefer is at any time so Delta lake has
always had support for spark And Hive
and some of these other kind of you know
more traditional Big Data formats but as
we've expanded the project we added
support for beam and power bi and Kafka
and we just recently added support for
Flink trino and Presto and in the near
future the ecosystem will continue to
expand as we add support for pulsar and
Google's bigquery
and one of the key Parts about Delta
lake is performance it brings a
significant performance Improvement not
only over just throwing a bunch of
parquet files but over many other
storage formats that had recently been
created we again ran this tpcds
Benchmark on Delta Lake and two of its
other open source competitors and what
we see is Delta lake is dramatically
faster in some cases it's you know
almost two times faster and in other
cases it's almost four you know four
times faster than these other formats
what that means is you can save a lot of
money just by using Delta Lake
and on top of now that we've kind of
figured out exactly how all the data is
going to be stored now we can talk about
how we're going to govern all of that
data and our answer for that is Unity
catalog
now before we talk about how unique
catalog solves these problems talk about
what these problems are so when you just
had a data Lake you were basically just
managing permissions on top of a big
collection of files and this is actually
pretty difficult because it's incredibly
coarse grained it means that you can't
you know Shield individual rows or
individual columns and that kind of
course granularity makes it very
difficult you end up creating lots of
copies in order to deal with it
uh another problem is the metadata about
the files and the metadata about the
tables was often stored in two different
systems and they can be they can be out
of sync and it was just a lot of
orchestration and management
and finally you know if you have a
separate data warehouse you now have a
different governance model you need to
think both about these files as well as
the management of your data warehouse
and if you're getting more advanced and
you start doing doing ml models and
dashboards you have yet another
governance model to worry about
and this is why we created Unity catalog
it's a single permission model for all
assets whether it's collections of files
or Delta tables or uh you know ml
dashboards and models you can manage all
of these things in a fine-grained way
using standard anti-sequel Grant and
revoke semantics and this means you not
only have this kind of centralized
governance but you also have centralized
auditing and lineage as well so you can
understand exactly where data is coming
for and how it's being used
and so to summarize the data lake house
brings all of your different data into
one place into one platform so you can
do all the different operations on top
of it that you want
but now I'd like to focus in on one of
those challenges which is data
engineering
and the reason I think this is so
interesting is because many
organizations struggle they can start
you know getting data in and analyzing
it but as the scale increases
maintaining that infrastructure at stale
that scale becomes very complex
and what we've seen is as a result of
this complexity a lot of data that is
collected is actually just never used
it's just thrown out and the reason for
this is it's a very hard problem to do
this data engineering data Engineers
spend a lot of their time on
undifferentiated heavy lifting what that
means that they spend a lot of time hand
coding data ingestion Transformations
they manage a whole bunch of scalable
infrastructure they have to think about
orchestration and managing their
production workloads and again as you
have all of these different tools you
end up having incompatible governance
models that need to be managed
and this is why we think the best data
engineering platform is a lake house let
me walk you through that
so the idea here is we're going to be
able to connect to all of those
different open sources that the lake
house enables will be able to adjust
transform and organize and then that
will allow us to do streaming analytics
bi and Reporting and data science and
MMO
and we have two kind of key tools for
doing this Delta live tables which is
the best way to do ETL that means
whether you're ingesting data using our
tools like autoloader and copy into or
transforming it until it's ready for
consumption uh doing automated ETL with
DLT in either batch or streaming and
then since you know once you've got all
the data in now you need to do all of
those operations on it and that requires
orchestration and this is where data
databricks workflows comes in this gives
you reliable orchestration for data
analytics and AI that works across the
entire platform let me dive into both of
these tools
first let's talk about ETL why are we
doing ETL well it's basically a fact of
life that all data starts off messy and
so you need to take this messy data you
need to filter it clean it augmented and
then you need to build business level
Aggregates that actually answer
questions and that enables all of this
Downstream stuff but this picture here
that I'm showing is really quite simple
compared to what it looks like in real
life a real data architecture often has
tens to hundreds of tables with complex
interdependencies between them and what
you often find is you end up spending a
lot of your time on managing this
brittle infrastructure you have a
problem for ETL so you bring in Amazon
blue in order to
do but then you need to understand
exactly how many rows are being
processed to make sure nothing's getting
dropped so you bring in datadoc you want
to make sure you're getting the best out
of your machine so you bring in some
low-level metrics from Amazon cloudwatch
you need to orchestrate all these
dependencies so you have to set up your
own airflow cluster
and then after that you need to ingest
some data from another data source since
you end up bringing in Azure data
Factory
and then I need to run some reports and
SQL queries and so I you know use Uzi
because this old Legacy team is doing it
then finally we can pull in another tool
like you know Azure synapse what you see
is this this tooling ends up dominating
all of your time
and this is why we created Delta lime
tables Delta live tables makes it dead
simple to build reliable data pipelines
and in fact these few lines of SQL are
actually a full end-to-end production
pipeline let me walk through it really
quickly
um first we see this create streaming
table where we're actually doing uh
ingestion of raw data but the Simplicity
of the SQL is actually kind of hiding a
lot of what's actually happening under
the covers so under the covers we're
taking this directory of Json data
called raw data we're automatically
inferring the schema to understand what
that data looks like cloud files also
allows you to automatically tune how
you're going to handle Evolution if new
columns appear or columns disappear they
change data types all configurable by
cloud files the other really nice thing
that it's handling is it's handling
ingesting this data exactly once at any
scale so we'll integrate directly with
these underlying cloud storage systems
to make sure that we're getting actual
push notifications so even if there's
tens files or tens of millions of files
it's going to work the same way
once the data is ingested now we can use
materialized views to transform and
clean it up and the nice thing here is
you can see we say select from live.raw
data that's automatically creating a
dependency between these two tables and
the system will automatically
orchestrate how data flows between them
what we what this allows you to do is
dramatically accelerate your ETL
development you can write code in SQL or
Python and DLT automatically handles the
orchestration of the dag and all of
these complex Cloud problems like
handling retries and changing schemas
we automatically take care of the
infrastructure for you you don't really
need to worry about exactly what type of
cluster and exactly what type of
instance we have automatic recovery we
have automatic Auto scaling and we
automatically optimize the performance
of how this pipeline is going to operate
we have built-in tools for ensuring High
data quality so we can automatically
basically allow you to specify these
declarative expectations about what good
quality data means for you and the
system will automatically Monitor and
enforce these rules
and then finally you know as you can see
in this example we unified batch and
streaming into one simple language so
you can have the Simplicity of SQL while
having the freshness of streaming all in
one single API
and the key part of the DLT story is
end-to-end observability and monitoring
so every part of how your pipelines
operates is actually captured in the
system called the event log what that
means is you can now create dashboards
that tell you not only how much data is
being processed but also kind of low
level details like exactly what the
quality is if there have been any data
quality problems over time so you can
find them and proactively fix them
and now that we've got all the data in
now it's time to orchestrate all the
analytics and AI that you can do on top
of it and the story for that is
databricks workflows data workflows
allows you to orchestrate anything
anywhere you could use it to run old
school spark jobs wrapped up as jars
databricks notebooks orchestrate Delta
live tables to train models to run
Python scripts or even use outside tools
like DBT
the nice thing about databricks
workflows is as a simple workflow
authoring experience all you need is a
web browser and you can log into our UI
to create these dags and ml workflows
it's deeply integrated into our platform
so you have centralized monitoring and
centralized observability and it's fully
managed it's incredibly important that
your production workloads always run on
time no matter what's going on in the
cloud and databricks workflows takes
care of that we have proven reliability
from you know 10 years of running these
these critical workloads for our
customers
let's walk through an example workflow
can look like so you often start by
ingesting data and transforming it using
autoloader and DLT then you can do some
more filtering and quality assurance
again using DLT SQL or python then you
can integrate with tools like ml flow
where you can actually do things like
feature extraction and finally we can
save those features and train a machine
learning model that allow us to predict
what's going to happen in the future
and you know one of the cool things
about the the data lake house is it's
not just about databricks there's an
entire ecosystem of partners for doing
everything from ingestion to data
pipelines to governance and then all the
tools on top for doing bi dashboarding
machine learning and data science
and we've had over 7 000 customers see
success with this over the years
I want to dive in on just one of those
customers uh you know where we're
talking about streaming an ETL with
shell the problem here is basically
predictive uh maintenance and analytics
so they have all of these big turbines
around the world and they're producing
tons and tons of sensor data on how
every individual component within that
large complex system are operating and
one of their key problems here is they
want to understand and predict when
things are going to break so they make
sure they have the right parts and the
right personnel and minimize the
downtime caused by that outage and using
DLT they were basically able to collect
all of this data you know
multi-trillions of records and
continually improve their ability to do
that analytics and what that means is
now they have hundreds of anal Engineers
collaborating on these different use
cases saving them millions of dollars in
repair costs
and we have similar stories with
databricks workflows in this case we're
talking about yippin data where they
collect you know different data from
hundreds of websites around the world in
the past they had a lot of problem with
siled teams that were unable to kind of
scale their legacy data technology
they're spending all of this time
managing their own airflow cluster
rather than actually spending the time
getting insights from their data and so
when they migrated from airflow to
databricks workflows they had a single
unified platform that allowed to manage
the entire life cycle of doing data
analytics and the end result here was
they lowered their costs by up to 60
percent and they also reduced the
processing time to buy 90 so that means
they're getting faster answers
significantly cheaper
and so to summarize you know we believe
the desk the best data engineering
platform is a lake house it gives you
reliable data for analytics and AI by
giving you simple batch and streaming
data ingestion with Delta live tables
reliable orchestration for all the
things you do with databricks workflows
and it's built on top of a foundation of
reliability performance and governance
that you get from Delta Lake Photon and
unity catalog thank you
thank you Michael for that great
introduction to the databricks like
house and data engineering on the lake
house we'll now hear a little bit more
about DLT and workflows in two Deep dive
sessions from Raju and Roland faustling
both staff product managers at
databricks we'll start with you Ray take
it away
all right thanks for having me here I'm
going to spend the next few minutes to
give you guys a little bit of an
overview and also a deeper dive um into
Delta live table or Dot dot is the first
ETO framework that uses a simple
declarative approach so building and
running reliable data pipelines at scale
and without any infrastructure
management
with DLT data analysts and data
Engineers can spend less time on ETO
tooling and focus on getting married out
of their data now I want to highlight a
few of the key dot capabilities
first DLT allows you to develop ETO
pipelines in a declarative manner where
you define the what as opposed to how
this approach is much faster and less
error-prone during the pipeline
development phase
next dot manages your ETO infrastructure
like scaling the compute recovering from
failures and optimizing table storage
performance for you
and third that data will occur
inevitably from the data source which
eventually impacts your analytics and
machine learning workloads Downstream
dot has built-in data quality capability
to prevent any bad data from flowing
into your production tables and
applications
and finally as more and more data
pipelines moving towards streaming for
more real-time results
dot is built with both streaming and
batch support from the get-go and all
through a unified API
this is a quick diagram of dot in the
context of the broader lake house
platform
DLT is the default and best-in-class ETL
product sitting on top of the Delta Lake
and governed by the unity catalog
products
the highly performant Photon engine is
natively supported and built into dot
for pipeline execution
dot handles end-to-end ETO from data
ingestion to transformation and again
with support for both Dash and streaming
once you get your claim data in the
production Delta tables you can run a
variety of Downstream workloads like
business intelligence interactive
analytics machine learning and
operational applications also The
databricks Lakehouse platform
now let's take a look at continuous data
ingestion which is typically step 0 of
any data workloads
dot supports continuous data ingestion
from a variety of sources like cloud
storage systems message buses and dbfs
when it comes to cloud storage which is
where the most of the data is stored
these days
dot supports a number of file formats
like Json CSV Avro and parquet
dot also supports automatic schema
inference Evolution rescue data column
through a feature called autoloader
in addition to data ingestion you often
need to perform Transformations and
aggregations on your data before they're
ready for Downstream consumptions
and an ETL pipeline in the real world
could get fairly complicated with lots
of dependent steps these pipelines are
hard to develop and even more difficult
to maintain when you consider scenarios
like failures reruns and backfills
DLT lets you define ETL pipelines with
its simple complex being a declarative
manner
what this means is that you tell dot the
intent of how you want your data to look
like and Dot generates the step-by-step
ETL pipelines for you and there are a
few advantages of this approach number
one it's much less aerochrome during a
development phase because dot will check
for and Generator of dependencies for
and number two because dot generates the
ETL pipeline the engine has inherent
knowledge about the data flow
which allows us to deliver valuable
features like end-to-end data lineage
automatic failure handling and
backfilling
change data capture which has
increasingly become a popular workload
apart from capturing change data from
The Source a critical capability for
this workload is to insert update and
delete changes into the target Delta
table
and before dot we provided the merge
into feature in the database runtime for
the CDC workload which had a set of
limitations and complexities like order
Handler
with DLT we now provide an enhanced
version called apply changes into
this new CDC feature allows you to
specify primary key of your change data
and the conditions for triggering insert
update or delete actions
but play changes into handles out of the
order data for your automatic speed and
also supports partial updates for Source
systems that only produce partial change
data
and finally apply changes into also
support schema Evolution as a source
data schema changes
next data quality which is among the
most important aspects of any data
platform and remains a big problem for
most data teams
DLT provides a feature called
expectation to manage data quality out
of the box you can specify the
expectations for your data such as the
value of the data should not be now as
well as what the OT should do when
certain data violates this expectation
and some possible violating
violation handling actions include
Fielding type pipeline drop the bad data
alert the users or in the very near
future you can quarantine the violating
data for reprocessing afterwards
and all the data quality related metrics
are logged into Dot's event log so that
you can query and visualize the
observability data for additional
insights
which leads me to the next topic
observability with the event log feature
so in addition to the data quality
related metrics dot also lauds other
pipeline observability metrics like data
lineage compute skating events pipeline
execution time data volume Etc
being a place what we call Dot event law
which itself is also a Delta table and
that means you can use databricks
products like DBC code or notebook to
query analyze visualize and even set
alerts on these observability data
stored in the dot event now
data platform is increasingly used in
Mission critical application
applications and has an increasing
impact on businesses when data is not
available delayed or be not accurate
we believe that software engineering
best practices like environment
separation testing documentation and
parameterization are equally important
for delivering highly available and
reliable data platforms and this is why
we build software engineering best
practices into dot by default
DLT supports the separation of
development and production environments
as well as testing and documentation
capabilities
so that you can have the confidence to
deploy any pipeline into your production
environment
in addition DLT also allows you to
parameterize your ETL pipelines which
allows you to do things like testing
your Dev Pipeline on a smaller data set
and deploy to production on a much
larger data set
it also allows you to separate long
paths and other variables from your
pipeline code and apply the same
transformation logic to different data
sources
ETL pipeline operations and that's big
deal
many data teams today spends 20 percent
of their time on Pipeline development
and 80 on operations and maintenance
so large ETL pipelines we run after
failure could get fairly complicated
when you consider dependencies and
duplicates
next maintenance like vacuum and other
optimizations need to be performed on
your tables in order to achieve good
Downstream query performance
and lastly data volume almost always
changes and infrastructure failure like
computer networking failure almost
always happen over a long enough period
of time
which means you also need to carry a
pager and spend effort scaling and
managing your infrastructure
now with DLT we handle all of the above
operational work for you so that you
don't have to
our goal is that once you set up your
ETL pipelines in DLT it keeps running
reliably forever until you stop it
and finally I want to cover oxidation
with our workflow product
data pipeline is almost always a part of
an end-to-end workflow for example once
your ETL pipelines are executed
successfully you might want to send out
an email notification or maybe updating
dashboard in your bi tool or maybe
trigger an update to your machine
learning model with the new data
databix workflow is the fully managed
orchestration product for the lake house
platform it's multi-cloud and allows you
to orchestrate dot pipelines as well as
any arbitrary tasks to complete an
end-to-end workflow