- So with spark 3.0,
we have now added broad support for SQL
we've also integrated it with Python
and in some sense its really unifying
now SQL analytics or data warehousing
and data science
and that's really exciting
because that's where we think the future is going.
In fact, our customers were
unifying these two things for awhile
and actually a pattern has emerged
in which they call this the Lakehouse paradigm.
So I want to talk a lot about that today.
But before I give into details of the lake house,
I want to provide you
some context of why we need it.
So data warehouses were built 40 years ago
and they were purpose built for BI reporting
and they work really great for that.
However, they don't have any support
for video data, audio data, and text data
which is a lot of the kind of data we have today
and as a result of that,
its very hard to the
data science machine learning
on these data sets in the data warehouses.
Also, they have limited support
for real time streaming
which is becoming increasingly important.
And finally, they're closed and proprietary formats
so the data is either locked in
in a data warehouse
or you have to take it out
if you want to operate on it with other tools.
Therefore, today most of the data
is actually first stored in a data lake
or blob store and a subset of it
is moved into a data warehouse.
So lets looks at these data lakes.
So the data lakes, they're great
in that they can handle all your data now
and you can do data science
and machine learning on it.
However, they're bad at exactly the things
that the data warehouses were good at.
So they have really poor BI support,
they're very complex to set up and configure,
you need advanced data engineers to do that
and then once you start using them
if you try to do BI ever on a data lake
you gonna find out that the performance
is really, really bad.
Hence, as a result what we're seeing
is that a lot of these data lakes
have turned into unreliable data swamps
and actually most organizations
unfortunately have to have
both of these things side by side.
So, that is why people believe
in the lakehouse paradigm.
The lakehouse paradigm brings
the best of both these worlds.
You get the structure,
you get the BI and reporting advantages
of data warehouses and you also get
the data science machine learning,
AI and real-time capabilities of data lakes
on all of your data.
In fact, the lakehouse is distinct from data lakes.
It actually starts out like a data lake
so it looks at the bottom layer
like a data lake,
but the crucial difference
is that it has something that we call
a structured transitional layer.
This layer brings quality, structure,
governance, and performance to data lakes.
And at Databricks we believe that open source
Delta Lake projects is a great way
to get structural transactional layer
on top of your data lake.
So I'm gonna talk a little about this next.
So before I go into the details
of the Delta Lake open source project,
I want to tell you a little bit about its history
and how it came about.
So Delta Lakes are built on top of data lakes
and as we know data lakes
they're great, they're really cheap.
In fact they have 10 nines of durability,
They can scale infinitely,
they can also store now all kinds of data,
video, audio, text.
They can also store structured,
semi-structured, and unstructured data in these.
And finally, they're based on open, standardized formats.
In recent years we've seen in particular
the Parquet format becoming the standard in this ecosystem
and lots of different tools can now directly operate
on these Parquet files.
So that's awesome.
As a result of this, organizations have now been promised
that if they just build a data lake,
they can get all these benefits.
They can get real-time analytics,
they can get data science,
they can get machine learning,
they can get data warehousing,
reporting, BI, everything.
So they've been very excited,
they've built up these data lakes
unfortunately though most
of these projects are actually failing.
And at Databricks we started
to actually see lots of, lots of demand
for professional services
and we had to put our own solutions architect
on these customers to look at
what's the problem that they're facing.
And what we did is we ranked ordered
the problems we have to address with these customers
and I'm gonna walk you through each of these problems
so there's nine of them,
in the order of sort of severity that we saw,
and then I'm gonna talk about how we addressed them
with the Delta Lake project.
Okay, so the first problem we actually noticed
that is happening over and over again
is actually very trivial simple problem
but its actually that we were getting called in
and customers were saying we have a hard time
appending new data into the data lake
while at the same time correctly reading it.
Turned out, some of the new data comes in
if you try read it at the same time,
because of eventual consistency,
you might not see the results.
So our solutions architects were creating copies
of the data in different directories
and switching it over when it was ready.
Needless to say this was
really, really complex and costly.
Second, modification of existing data
is really difficult with data lakes
and this is exasperated when companies
are trying satisfy GDPR or CCPA.
So in GDPR and CCPA if a customer says,
"I don't want you to store any records
in your data sets about me"
you have to go and scrub those and delete those.
All that's really hard with traditional data lakes
because they're basically battery-oriented systems
so you have to run a big spark job
all over your data set once a week
to scrub it to make it complaint.
This was again extremely costly.
Third, some of our customers were having problems
that were hard to diagnose.
After we looked into the logs and did diagnostics,
it turns out that many years back,
maybe some spark job failed half way through,
some of the data it was ingesting made it in,
the rest was missing,
and as a result the whole project was failing.
Four, real-time operations
are really, really hard on data lakes.
This is a special case of problem number one
but mixing real-time streaming with batch
led to inconsistencies.
In particular, if you're appending new data
in batch operation but at the same time
in real-time you're trying to read it,
you're gonna have problems
and this one is actually harder to solve,
its not just to copy files or copy directories.
Five, organizations want to keep
historical versions of their data,
especially regulated environments
you have to reproduce your data sets
cause there's auditing and governance that happens.
So here again, the poor man's version
was keep copies of all your data sets
and different directories with different date names.
Very, very costly.
Six, difficult to handle the metadata itself.
Now as these data lakes were growing,
the data itself was growing into petabytes.
The metadata for these systems was itself
now becoming terabytes in size.
And as a result of this the catalogs that are storing
the metadata were falling over, they were not scalable
so we were pulled in to do
professional services again to fix that.
Seven, the "too many files" problems.
Because data lakes were file-oriented,
you could end up in a situation were you had millions
of tiny files or a few gigantic files
again leading to lots of problems
when you're trying to consume it.
Eight, it was hard to get great performance
because you're just dumping your data into the data lake
without thinking about how its gonna be used later.
How do you know if the data layout
is sort of optimized for the performance
that you need later.
And finally, the most important problem in my opinion,
but a hard one to actually know
was data quality issues.
Its constant headaches where customers would pull us in
and it turns out that the data
was changing in a subtle way,
or the semantics were changing over time,
or all of it wasn't there.
So this was leading to all kinds of problems
with these projects.
So, at Databricks we wanted to take
an opinionated approach.
We said we want to build a system
that from the get-go tries to get all these things right
so that we don't ever have to configure them
or ever have to address these issues ever again.
So the Delta Lake open source project
addresses this by adding massive reliability,
quality, and performance to your data lake.
And the way it does it is by bringing the best
of data warehousing and data lakes to one place, together.
And finally its based on open source Parquet format
so it can keep all your data on your data lake
in this open source format with all these benefits.
So how does it do this?
So these were the nine problems that I mentioned earlier
and I'm gonna go through each of these
and mention how we actually address
this with the Delta project.
Well, luckily it actually turns out
the first five problems could all be addressed
with the same technique.
By having a transaction log that stores all the operations
that you're doing on your data lake,
you can actually now make it fully atomic.
That means that every operation
that you do on your data lake
either fully completes or it aborts
and clean up any residue and you can try it later.
The way that actually happens is that there is a delta log
and it stores every operation.
So for instance here we have an example
where one operation is adding two files,
two Parquet files.
They will atomically be added.
Then after that, maybe that first file is deleted
and a third file is added.
The key thing here is that Delta
always makes sure that each of these are happening
as if it was sequentially executed.
Okay, so that's awesome.
Now, we can make sure that one,
if were appending new data into the system
its always either gonna be there
or its gonna be canceled
and you can read it at the same time,
you won't have any issues.
Two, modifying existing data if you want
to get GDPR or CCPA compliance actually is really easy now
because transactions enable you
to do fine-grained up search.
So you can go in and modify small data
in your data set in a transactional manner.
Jobs failing midway, that also goes away
because the job either fully succeeds or fully is aborted.
And then real-time operations, its the same thing.
All operations, whether real-time or batch
are consistent with the transaction log.
Now finally, because we have all the deltas
of all operations that happened on your data set
we can actually implement something called time travel.
Time travel lets you submit queries
and then say, "I want the answer to this
as of the time two years ago."
And it gives you the response as if
the query was submitted two years ago.
So this is great now for compliance and governance.
Okay, so that's awesome.
Now we solved those first five problems.
How do we deal with the metadata problem?
The fact that metadata is getting large
and the catalogs are falling over.
Well luckily it turns out that
there's a project called Apache Spark
and its really, really good
at large scale parol operations.
So under to hood we use Spark and all our metadata
is stored in Parquet format
right next to the rest of your Parquet files.
That way you can actually move your data
from one place to another by just copying the data.
You don't have to worry about catalogs that are out of sync
with the data that you have.
So that's awesome.
So okay, now we can scale the system
with the Delta Lake project to really large metadata.
What about the "two many files" problems?
Well, the "two many files" problems
and the performance problems that we're seeing
with data lakes we now automatically fix those
because we automatically optimized the layout of your data.
How did we do that?
Well, a bunch of indexing techniques.
One, we partition of course your data.
But two, we also do data skipping.
Data skipping enables you to prune the files
based on statistics that you have on numerals.
So you know exactly what's the max and minimum
of every data that you have in the files
so that you can actually avoid reading
a lot of the files when you get a query.
And then finally, we have something called Z-ordering.
Z-ordering action enables you to optimize
multiple columns at the same time
so if you have a year, months, dates index
you know can actually search on just date
and it can be as fast as if you searched on year
which is hard with partitioning.
Okay, so this is awesome.
Now we get all these advantages that we had in
data warehouses as it comes to performance,
what about the last problem?
The most important and the hardest problem.
The quality issues.
Well, it turns out that we can actually solve that
with schema validation and evolution.
So all data in delta tables have to adhere
to a strict schema.
It could be a star schema, a snowflake schema,
but as soon as you have the schema,
we make sure that all your data
100% adheres to this schema.
If any data appears that's not satisfying
the schema that you have we actually put it
on the side in a quarantine that you then look at,
you can clean it up,
and when you cleaned up the quarantine
the data makes it back into delta
but now you know that all your data
in a delta table is always pristine.
And finally, we've actually added something
called Delta Expectations.
Delta Expectations is a novel way
in which you can actually express
quality expressions using SQL
or user defined functions UDFs
where you can specify pretty much any business logic
for quality that you like
and now we guarantee that your table satisfies
all those quality metrics, always.
And with expectations and schema evaluations
the pattern that we're seeing our customers
use in data lakes is that they're actually
now creating a curated data lake.
The curated data lake now has bronze tables
where you might have the raw data coming in
and then that gets refined into filtered, cleaned,
and augmented tables that we call silvers tables
and then finally, the most optimized business-level
aggregated data sets are in so-called gold tables.
And those are the ones that you really sort of
serving into your BI needs.
So zooming out, the Delta Lake project solves the problems
we had with the data lakes using four techniques.
First, it uses ACID transactions.
Second, it uses spark under the hood
to get the scale for the metadata.
Third, it uses indexing and lots of different
indexing techniques to get speed ups
that we traditionally use to have in data warehouses.
And then finally, it uses schema validation
and exceptions to get the quality we expect to have
if were using a data warehouse.
We really want this to become the standard
for how people create their lake houses.
So as a result of this, we've made this project
work well with all these other systems.
So you can now actually read and write both from Delta
and Hive, and Spark, Presto, Amazon Redshift,
Amazon Athena, and snowflake.
And creating data with Delta is really easy.
In SQL, instead of saying using Parquet,
you just say in SQL say using Delta
and you're just using Delta.
So, with Delta Lake we're able to fill in
critical layer of the modern lakehouse architecture,
bring in structure and reliability to your data lakes
in support of any downstream data use case.