- [Instructor] The Lakehouse
is a simple and open data platform
for storing and managing all of your data
that supports all of your analytics and AI use cases.
Delta Lake provides the open, reliable, performant
and secure foundation for the Lakehouse.
It's an open source data format
and transactional data management system
based on parquet that makes your Data Lake reliable
by implementing acid transactions
on top of cloud object storage.
Delta Lake tables unify batch and streaming data processing
right out of the box.
And finally, Delta Lake is designed to be 100% compatible
with Apache Spark.
So it's easy to convert your existing data pipelines
to begin using Delta Lake with minimal changes to your code.
To get started with Delta Lake,
when writing out your tables with Spark,
instead of parquet, simply say Delta.
You can also use SQL to create a Delta Lake table
from scratch with the create table using Delta command
or you can use the convert to Delta command
to convert your existing parquet files
to the Delta Lake format in place.
Now that we've converted our data to Delta Lake,
let's look at the data in our table.
We have 14,705 batch records in this table.
And you can see here what the data actually looks like.
Next, I'll show you how Delta Lake is able to handle batch
and streaming data with ease.
By setting up two streaming rights to our Delta Lake table
alongside to streaming reads
that will happen simultaneously.
So as you can see in this chart,
the initial 14,705 batch records in our table
are still present,
but new data is now streaming in from stream A and stream B
at around 500 records per second each.
Each new bar represents a ten second window,
during which both stream A and stream B
are writing to our table concurrently.
Meanwhile, Delta Lake can also serve
consistent views of this table to our streaming read queries
which are powering the visualization below.
Finally, just for good measure,
we can throw in a batch query.
So how is all of this possible?
It's possible because of acid transactions.
Delta Lake uses a transaction log
that serves as a master record
of all changes made to each table.
That also determines
which files are part of our table at all times.
You can view the transaction log at any time,
by running the describe history command.
When we run this command,
you see that each right to our table
has been recorded atomically,
as a streaming update.
Once a new right is recorded in the transaction log,
all of our downstream readers will now include
that right transaction
in their view of the table immediately.
So everyone always agrees
on what constitutes a Delta Lake table at all times,
they simply refer to the transaction log.
This all makes Delta Lake tables really powerful.
And at Databricks, we found that many of our customers
are able to simplify and streamline
their overall data architectures using Delta Lake.
By building a simple multi-stop data pipeline
with Delta Lake tables, you can reliably transform
raw batch and streaming data
into high quality structured data
that multiple downstream apps and users can query at once.
And what this means from a business ROI perspective,
is lower cloud computing costs, less complexity to manage
and less time wasted on fixing systems issues
like corrupted data.
But Delta Lake does a lot more
than just use acid transactions
to combine batch and streaming data.
It also offers tools like Schema enforcement
to protect the quality of the data in your data tables.
Without schema enforcement,
data with mismatching schemers
can break your entire data pipeline
causing cascading failures downstream.
So we use schema enforcement
to ensure that that doesn't happen.
But schema enforcement alone is not enough.
In the event that we do want to change our table schema,
we also need Schema Evolution.
With Delta Lake, we evolve our schema quickly and easily
by simply adding the merge schema option
to our spark right command.
And when we run this command,
we can see that the right command that previously failed
has now occurred successfully.
And the new credit score column is now present in our table.
Another major feature at Delta Lake
is the ability to travel back in time using time travel,
also known as data versioning.
Because every change to our table
is recorded as an atomic transaction
in the transaction log.
We can use this information
to recreate the exact state of our table
at any point in time.
Time travel helps you avoid
making irreversible changes to your tables.
It makes your data sets and experiments reproducible
and offers a verifiable data lineage
for audit and governance purposes.
So let's first review the transaction log
using the describe history command that we saw earlier.
And as you can see, each version of our table
is saved by version number and by timestamp.
So we can use this information to do things
like query historical versions of our table.
So let's go ahead and do that.
By adding the version as of command to our SQL query,
our query runs on the very first version of our dataset,
version zero, and returns account of 14,705.
The same number of records that were originally present
in our table.
Taking time travel one step further,
we can roll back our table at any time
using the restore command shown here.
This is really useful if we decide
that we've made a change to our table
that we want to completely undo
and simply roll back our data to an earlier version.
So as you can see, now when we query our table,
those original 14,705 batch records
are the only records present in our table.
Version zero has become the current version.
The next Delta Lake feature I want to demo for you
is full support for transactional DML commands
like update, merge, and delete.
These are the SQL commands that make manipulating
big data tables quick and easy.
Before Delta Lake, deleting a user's data
from a data Lake to comply with the GDPR request
was difficult to perform without running the risk
of data loss or corruption.
But with Delta Lake,
we can delete the user's data transactionally
in just one line of code.
First, let's view the user's data.
We'll choose the user with loan id 4420.
And now when we run this delete command,
you can see that our user's data
is successfully deleted transactionally.
The same idea applies with other DML commands,
like insert update and merge.
Delta Lake makes his operations really simple to perform
with minimal code,
all backed by the reliability of acid transactions.
So let's go ahead and use time-travel
to insert that user's data back into our table.
And when we run the command, we see now, in fact
there is a record in our table
with loan id 4420.
Next, let's go ahead and update our users' data
with a single command as well.
Here, we're going to set the funded amount
equal to 22,000 for the same user as before, number 4420.
After running the command,
we see that the user's funded amount
has been updated successfully.
Finally, Delta Lake also supports upsert,
which are a mix of inserts and updates
using the merge command.
Normally, merges are a difficult expensive operation
that involves several intermediate steps.
With Delta Lake, we can skip all that complexity
and simply use the merge command.
So first, we'll create some dummy data to merge,
one row with an update of user 4420s data,
and one row of new data to insert in our table.
So when we put together a merge command,
we can specify that we want to update the table
when the record already exists
and insert it when it doesn't.
So when we run the merge command, as you can see,
we successfully updated user 4420s data
and inserted our new user's data.
Finally, before wrapping up,
I want to show you just a couple of additional commands
that improve performance
when working with big Delta Lake tables.
First, there's the Vacuum command.
The Vacuum command allows us to Mark any data files
that are no longer being used
by the current version of our table for deletion.
The next couple of commands are only available
with Delta Lake on Databricks.
First, there's the Cache command,
which allows you to cache the results
of any frequently used query and memory
to speed up that query in the future.
And finally, there's the Z-order Optimized command
shown here, which uses advanced multidimensional
data clustering techniques to co-locate related data
for the fastest queries possible.
Z-order significantly speeds up many queries
and reduces cloud costs
by enabling more efficient reads of our data.
So that concludes our hands-on demo
of Delta Lake.
Try Delta Lake on Databricks today
and join the community online
to learn more about how Delta Lake
provides the perfect foundation
for your Lake house architecture.