- [Instructor] Databricks provides a unified,
open platform for all your data.
It empowers data scientists, data engineers,
and data analysts, with a simple collaborative environment
to run interactive and scheduled data analysis workloads.
Databricks is from the original creators
of some of the world's most popular open source projects.
Apache Spark, Delta Lake, MLflow, and Koalas.
It builds on these technologies to deliver
a true lakehouse architecture,
combining the best of data lakes, and data warehouses,
for a fast, scalable, and reliable data platform.
Built for the cloud, your data is stored
in low cost cloud object stores,
such as AWS S3, Azure Data Lake Storage,
and Google Cloud Storage.
With perform and access enabled through a caching,
data layout, and indexing layer.
To work with your data,
you can launch clusters with hundreds of machines,
each with a mixture of CPUs and GPUs
needed for your analysis.
If you're on a large data team,
policies can define limits
on cluster sizes and configuration.
There is a Databricks runtime for data engineers
and data scientists, as well as a runtime optimized
for machine learning workloads.
Here we're creating a cluster with up to 390 workers.
See how easy that was?
(upbeat music)
In the data science workspace,
you can create collaborative notebooks,
using a variety of popular programming languages
such as Python, SQL, Scala, or R.
(upbeat music)
Just like you can share your Google docs
with your colleagues and groups of colleagues,
you can also share these notebooks.
Plus built in commenting, tied to your code,
helps you exchange ideas and updates with your colleagues.
In addition to using the notebooks
for exploratory data analysis as you see here,
many Databricks users love the powerful integration
with popular machine learning frameworks like MLflow.
Here, we're training a model and testing it,
but looking at the top right,
we can see the MLflow experiment tracking feature,
which allows us to track the previous experiment runs,
and look how important variables like accuracy
have changed over time.
Now MLflow is just one of the integrations
that Databricks provides with popular frameworks
for machine learning and data science.
Databricks also supports a variety of open source libraries,
which are popular in the community.
(upbeat music)
Want to know more about what data your colleagues
have shared with you?
Take a look at the data tab where you can see
individual tables with schema, and sample data.
Importantly, you can see the history of operations
performed on each table, AKA, the transaction log.
Now why does history matter?
Well, it is important for compliance and security audits
in many industries.
But it also enables you to explore your data
by another dimension, time.
Let's see how by opening up the SQL Analytics interface.
SQL Analytics gives us the ability to create
visualizations, and dashboards,
as well as query our lakehouse
with performance exceeding,
or comparable to traditional data warehouses.
We achieve this level of performance, reliability,
schema enforcement, and scale,
through advances in Delta Lake and Delta Engine.
Delta Lake is an open format storage layer
built on top of parquet,
which adds asset transactions to your cloud data lake.
Let's show you how the transaction log enables
Delta Lake time travel.
Here, we're looking at a series of loan risk scores
based on where our property is located.
When we originally created this data set in version zero
on the left-hand side, we didn't have any data for Iowa.
We didn't have any loan applications there.
But as time went on and we reached version 40 of this table,
you can see a score was added for Iowa.
Now, let's show you the SQL that powers these queries.
(upbeat music)
You can see here in the SQL statement,
we are querying the table loan by state,
and specifying version as of zero.
This is how we use the Delta Lake time travel feature
in order to find data at a particular point and time.
Now, what if you've just started filling your datalake?
Well, the Databricks ingest feature allows you
to easily load data into your lakehouse to enable BI and ML.
So it's really easy to get started today.
Well I hope you've seen how simple and powerful
Databricks can be, for your entire data team.
Whether data analysts, data engineers, or data scientists,
they can collaborate together to do their data
plus AI, on Databricks.
learn more at databricks.com.