(dramatic music)
- [Instructor] Welcome to Databricks.
The Lakehouse is a simple and open data platform
for storing and managing all of your data
that supports all of your analytics and AI use cases.
It's where data scientists and data engineers
and analysts collaborate to prepare and analyze data,
build models, and deploy them to production.
Today, we will focus on the data scientists
who as you will see here
are trying to explain life expectancy
from health indicator data.
And along the way,
we'll see how our platform supports the whole life cycle
from data ingest to production.
The notebook based environment you see here is Databricks'.
You can edit, run, and share code
and documentation and visualizations and output
in many languages.
We will look at the problem in three parts:
data access and preparation,
modeling and interpretation,
and finally deployment.
We're going to only briefly review the work
of the data engineer and Databricks here.
Her goal is to make raw data usable.
This includes fixing errors in the input
or standardizing representations
and joining disparate data to produce tables
ready for use by modelers and analysts.
Here, the input are just CSB files
from the World Health Organization and World Bank,
and an additional drug overdose data set
that we included to explore as a factor.
And they can be read directly
as is from distributed storage
as Spark DataFrames.
Note that these data sources could just as easily
have been a SQL database or JSON files
or Parquet files and so on.
The schema is automatically read
or inferred in the case of CSV.
These files contain about 2,000 different health
and demographic features for 16 developed countries
over the last several decades.
The goal will be to learn
how these features predict life expectancy,
and which one or two are most important.
So the data engineers might prefer to work in SQL
and Scala as here.
Databricks supports these
in addition to Python and R
which the data scientists might prefer.
All may be used even within one notebook.
And along the way,
even the data engineers can query the data with SQL
and see the results with built-in visualizations,
like this one.
They might wanna take an early look at the data.
And we see that clearly
the trend in life expectancy between 2000 and 2016,
looks different for the USA.
It's low and declining,
and the question will be, why?
So at the end of the data engineering workflow,
the three data sources are joined on country and year,
and written as a registered Delta Lake table.
So to the data scientist
this representation doesn't make much direct difference.
It's just a table
that can be read like any other data source.
But Delta Lake provides transactional rights
and lets the data engineers update and fix data
from a bad feed,
or gracefully modify the schema
or enforce certain constraints on the data.
And that does matter to the data scientists.
Anyone who's faced modeling on dumps of data
from a database that might break
or change in subtle or silent ways,
will appreciate the real data science problems
that these factors can cause downstream,
and that Delta Lake helps fix.
Or, for example,
consider the need to recall exactly
what data was used to produce a model
for governance or reproducibility.
Managing the data as a Delta Lake table
allows the data scientists
to query the data as of the previous point in time,
and this helps with reproducibility of course.
Note that everyone can collaborate on this notebook,
perhaps leaving comments like this one.
So here we pick up from the world of the data scientist.
Her goal is to explore the data,
further enrich and refine it
for this specific analysis here,
and produce another table of featurized data
for modeling and production deployment.
So the collaboration began
simply read this table of data
that the data engineers produced.
This notebook uses Python
and the ecosystem may be more familiar
and useful for data science.
The Spark API is the same,
and the data tables are available
and exactly the same way however.
Now, you don't have to use just Spark and Databricks only.
For example,
here we dropped into pandas
to fill in some missing values
before returning to Spark.
You can define efficient UDFS
or user-defined functions
for Spark that leverage pandas too.
The Python ecosystem offers libraries
for visualization too.
And with Databricks, likewise, you can just use these.
Common libraries like Matplotlib and Seaborn
are already built into the ML Runtime,
and others can be added easily.
Here the data scientist uses Seaborne
to generate a pair plot of a few of the features
such as life expectancy,
literacy rate,
and opioid deaths per capita.
It can highlight correlations
such as that between per capita expenditure on healthcare
and GDP.
And it also reveals a clear outlier.
So the outlier here with respect to opioid deaths
is going to turn out to be the United States again.
After some additional featurization,
the data is written as another Delta Lake table.
Now, the data set is small enough
that it's possible to manipulate and model with tools
like XGBoost and pandas,
all of which are available already in the runtime.
But Spark will still be relevant in a moment.
The data scientist is here
developing modeling code using XGBoost.
It will regress life expectancy as a function
of the 1,000 or so features left in the input.
But building a model really means building hundreds
of them in order to discover the optimal settings
of the model's hyperparameters.
Typically,
a data scientist might perform a grid
or random search over these values
and wait hours while the platform crunches
through each of them serially.
Databricks however,
provides a Bayesian optimization framework
called Hyperopt in it's runtime,
which is one modern and efficient way
to perform the search in parallel among others.
So given a search space,
Hyperopt runs variations on the model
in parallel across a cluster,
learning as it goes
which settings to give increasingly better results.
Hyperopt can run these variations in parallel using Spark
even though the simple modeling here
doesn't need Spark itself.
This parallelism can dramatically lower the wall-clock time
that these hyper parameter searches take.
The results are automatically tracked using
MLflow in Databricks.
This gives a quick view of the modeling runs
that were created in this notebook,
and we can drill into the experiment view.
This is the MLflow tracking server,
and this experiment shows a more detailed overview
of these trials.
We can use this to search through runs
and even compare them as here.
For example,
we might wanna compare all the runs
using a parallel coordinate plot,
and in this way figure out which of the combinations
of hyperparameters seem to produce the best loss.
This detail view also shows, for example,
who created the model
and with what revision of the notebook,
when it was created,
and all the details of the hyperparameters including loss.
It also includes the model itself
along with feature importance plots
that the data scientist created.
The model can now be registered with the model registry
as the current staging candidate model for further analysis.
The model registry is one centralized repository
of logical models that are being managed by Databricks.
It manages artifacts and versions of this model,
and manages their promotion through staging to production.
This particular model has a few versions registered
as versions of the same logical model,
and they can exist in states like staging and production.
The model registry is also accessible
in the left navigation bar.
Instead of managing models as for example,
a list of coefficients written down in some file
or a pickle file stored on a shared drive,
the model registry puts a more formal workflow
around tracking not just the artifacts of the model
but which ones are ready
for which stage of production deployment.
So next,
a manager might review the model and the plots.
Here is a feature importance plot created by Sharp.
And it shows that the feature that most influences
predicted life expectancy is mortality from cancer,
diabetes, and heart disease.
Low mortality, in blue,
indicates higher life expectancy
and appears to explain plus one to maybe minus 1.5 years
of life expectancy.
And year is next most important,
which unsurprisingly captures many of the cumulative effects
of better health over time.
So notice that drug related deaths
do not appear to be a top explanatory feature overall,
other diseases still dominate.
So after reviewing the model and other plots,
the manager might finally approve this model for production.
The deployment engineer takes over here.
She loads the latest production model
from the model registry,
the one that has been approved for deployment
but MLflow automatically converts it
if desired to a Spark UDF or user-defined function.
This means it can be applied to featurize data
at scale with Spark, with just one line of code.
Now, this could work equally well in a batch scoring job
or with a streaming job as well.
So for a second
compare this to, for example,
handling the model or even just the coefficients
to a software engineer
to attempt to correctly reimplement as code
that could be run in production.
Here, the exact model that the data scientist created
is made available to production engineers
and nothing is lost in translation.
So in this production job for example,
the data is applied to inputs from 2017-2018
for which life expectancy figures are not known.
And this can be joined with data up to 2016
to complete the plot we saw earlier
showing the extrapolated trends.
Now notice that this could be registered
not only as a UDF in Python
but also in SQL as well.
So note that the predictions appear fairly flat,
but this is mostly because over half the feature data
is missing in later years,
and this model could also have been deployed as a REST API
or as a service in Amazon SageMaker or Azure ML.
So this has been a simple example
of how Databricks can power the whole life cycle
from data engineering through to data science and modeling,
and finally the MLOps tasks
like model management and deployment.