- Thank you and hello, and welcome to Databricks.
Databricks is a Unified Data Analytics Platform,
where data scientists and data engineers
and ML engineers and analysts,
collaborate to prepare and analyze data and build models,
and deploy them to production.
Today we'll focus on the data scientists,
who as you will see here,
are trying to explain life expectancy,
from health indicator data.
And along the way,
we'll see how our platform supports the whole life cycle
from data ingest, to production.
The notebook based environment you see here is Databricks.
You can edit, run and share code,
and documentation and visualizations,
and output in many languages.
The notebook can be attached to existing clusters,
or if necessary, a new one can be spun up.
Here we're just using an existing cluster
that we'll turn on, and scale up,
and scale down, and turn off when idle.
We will look at the problem in three parts.
Data access and preparation,
modeling and interpretation,
and finally, deployment.
We're going to only briefly review
the work of the data engineer in Databricks here.
Her goal is make raw data usable.
This includes fixing errors in the input,
or standardizing representations,
and joining disparate data to produce tables,
ready for use by modelers and analysts.
Here the input are just CSP files
from the World Health Organization,
and World Bank,
an additional drug overdose data set two,
which we included to explore as a factor.
And they can be read directly as is
from distributed storage as Spark DataFrames.
Note that these data sources
could as easily have been a SQL database,
or JSON files, or Parquet files and so on.
The schema is automatically read,
or inferred in the case of CSV.
These files contain about 2,000 different health
and demographic features, for 16 developed countries,
over the last several decades.
The goal will be to learn
how these features predict life expectancy.
Which one or two are most important.
So the data engineers might prefer
to work in SQL and Scala as here.
Databricks supports this, in addition to Python and R,
which the data scientists might prefer.
All may be used even within one notebook.
And along the way,
even the data engineers can query the data with SQL,
and see the results
with built-in visualizations like this one.
They might wanna take an early look at the data,
and we see that clearly the trend in life expectancy,
between 2000 and 2016 looks different, for the USA.
It's low and declining, and the question will be why.
So at the end of the data engineering workflow,
the three data sources are joined on country and year,
and written as a, registered as a delta table.
So to the data scientists this representation
doesn't make much direct difference.
It's just a table that can be read,
like any other data source.
But delta provides transactional rights,
and lets the data engineers update,
and fix data from a bad feed,
or gracefully modify the schema,
or enforce certain constraints on the data.
And that does matter, to data science.
Anyone who's face modeling on say,
dumps the data from a database,
that might break or change and settle
or silent way will appreciate
the real data science problems
that these factors can cause, downstream,
and the delta helps fix.
Or for example, consider the need to recall exactly,
what data was used to produce a model,
for governance, or reproduce ability.
Managing the data as a delta table,
allows that data scientist to query the data,
as of a previous point in time.
And this helps with reproducibility of course.
Note that everyone can collaborate,
on this notebook, perhaps leaving comments, like this one.
So, here now we pick up from the role,
of the data scientist.
His goal is to explore the data,
further enrich and refine it,
for the specific analysis here,
and produce another table of featurized data
for modeling and production deployment.
So the collaboration begins simply.
Read this table of data that the data engineers produced.
This notebook uses Python,
and the ecosystem may be more familiar,
and useful for data science.
The Spark API is the same, and the data tables
are available in exactly the same way, however,
now you don't have to use Spark in Databricks, just, only.
For example, here we drop to Pandas,
to fill in some missing values.
Before returning to Spark.
You can define efficient UDFs,
or user defined functions,
for Sparks that leverage pandas too.
The Python ecosystem offers libraries for visualization too,
and with Databricks, likewise,
you can just use these.
Common libraries, like Matplotlib and Seaborn
are already built into the ML Runtime.
And others can be added easily.
Here the data scientists uses Seaborn,
to generate a pairplot of a few other features,
such as life expectancy, literacy rate,
and opioid deaths per capita.
It can highlight correlations,
such as death between per capita expenditure
on healthcare and GDP,
and it also reveals a clear outlier.
So, the outlier here with respect to opioid deaths,
is going to turn out to be the United States, again.
After some additional featurization,
the data is written as another delta table.
Now the data set is small enough
but it's possible to manipulate and model
with tools like XGBoost and Pandas.
All of which are available already, in the runtime.
But Spark will still be relevant, in a moment.
The data scientists is,
here developing modeling code using XGBoost.
It will regress life expectancy,
as a function of the 1000 or so features left, in the input.
But building a model, really means building 100s of them
in order to discover the optimal settings
of the models hyperparameters.
Typically, a data scientist might perform a grid,
or random search over these values,
and wait hours while a platform
crunches through each of them serially.
Databricks however, provides
a Bayesian Optimization Framework, called HyperOpt,
in it's runtime, which is one modern,
and efficient way to perform the search,
in parallel among others.
So given a search space,
HyperOpt runs variations on the model
and parallel across a cluster,
learning as it goes,
which settings gives increasingly, better results.
HyperOpt can run these variations in parallel
using Spark, actually,
even though this simple modeling code
doesn't need Spark itself.
This parallelism can dramatically lower
the wall clock time that
these hyperparameters searches take.
The results are automatically tracked,
using MLflow in Databricks.
This gives a quick view, of the modeling runs
that were created in this notebook.
And we can drill into the experiment view.
This is the MLflow tracking server,
and this experiment shows a more detailed overview
of these trials.
We can use this to search runs
and even compare them, as here.
For example, we might wanna compare all of the runs
using a parallel coordinate plot.
And in this way figure out,
which of the combinations of hyperparameters,
seem to produce the best loss.
This detailed view also shows, for example,
who created the model,
and with what revision of the notebook,
when, and all the details of the hyperparameters
including loss.
It also includes, the model itself.
Along with feature important plots,
that the data scientist created.
The model can now be registered, with the model registry
as the current staging candidate model,
for further analysis.
(mouse clicking)
The model registry is one centralized repository
of logical models, that are being managed by Databricks.
It manages artifacts, and as versions of this model,
and manages promotion through, staging to production.
This particular model has a few versions
registered as, versions of the same logical model.
And they can exist in states, like staging and production.
The model registry is also accessible in the, left nav bar.
Instead of managing models as, for example,
a list of coefficients written down in some file,
or a pickle file stored on a shared drive,
the model registry puts a more formal workflow
around tracking not just the artifacts of a model,
but which ones are ready
for which stage of production deployment.
So, next, a manager,
might review the model and the plots.
Here's the feature important plot created by SHAP.
And it shows that the feature
that most influences predicted life expectancy,
is mortality from, cancer, diabetes and heart disease.
Low mortality, in blue, indicates higher, life expectancy
and appears to explain plus one to maybe minus 1.5 years
of life expectancy.
And here's next most important,
which unsurprisingly captures
many of the cumulative effects, of better health over time.
So notice that drug related tests
do not appear to be a top explanatory feature, overall.
Other diseases still, dominating.
So after reviewing the model, and other plots,
they might have finally approved this model, for production.
(mouse clicking)
The deployment engineer takes over here.
She loads the latest production model,
from the model registry,
the one that has been approved for deployment.
But MLflow automatically converts it, if desired,
to a Spark UDF, or user defined function.
This means it can be applied to featurized data,
at scale at Spark with just, one line of code.
Now, this could work equally well, in a batch scoring job,
or with a streaming job as well.
So, for a second, compare this to you, for example,
handing the model, or even just the coefficients
to a software engineer to attempt
to correctly re-implement as code,
that can be run in production.
Here the exact model that the data scientist created
is made available to production engineers.
And nothing is lost in translation.
So in this production job, for example,
the data is applied to, inputs from 2017 and 2018.
For which life expectancy figures are not known.
And this can be joined with data up to 2016,
to complete the plot we saw earlier.
Showing the extrapolated trends.
Now, notice that this could be registered
not only as a, UDF in Python, but also in SQL as well.
So note, that the predictions appear fairly flat,
but this is mostly because over half
the feature data's missing in later years.
And this model could've also been deployed as a rest API,
or, as a service in Amazon SageMaker or Azure ML.
So this has been a simple example of how Databricks,
can power the whole lifecycle,
from data engineering,
through to the science and modeling
and finally the MLOps tasks.
Like model management and deployment.
Thank you very much.