(upbeat music)
- Welcome to a new episode
of the Machine Learning Essentials series,
I'm Seth Juarez.
In prior videos we've seen how Azure, and in particular,
Azure Machine Learning, is so well equipped
to handle our ML workloads.
Right from he tools and IDEs we need
to the frameworks we're familiar with,
through to deployment of the models,
everything is seamlessly provided in a single workspace.
But is Azure ML the only way to do ML on Azure?
What if your organization has made a big bet
on another engine, or has experts who want
to leverage their choice of environment?
Well, Azure is not at all opinionated,
and provides you with all the flexibility of choice.
Machine Learning on Azure can happen anywhere:
within Azure SQL DB, Azure Synapse Analytics,
or a very popular Apache Spark-based service on Azure
called Azure Databricks.
In this video, we'll take a look at why
and how you can leverage Azure Databricks for ML.
Now, some of you must be wondering about Apache Spark.
For those of you who are not aware of Apache Spark,
it is the most popular open source engine
for distributed data processing.
It leverages compute clusters, scalable to hundreds
or thousands of nodes, to process large amounts
of structured and unstructured data in an efficient way,
making it a compelling choice for both big data processing
for analytics, and complex iterative processing
associated with machine learning.
Well, if you ask a data scientist
to describe their typical tasks, you will notice
that they spend about 70% of their time cleaning,
curating and processing data to make it ready
and relevant for the subsequent modeling process.
This is important to understand.
If you're spending so much time dealing
with data pre-processing, you might as well use an engine
which provides you with the most comprehensive set
of capabilities to do it in a scalable way.
Today, Spark is the most popular and feature-rich engine
to process any data at scale.
Further, what is machine learning?
Well, at its core, it is basically an iterative process
to apply linear algebra and calculus-based equations
on the data to understand correlation, causality, and such.
So basically you're making a function with math,
which means there is a lot of data processing
that is happening, and it could take advantage
of sophisticated distributed processing.
That is exactly what Spark brings to the table.
Spark also offers a native ML library called MLLib,
and its contributors collaborate in the OSS space
to enable other popular frameworks like Tensorflow, PyTorch,
Scikit-Learn to execute in a scalable
and distributed manner.
Hopefully, now you understand why
there is such a big community for ML practitioners
on Spark, and such pace of innovation
in the Spark space for ML.
Bringing all this innovation in Spark
and extending it's ability in a single unified service
that provides the right processes and tools for ML,
is what Azure Databricks is all about.
Azure Databricks is built by Databricks,
who are the founders of Spark and contribute a bunch of code
to the Apache Spark project.
Azure Databricks uses Spark as its core,
but builds even more performance optimizations
and ML workflow streamlining with managed experiences
for Delta Lake and MLFlow.
These are some more projects that Databricks
has open-sourced recently and are now growing exponentially.
Another innovation Databricks brings
to ML data pre-processing is Koalas.
For those not aware of Koalas,
it is a data processing library designed around the syntax
of another very popular library called Pandas.
Pandas is by far the most widely used Python library
for data processing.
It is extremely useful but it has one shortcoming,
it is designed for small amounts of data, or should we say
it is designed to leverage only scale-up architecture.
To overcome the single node execution of Pandas
and enable a scale out architecture for processing,
data scientists and/or data engineers
would rewrite the data processing in native Spark.
Koalas eliminates the need to do that.
You can use native Pandas syntax and still be able
to scale out your data processing.
This means data scientists and data engineers
can now be more productive.
All this comes natively as part of Azure Databricks.
As we look at creating ML models in Azure Databricks,
you will also notice that they've introduced the concepts
of ML projects.
This ties deeply into the ML workflow process
and integrates with MLFlow seamlessly.
In addition to that, Azure Databricks
offers a rich collaborative notebooks experience
with co-editing capabilities, review/commenting functions,
and GitHub integration.
This means your data science teams can work seamlessly
on projects and complement each others' skillsets.
Azure Databricks not only makes the job
of the data scientists easy, it also packs a serious punch
in the underlying compute it provides.
You can bring your choice of compute VMs
as nodes for the cluster.
It enables both CPU and GPU VMs,
providing specialized Spark runtimes
to help leverage the GPU distribution even more effectively
to provide optimal RoI.
Talking about ML in Azure Databricks cannot be complete
if we don't mention MLFlow.
MLFlow is a managed ML workflow experience
in Azure Databricks.
It has quickly become a very popular
OSS ML workflow management tool.
It has over 2 million downloads every month,
and has over 270 contributors.
This is huge for any OSS project.
MLFlow in Azure Databricks provides an effective way
to track your experiments, provision reproducibility
for your runs, package your models and environments,
manage these packages in a registry and deliver
end-to-end ML experience for Spark enthusiasts.
Microsoft is a core contributor to MLFlow
and has built integration between MLFlow and Azure ML
to allow data science teams to be able
to centralize all their ML management in a single place,
Azure ML.
You can choose to set your Mlflow backend as Azure ML,
enabling teams to continue to use MLFlow APIs
but register all artefacts in Azure ML.
This is particularly useful when large teams
want to collaborate using different engines in Azure,
for example, DASK, Spark, SQL, whatever.
All teams can use the tools and engines of their choice
and still keep their work centralized.
That is the flexibility of Azure.
As we've seen throughout this episode,
there is no reason for anyone wanting
to use Apache Spark as their preferred ML environment
to look any further than Azure Databricks.
With built-in capabilities to distribute your ML processing
and the backing of the latest in OSS innovation
like Delta Engine and MLFlow, Azure Databricks
is your one-stop shop for all things Spark on Azure.
Hope you got a glimpse of some of these capabilities
in this episode.
Looking forward to seeing you in the next one!
(upbeat music)