(gentle music)
- Hello, and welcome to another episode
of the Machine Learning Essential series
where we talk about some fundamental concepts
of machine learning in the cloud.
I'm Nishant Thacker, and in this video,
we'll talk about model deployment and inferencing,
but first, what do you mean by model deployment?
Well, once data scientists create machine learning models,
these models need to be put to use.
Now, a crude way to do that would be to rewrite the model
in the same language or framework your app
or your analytics solutions are designed in
and embed the model as a function
or a sub-routine inside of it.
Although it is possible,
but would require a complete rewrite
every time a new version of the model is made available.
The more scalable, and definitely the more preferred way
is to package the model
and host it as a web service end point
to be called as needed.
This call to the web service end point
is what we refer to as an inference call.
Now, what this allows us to do
is to create models and frameworks
or languages that are best suited for these
and keep a microservice architecture
for scale and flexibility.
Now, there are a few concepts to understand
for scalable deployment and inferencing.
First, inference modes,
second, deployment topologies,
third, deployment infrastructure,
and forth, inference optimization options.
Let's understand these one by one.
First, the inference modes.
When we talk about making inference calls to a model,
what we are really doing is sending some data
for the model to analyze and send back as a response,
usually in the form of a prediction classification
or segmentation exercise.
We can send these requests to the model in two forms:
either send a request for every data point
or tuple to get a response for each one
or send a batch of data tuples to the model
and the system can score and store them in a data store.
These are exactly the two inference modes.
The first one where we sent individual tuples
as inference requests is used mostly in scenarios
where the model is warranted to respond in real time,
and thus, we call it real-time inferencing.
And second, where we send a pointer
to the location of the batch of data
and expect to see a score data set as output
once the entire batch is analyzed and inferenced upon.
Since it's a batch being inferenced upon,
we call it batch inferencing.
There are different applications
and as a result considerations
for each of these inferencing modes.
real-time inferencing, for example,
is used in scenarios where low-latency is extremely crucial.
Either a user or an automated system at the other end
is waiting to make a decision
based on the inference output,
thus it is important to design a deployment
that can support very low latency.
Examples of these include real-time personalization,
predictive maintenance, and many more.
On the other hand, for batch inferencing,
latency may not be as important,
but these could involve heavier pre-processing on the data
and more time-consuming model processing requirements,
thus the need for these are more towards automation
and orchestration.
Examples of these include: sales forecasting,
customer churn prediction, and many more.
Now let's talk about deployment topologies.
For models to be deployed as an endpoint,
you need the following components:
first, you need an environment.
This environment encapsulates the dependencies required
to run your model for inference.
Second, you need scoring code.
This is a script that accepts requests, scores the requests
by using the model and returns the results.
Third, you need inference configurations.
The inference configuration specifies the environment,
the entry script
and other components needed to run the model as a service.
These are all placed into a base container image
which becomes the execution environment for the model.
The container image has a load-balanced HTTP endpoint
that receives scoring requests
that are sent to the web service.
Now, this process can happen in three different ways.
First, we can use no-code deployment.
They use UI capabilities to let Azure Machine Learning
containerize the model with all of its dependencies
and deploy it for you.
This are mostly used when you deploy models
created using Azure Machine Learning designer capabilities
or the automated machine learning capabilities
with an Azure ML.
The second way to do this is to use system images.
They leverage the Azure Machine Learning SDK
to utilize a base container
provided by Azure Machine Learning
to deploy the model into.
This base container comes with the most common dependencies
like Conda environments, the latest frameworks, et cetera
to make containerization process much simpler and faster.
This is used when deploying models
created in Azure Machine Learning
using the code-first options like notebooks and VS code.
The third is you can bring your own containers.
This is for more advanced deployments
where you're maybe using custom frameworks
developed in-house
or need to maintain tighter control over component versions.
In this topology, you create your own custom base image
with all the software
and dependencies required by your model.
Now, depending on your needs and expertise,
you can deploy models
using any of the container topologies above.
The next question that arises
is where do you deploy these containers?
That is what we'll discuss
as part of deployment infrastructure.
Again, depending on the stage of deployment,
you can choose from a variety of options.
First, you can have a local web service.
This are mostly used while you're testing
and troubleshooting your deployment packages
on your local machines.
Second, you can have Azure Container Instances.
Again, mostly recommended for a dev test environment.
This is used to validate the containerized deployments
before moving them into full scale production.
It is mostly for low scales CPU-only deployments.
Third, the Azure Kubernetes Service or AKS.
This is the preferred deployment environment
for production workloads that require real-time inferencing.
Given it's a fully managed Kubernetes environment,
it's in scale to your needs from tens to thousands of nodes
for very high queries per second requirements.
It supports both CPU and GPU-based deployments
offering even more flexibility
for low latency response times.
And the fourth
is the Azure Machine Learning compute clusters.
Yes, these are the same clusters we discussed
in the model training video two.
These can also be used to deploy models.
This option is only for batch inferencing needs.
It supports both CPU and GPU-based deployments
with an option for low-priority VMs.
As we discuss various compute environments,
we should rest assured
that all production-ready compute environments
like the Azure Kubernetes Service
and the Azure Machine Learning compute clusters
come with enterprise grade monitoring,
security and governance.
These support both key-based and token-based authentication
for secure inferencing requests.
These can be configured inside VNets if needed.
And these integrate with Azure app insights
for granular information
about what's happening inside these environments.
In addition to the ones above, Azure Machine Learning
is also previewing a new managed endpoint.
This is designed to take away any administration
and upkeeping of the AKS
or the Azure Machine Learning compute clusters
even for production workloads.
It can work for both real-time inferencing workloads
with online endpoints
and batch inferencing workloads with batch endpoints.
With managed endpoints,
you just provide your deployment configuration
like the model part and version
alongside the compute instance definition
in the form of a YAML file.
And Azure Machine Learning
takes care of setting up the compute,
deploying the model with all of its dependencies specified
and maintaining this compute environment on your behalf
with auto-scaling capabilities built in.
These managed endpoints also support managed identities,
whether they are system-managed or user-managed,
stored in key-vault or blob storage,
managed endpoints can recognize them
and authorize inference requests appropriately.
In due course,
these would also support VNets with private IP
so that customers can completely log these end-points
to their VNets.
Please explore these managed endpoints
and share any feedback you may have with us.
We've talked a lot about inferencing,
but haven't yet explored
what optimization options are available.
Optimizations can happen at two levels.
First, at the model and frameworks level
and second at the hardware level.
To understand optimization,
it is important to first understand
the baseline deployment architecture itself.
For example, a deployment to Azure Kubernetes Service
or Azure Container Instance, by default,
users are single-threaded general purpose web framework.
Multiple Flask worker nodes are started
to concurrently handle incoming requests.
These workers handle preprocessing tasks like featurization,
calling the model.run method,
and then any post-processing tasks
like aggregation, et cetera.
And these are framework-agnostic.
To optimize these inference requests
with GPU-based hardware, one can use Triton.
Triton is a framework that is optimized for inferencing.
It provides better utilization of GPUs
and more cost-effective inferencing.
On the server side, it batches incoming requests
and submits these as batches for inference
instead of the individual tuples.
Batching better utilizes GPU resources
and is a key part of Triton's performance.
With Triton, the inference configuration deployment
looks like this.
Multiple workers are started
to concurrently handle incoming requests.
The requests are forwarded to a Triton server.
Triton processes requests
and batches to maximize GPU utilization.
And all this happens
with no change required on the client site
as a client still uses a same Azure ML scoring URI
to make these requests.
Here's a pro-tip.
Triton should be a no-brainer
for high frequency inferencing workloads.
It will help you take advantage of GPUs better
and automatically improve throughput of your deployment.
Further optimizing machine learning models on framework
or the model side is difficult
since you need to tune the model and the inference libraries
to make the most of the hardware capabilities.
The problem becomes extremely hard
if you want to get optimal performance
on different kinds of platforms like cloud and the edge,
CPU or GPU-based architectures, et cetera.
Since each one has different capabilities
and characteristics, the complexity increases even further
if you have models from a variety of frameworks
that need to run on a variety of platform.
It's very time-consuming
to optimize all the different combinations of frameworks
and hardware.
This is where ONNX comes in.
Microsoft and a community of partners created ONNX,
which is an open standard
for representing machine learning models.
Models from many frameworks including PyTorch,
TensorFlow, SciKit-Learn, Keras, Chainer, MXNet, MATLAB,
and SparkML can be exported
or converted to the standard ONNX format.
Once the models are in the ONNX format,
they can rerun on a variety of platforms and devices
with the ONNX Runtime.
This ONNX Runtime is a high-performance inference engine
for deploying ONNX models to production.
It's optimized for both the cloud and the edge
and works on Linux, Windows and Mac.
ONNX Runtime supports both DNSs and traditional ML models
and integrate with accelerators on different hardware,
such as TensorRT on NVIDIA GPUs,
OpenVINO on Intel processors, DirectML on Windows
and many more.
By using ONNX Runtime, you can benefit
from extensive production grade optimizations,
testing and ongoing improvements.
Here's a pro tip.
If you're looking at deploying models to IoT Edge,
ONNX comes in handy with its ecosystem of device partners
to enable ONNX Runtime compatibility.
Well, this went really deep, really quickly.
Let's recap.
We discussed two inference modes: real-time and batch,
both their specific needs and uses.
Then we discussed three deployment topologies:
no-code based deployment, system image-based deployment,
and bring your own container-based deployment.
We also discussed compute targets: local compute
Azure Container Instances, Azure Kubernetes Service,
Azure Machine Learning compute clusters,
and the newly introduced managed endpoints.
Finally, we looked at optimization options
like Triton and ONNX.
I hope this gave you a clear view
of the deployment landscape
for machine learning models at scale.
To try these out,
please spin up a free Azure Machine Learning workspace
at aka.ms/aml-trial
and download a 30-day learning journey
from the Data Scientists resources page
at aka.ms/data-scientists.
That's it for this video.
See you in another episode
of the Machine Learning Essentials series.
Thank you.
(gentle music)