SPEAKER: Once your ML model has been trained
and is ready to make predictions,
you'll need to be able to serve it.
Kubeflow Serving or KFServing is a powerful open source tool
you can use to serve your models with the scale
and performance of Kubernetes.
Let's look at how KFServing gives you
an easy-to-use interface so you can serve models in production.
To see the documentation on KFServing,
check out the link below.
Kubernetes brings powerful features
to training and serving your ML models.
But it can be tricky to handle the configuration of servers,
auto scaling, networking, and health checks of Kubernetes
clusters to run your models in production.
KFServing aims to hide this complexity
to bring you a simple, pluggable, and complete story
for our production ML serving.
This includes everything from prediction
to pre-processing, post-processing,
and explainability.
It gives you standard interfaces for common ML frameworks
like TensorFlow, XGBoost, Scikit Learn, and PyTorch,
giving you a consistent experience
for models of all kinds.
Its interface is a YAML file, typically checked
into a Git repository.
Simply point your specification, or spec,
to a serialized model file in cloud storage,
and you get a live model at an HTTP end point.
On top of that, KFServing lets you
leverage cutting-edge production serving features,
like using GPUs, auto scaling, scaling to zero,
and Canary rollouts to your ML deployments.
KFServing is installed by default
as part of a Chipotle installation,
but you can also install it as a standalone using
a quick script.
Check out the link below to learn how.
Behind the scenes KFServing is built
using the extensibility of Kubernetes
and the layers that sit on top of it.
Istio is a service mesh layer that
takes care of the load balancing,
security, and monitoring of the micro services
that make up a Kubernetes cluster.
It enables KFServing to handle traffic routing and ingress
to your deployed model.
Knative is a layer that lets you deploy and manage
serverless workloads on Kubernetes.
It enables KFServing to handle networking and Canary rollouts
so you can safely make changes to your end point
without dropping any traffic.
And it even enables you to auto scale GPUs and TPUs.
For ML scientists, this means adding something
to the KFServing resource is as simple as using the KFServing
interface that's provided first class.
For example, when you want to tweak min-max replicas
or perform a canary test.
For ML ops engineers, this means applying policies
under the hood becomes much more manageable
with the layered Kubernetes architecture,
like when you want to apply security policies, failure,
recovery metrics, and monitoring.
Let's walk through an example interface.
While you can create a short YAML file
to quickly serve your model and get an end point,
there are many ways you can customize things
like authentication and Canary rollouts.
Here the min and max specs are a way
to control your provisioning of back end pods
to make sure that you don't drop requests.
And you can meet demand without overspending.
Behind the scenes, Knative handles routing and queuing
of traffic.
You can also specify resource requests like selecting GPUs.
And thanks to Knative's concurrency-based auto scalar,
you can effectively scale CPUs or GPUs.
By default, KFServing sets target concurrency to 1.
So if you load the service with five concurrent requests,
the auto scalar will scale up to five pods, and back down to one
when requests die down.
For most ML scientists, one of the biggest challenges
is how to deploy models in production
using Canary rollouts.
You want to be able to easily test and update
your data models to production without impacting users.
While traffic splitting to back ends is a challenge,
KFServing lets you slowly increment traffic
from an old model to an experimental model.
In this example, we're rolling out
a new version of an IRIS model.
As we slowly increment traffic percentage to our B2 model,
we can be confident that Canary isn't
going to disrupt our users.
Eventually, we'll move 100% of traffic to our V2 model,
then delete the original deployment,
allowing you to completely manage the lifecycle of data
models in production.
When it comes to processing data,
ML frameworks don't give you a place
to execute code that processes requests.
KFServing includes hooks so you can provide your own container
implementations to do pre and post processing.
You just need to add the hook in your KFServing interface,
specify a container image spec for your processing handler,
and KFServing automatically wires it up.
The hook called a transformer can
help you process requests, like if you want to translate a user
ID into a set of new features about the user.
These new features can then be fed into another hook called
an explainer that can perturb the features
and send prediction requests to them.
Perturbing features to see its effect on a model's output
is one way of explaining or shedding light on your models.
Finally, the predictor can do predictions on those requests,
post process them in the transformer,
and send the transformer requests back to you.
Together, these hooks give you a simple way
to serve an end-to-end ML model with explainability
at your fingertips.
KFServing gives you out-of-the-box standardized
model serving across frameworks.
It's built for production inferencing with features
like canarying, auto scaling, explainability,
and a clean interface, making it easy
for you to review models and change config on the fly.
[MUSIC PLAYING]