- [Presenter] Our next speakers are Yong-Gang Cao
and Ji-Wei Cao from Outreach, a platform that encodes
and automates sales activities into workflows.
Continuous integration and continuous deployment are widely
adopted software engineering practices.
Our next speakers describe how
they approach CICD in the context of ML.
Please welcome Yong-Gang and Ji-Wei from Outreach.
- [Ji-Wei] Hello everyone and thanks for watching.
Hope for everyone is doing great staying at home.
I'm Yong-Gang.
In this talk Ji-Wei and I will introduce
our CSD work in Outreach for ML series.
We will start with a brief overview of our company
and our team and Ji-Wei will lead us
there into our CICD pipeline
with some details and then I will bring up
some special consideration we did,
and finally we will conclude with some future work.
Outreach is a sales engagement platform
which is powering more than 3000 customers.
Sales engagement platform, in a nutshell,
is to enhance sales people's sales tools
like a sales force and phone call and email
and account management system and more.
So we organize all those sales activities into sequence,
as you can see.
So our work is basically to bring the automation
and the intelligence into activities of each sequence
to make the salespeople more efficient and more effective.
This science arch is illustrated by this fly wheel.
So we start and stop with data,
which we call as a situation here.
And we collect till relationships between, email,
phone call, calendar in white et cetera,
as knowledge assets, and then we build models
and apply the models
as the insights or give assistant to salespeople.
And then we gather the usage data from the salespeople
and then we feedback to detect more situations.
So as you can see, for this part we want to
speed up this cycle by building a more solid platform CICD
is one important part of it.
Next QA we'll dive more into the CICD pipeline
which is powering our MLOP birth model.
Ji-Wei?
- [Ji-Wei] Yeah.
So first, MLflow is open source too
to manage the ML lifecycle including experimentation,
reproducibility, deployment and a central repolitory.
So in Outreach, we use a lot.
So we're leveraging help of the ML products
to standardize the source code as the GitHub
of each ML product, we use Databricks Runtime
to train a lot motors to MLflow experiments.
After the motor has been trained and validated,
we want to deploy MLflow motor into several locations
like Kubernetes, Amazon SageMaker and Azure.
So I wanna talk about how we structure
our MLProject in practice.
So here is a basic example for MLProject.
First, MLProject is the format introduced by MLflow
to describe steps and links to the conda environment.
Each MLProject contains a MLProject file which identifies
the dependencies, entry points and parameters.
So all the dependencies are actually specified
in the conda of the ML file.
So our dependency include both public packages like MLflow,
scikit-learn, cachetools, as well as internal packages.
So we use MLflow API to trigger Databricks Runtime
to run training.
So we need to properly enable Databricks
to read internal packages.
So the next slides I'm gonna talk about
how we handle credentials like Databricks token,
GitHub token especially we don't want to put any tokens
into docker image or logged conda diana file.
So here, as example, all the packages start with a ds,
I actually internal-packaged.
I explain how we handle that in the next slide.
So we come up with a solution to have a conda template file
which contains a GitHub token as a mask.
So when we're at the Runtime, we build a docker image
to deploy, so we read the token from Enviro Varavos
and I get GitHub internal dependency.
We build a pack of wheels and put into a docker image,
and in conda file for deployment
we actually put it referenced to will files
in the docker image.
So you can see from the slides here
now it's all the dock image and also the conda file
which contains internal references.
We can deploy to the ML model's to endpoints like SageMaker.
So next I'm going to talk about the continuous
integration and delivery of ML service.
There's a very boundary between CICD
so we set the boundary with the CI to have the docker image
and MLflow logged model's ready for deployment.
So for every checking from internal dependencies,
we run integration test for a small sample of data
including, and run all the steps including model building,
MLflow logging, docket image-building,
push, and also deployment test.
So we use CircleCI in Outreach and we trigger a full
model-building before the actual deployment.
So after model-building,
so we signal deployment pipeline to start.
So ML service deployment is a combination models
and the service configurations.
So we define a deployable version to include ML models
and docker image and service configurations
like AWS region, instance count, instance type
and also deploym version contains actual code
for doing deployment such as the code for pipeline,
the code for deep ping, regression test and load test.
So we also develop mechanism to promotion
or rollback strategy which also use the MLflow registry
and s3 to track all the model and image urls.
- [Yong-Gang] Thanks Ji-Wei.
As you can see , we have many steps in the CICD pathna.
Here I want to highlight some of the special considerations.
One thing we do is that we actually transform
our pytorch model into scarekiner model format.
I introduce a wrapper, actually we can do more than that
in the wrapper, we customize our interface
by embedding the metrics, some logging information,
check information into it,
as well as we added the caching into the wrapping step.
As you see, we do sampling in the CI step,
actually we are using stratified sampling for the CI
to have the full carriage for all the labels.
And we did parameter passing for both CI and CD.
In the second slide we used a local file
base the parameter passing a and the triggering
nd for the cd we use GitHub SHA
and sc file for the parameter passing and the triggering.
When we try to use MLflow model registry
we find that actually the registry itself
don't contain slots for actual metadata
such as the humegiohorse.
And then we figured out to use the wrong ID.
Basically you can see from here there is a link
to the MLflow run which can contain
whatever metadata you want to put into.
So we use this to associate the model
with additional metadata.
In terms of tests, we use mock, mock the test
for doing those interface tests to verify interface
and there's a cache in logic, as I mentioned.
Without any real model file or implant,
as you can see from this screenshot,
it could be a very simple fake model.
And we do have our additional step as Ji-Wei mentioned
in cd, we call it the regression test,
which is after the stage in deployment
and before we promote to prod we want to gather
against the endpoint and evaluate it
both its quality and there's latency.
Then we collected the telemetrics
from the applications side.
It's kind of like the clickstream but some actions
we can turn it into online test, for example,
if people change our predicted class into something else
that could be our false positive of the original class
and false negative for the chosen class.
So this is kind of a online test we did.
Of course, last but not least, we need to deal with
all kinds of unnoticed traps, some different tools.
For example, MLflow, COI and don't support SynCOR
if you lie it locally and trigger a dead break job,
it will return immediately.
And SageMaker cannot recover from failed deployment.
You have to delete the endpoint before you can deploy again.
CircleCi actually don't support Autocase base
the parameters for rest the core for some strange reasons.
So yeah, just some interesting finding.
- [Ji-Wei] I'll do a quick lab demo for our CICD tool here.
As you can see, this is where our project in the CircleCI.
Besides the unit tests, we have other steps
for training the model until the deployment.
As you can see in the training step we have two branches:
one is to build the full model and the other is
to use the 100 sample derive code
to build the sampled model.
So when we build the full model we adjust that
via the docker image and notify the cd
to deploy the full model.
And when we checking some new code, it will do a sample
of the model building and there it will actually push
to SageMaker directly, just in case anything goes wrong,
till the end of the deployment.
This is our concourse cd pipeline with that
with triggering the full model building, as I've shown
still happening in CircleCI, and CircleCI will notify the cd
when it's ready to be deployed so then the concourse
will take hour to deploy to staging
and then we will do regression test
against the staging implant.
We will do those, all kinds of metrics evaluations
in the regression test,
and after that, we will promote prod,
it automatically pushed to both US-east and US-west region.
We also support here the manual-driven rollback.
If you trigger this step, it will roll back
to the previous version for both us-east and the us-west.
So behind the scene, we actually is using the model registry
from MLflow in Databricks.
So you can see we started with triggering for version nine,
we deployed to a staging, and then later we deployed to prod
and then we rolled it back.
For this version, you can see the version eight
is current prod version.
In conclusion, Databricks and MLflow provided a great
environment and tuning for us to run and weave ML jobs
with the fantastic checking abilities.
We do need some customizations for our enterprise needs
and the integration with other tools, as we mentioned,
the security concerns and the wrapping logics et cetera.
In the future we are looking for either
to do more supporting and more automation
on the experimental environment,
for example, the hyperparameter tuning, feature engineering.
So those are still ahead of us.
We want to dive more into that.
And of course, we want to catch contribute back our tunings
of MLflow to the community
so that it can benefit all of us in the future.
Finally, we'd like to acknowledge Yong, Andrew
and LiangYu for the helps.
By the way, Yong and Andrew, they will present more details
about the ML model in the coming Spark AI commit.
And thanks Pavel, Yifei, John
and Databricks for the support.
We'd like to answer questions
in the question-answering session, and thanks a lot.
Bye.