>> You're not going to want to miss this episode of The AI Show,
where Sethu talks all about a brand new feature in
Azure Machine Learning called
Managed Endpoints. Make sure you tune in.
[MUSIC]
>> Hello and welcome to this episode of The AI Show.
We're talking all about Managed Endpoints.
I've got a special guest. Sethu, why don't you introduce yourself,
tell us who you are and what you do my friend.
>> Hey, Seth. My name is Sethu Raman,
I'm a Product Manager in Azure ML,
focusing primarily in the [inaudible] side of things.
>> Fantastic.
Let's start fast with,
look, Software deployments in general are hard.
It's got to be even harder with Machine Learning and
with machine-learning based software,
what are you-all doing to help?
>> We've been hearing from a lot of
our customers who have been working with us,
and we'll be releasing this new capability
called Managed Online Endpoints
as a part of public review in Build,
which gives users the ability to bring their model and
deploy it across powerful GPU and CPU compute in Azure.
Primarily, I want to call out three things, Seth.
>> Okay.
>> First one is the managed infrastructure.
Users need not worry about creating a cluster,
upgrading it, and so on.
The system takes care of
the underlying patching of the software host,
upgrading the host images,
decoding them failure, all of this is handled for you.
The second thing is we spent a lot of time thinking about
the ops support for Machine Learning.
Here, we give users the ability to do
safe roll out of their models
using [inaudible] support for blue-green.
Deployment we give a built-in support
for Azure Monitor integration to give SLA monitoring,
and then we also integrate with
log analytics and app insights and more.
The third thing is debuggability.
One primary thing I want to call out here is
the local endpoint support that users would be able
to deploy and test and debug
iteratively the endpoint logic in their local docker container.
I want to talk about a new concept we are introducing here,
which is the endpoint and the deployments.
Here, they picks clients that
score want to do the invocation hit an endpoint,
and the endpoint has the SSL termination.
It has got the op, and it has got a traffic split logic there.
Users have an endpoints,
the developers deploy an endpoint
and then multiple deployments underneath this.
Deployment, you can have more than one.
In this example, you can see I've got a blue and a green.
Blue is running at different version of the model that's
running CPU machines and green is running on GPU machines,
you'll be able to do safe roll out using
the endpoint and the deployments and the managed endpoints.
>> This is really cool
because rolling out software safely is hard,
and being able to do with Machine Learning is even better.
Can you show us how it works?
>> Absolutely, Seth. Here, you can see I have two folders.
First of all, I'll tell you what I'm trying to do here.
I'm going to deploy a Machine Learning model
in GPU servers in Azure,
so that's what we're trying to do here.
You can see two folders here.
One is the GPT-2 model,
so in this case I'm using a GPT-2.
Machine Learning model is a popular model
for natural language generation.
I just flaunt it from hugging page.
It's a open source repo.
Then I've got an Azure directory
in which I have three files that you need for the deployment.
One is the score.py,
which has the logic for loading and storing the file.
Then you have your python dependencies in the command YAML,
and then finally you have the endpoint YAML here.
The endpoint YAML, just is a configuration for wiring up
all your inputs that are needed to deploy create the model,
so you have the endpoint later detail,
which is the name and ought and traffic
details and the deployment,
you can here only one deployment in blue,
you have model specific details, code details.
You're giving all these wiring of
these details for deployment, and finally,
you have GPU machine instance type.
It's clear you want to use for GPU.
Once you set this up, the only prerequisite to
create this endpoint is you need to have
Azure in My Workspace no other prerequisite.
I'm running this single command which is azml Endpoint
Create from my CMI and just passing this YAML file.
Once you do that, you will get endpoint,
which is going to be deployed in Azure in
the GPU nodes and you get a rest endpoint for scoring.
In order to save time,
I'm going to run these commands and I'm going to
show you how the experience works in hindsight.
>> Got it.
>> I've run this command here.
It takes a few minutes and you can see that,
hey, the whole thing is done,
the model is registered,
I've got an endpoint created,
now let's pull the endpoint details.
In order to score it, we need two details.
We need these coding URI and the auth details,
so I'm using a shell command here to pull the coding URI here.
Here, I can see this coding URI good,
and I have a command for pulling the auth, again,
credentials, and I have actually filled in the details
here is a rest client similarly to a postman but bit into VS code,
but you can use a rest client of your choice.
Here I have put the details of
the URI and the token information here.
This is the input [inaudible] and
I expect the Machine Learning model to auto
complete so is making an invocation right
now and executing in the GPU nodes,
and there is our response here.
It generated some interesting text here.
>> That's cool.
>> This shows you how you are able to deploy
a model into GPU and get an endpoint within a few steps,
so this is all well and good.
I also wanted to show you how users can debug this.
People who have tried to deploy models into
Cloud know how difficult it is to debug their models,
because whenever you want to make a change,
you want to wait for a few minutes,
it goes to the Cloud and deploy it.
Here what we have is a concept of local deployments.
In this, you are able to create any command that we run,
you are able to run that, and all you can do
is just do a dash dash local in the end.
In this case, we're doing a azml endpoint create,
and you have to do a dash dash local.
I think when I zoomed in,
some of the CLA got clipped here.
>> Yeah.
>> But I'll show you how it looks.
The same command that we did,
you just do a flag here dash dash local and you'll
be able to execute all of
these commands like az ml endpoint invoke,
and gate logs, all of them,
you'd be able to do, like debug locally in docker container.
>> This is so easy.
>> Yeah.
>> This isn't so hard to do.
>> Exactly. I want to show you
because the information got clipped.
I will show you how it is done. Here's what we do.
When you do this, you create
the same YAML that you used for production,
but you just do dash dash local,
so it's going to deploy in a local docker image.
>> That's really cool.
>> There's first part of the demo,
now let's go to the part that you asked with a safe roll out,
how can users do the safe roll out?
>> Let's hit Control plus just a couple of
times to make sure we can see the text.
>> Yes.
>> There you go.
>> Here in this part of the demo,
I'm going to show you the safe roll out capability.
You have a linear endpoint
and a deployment which is deployed in production.
Let's say one diploma named blue,
which is up and running.
Now, I want to roll out a new model.
New what's not the model, a more efficient version.
What I'm going to do is I'm going to
update the same input, same YAML,
and I'm going to add a new deployment called green here,
and here, if you look at my scoring code,
I am purposefully putting scoring with error.
I'm introducing a parsing error so that I can show you how
users would be able to catch this unlike roll out a new version.
>> Right.
>> When I want to roll this into production,
all I need to do is do a azml endpoint
Update and just pass the name of the YAML.
You can update it in the same YAML,
but for the sake of the demo,
I created another one here.
Once I do that, let's go to
the metrics Azure Monitor integration and see what happens.
Before that, we had zero percentage traffic to green,
so let's try and divert some percentage of traffic to green.
I'm saying it's 10 percent traffic to green to start with,
and I'm clicking "Update".
Once I update, now
green is going to take 10 percent light traffic.
Initially it took zero percent,
now it's taking 10 percent and let go and
see the metrics whether our roll out was successful.
Here you can see in the metrics page.
The first chart shows you that the traffic split
between blue and green is good, we have 10 percent,
90 percent like 115 requests per minute
in blue and 10 request per minute in green, looks good,
but if you look at the status code split that we have here,
we're preparing on green and
we're displaying the HTTP status code.
You can see in the bottom chart
all the input that came to green are error,
the HTTP status for x x.
That means something has gone wrong in our deployment,
so what we do is we pull our log analytics
and we can pin it to green and filter on a specific time,
and actually, I can see that there is an error here.
I can see in my score.py there are some exception thrown.
Looks like a parsing error, so I found my issue.
I am going back and I'm updating the same YAML,
but this time I'm going to use a scoring fix.
I fix that scoring file
grabbing the same score.py that's updated by the day,
and I found, an Update command again.
Here you will see the power of
how we declaratively able to update the YAML,
so this is like an aligned with
the principles of [inaudible] about
where all the changes go through,
get in a declarative way and everything is audited.
Now we've fared the update and let's
see if our roll out was successful this time.
This time you hadn't notice it's an in-place Update,
the first time we did a blue-green,
now it's an in-place Update
and it's a rolling update that happens in the back-end.
We go here, hey,
it looks like all the red has gone down,
so basically all the HTTP status code 400 as going down
after we passed the fakes and all the HTTP 200 is rising up.
This basically means that our fix worked,
so all success is after the fix is done,
so this is great news.
We can go here back and Update.
We can give more traffic here,
50 percent traffic, and we can see what happens,
whether we get the CLA,
you'll be able to do that using the CLA escrow.
We are now going to see in the screen
how the 50 percent traffic looks like.
You will see that blue is coming down and green is popping up.
>> As that's coming up,
I just want people to understand because we're going fast.
Basically, you're able with the new managed points,
you're able to create deployment slots with different models.
He showed one that broke everything,
but it didn't break us because
only 10 percent of the traffic went
there and he was able to simply Update it, which is really cool.
Show us what you got right here, my friend.
>> Yeah, absolutely, Seth. Here you can see
that now after green is taking 50 percent,
you can see the traffic blue and green are like near 50-50,
so this looks good,
our traffic update worked.
We can see the latency that blue is taking 95 milliseconds and
the new efficient model just taking
23 milliseconds are p90 latency.
Our roll out is successful,
our model is efficient as we expect, so great.
Then we can of course, roll out
the 100 percent traffic and we are ready to delete blue.
This shows the journey on how user starts from playing
locally with the docker to make sure
all the core and the dependency is split.
Then move to the Cloud,
in the first version monitor SLAs,
and then going with the other second version with
a safe roll out and able
to debug any issues
and move forward and phase out the old version.
>> That was cool.
I know we went by really fast,
but that journey is super important because before
people would deploy a Flask app and everything would break.
Here, you can manage it safely with
a deployment with 10 percent and then it breaks,
you switch it out and then you move stuff up.
This is really cool, my friend.
Where can people go to find out more?
>> There is a link that's right there displayed.
People can go there and here users would be able to run
a hands-on exercise from starting to end.
Users would be able to experience this themselves.
>> Awesome Sethu. This has been amazing.
You've been learning all about Managed Endpoints.
Here on The AI Show thank you so much for
watching and hopefully we'll see you next time. Take care.
[MUSIC].