you're not going to want to miss this episode of the ai show where setu
ramon shows us all the wonders of man managed online endpoints in azure ml make sure you tune in
hello and welcome to this episode of the ai show where sethu ramen is going to tell
us some exciting updates from managed online endpoints in azure ml setu how are you doing
my friend why don't you introduce yourself hey sir uh doing great happy to be back on the show
um so my name is seth ramon i'm the pm product manager for the manager on london points in azure
ml fantastic so last time we talked a little bit about this feature can you get us caught up
on what the feature is and maybe what's happened since absolutely so quick recap on the feature
on the capability for your viewers so managed endpoints uh give users the ability to bring
their ml models and deploy it in scale across cpu and gpu machines in a turnkey way so that they
don't need to worry about the infrastructure the the system takes care of serving scaling securing
and monitoring your ml deployment so that's that's what the whole feature is about
and so today we are excited to um like so last week we launched a bunch of new features
and uh i want to share that today in the show uh first one is auto scaling so users now have the
ability to you know like right size their deployments and reduce cost by using both
schedule based and metrics based scaling so this has been one of the most asked features
from our customers so that that's something that we have released and we also have released ml
flow dip model format deployment so if you have a model for ml model format you can deploy it
uh like without any code we call no code deployment so that is supported we also support
now deploying auto ml models in a no code way in our management point so this is supported as well
and other than these we are also like lighting up two new features one is for
uh interactive debugging of models before you deploy it in production okay and then the second
part is like automated cicd safe rollout uh you know uh capability i mean this is a there's
a ton of things like some of them were super interesting to me well all of them are but one
that was most interesting to me was this notion of debugging so i i'm really interested in seeing
how this works can you can you show us a little bit about how one might go ahead and use some
of these features in an end-to-end solution uh absolutely seth so let me bring up my screen here
um all right so in this the user's the the user story that i want to show here is
somebody wants to deploy a new model into production without disrupting the the current
model deployment that is already running so so that's the user story of course the step one we
need to develop the when we talk about a new model it can be changed to the model itself or it can be
changed to the the code that you might have for feature engineering right your scoring code or
the environment so in this case i'm going to start with an example of changing the the feature code
and then how we launch a new model so let's start with the first part of it so here i'm going to so
i'm going to execute a cli command here you can see i'm saying hey azml online deployment
and create you've seen this before from the previous show and here we are specifying the
deployment configuration file that has pointers to the that has information on your where's your code
what's your model and the environment details and then the endpoint name so this is pretty standard
on how you create in the cloud now you have these two flags that we are passing when we say dash
dash local and dash dash vs4 debug what's going to happen is uh the whole setup in terms of the the
whole image is going to be downloaded your model is going to be downloaded to be debugged locally
so let me just show what happens now so when i click this it's going to create the environment
locally within it with the docker container here and it's going to launch vs code uh
that has access to all your code and all your dependencies so that's what's happening right
now so just a quick question what what kind of things are in the deployment.yaml file
so three things primarily i would say like it has your code you know where where your code
is what your code is it has your um configuration of the deployment itself like how many nodes what
is this queue you want how many nodes you want and then it has information on your model you
know which model you want to deploy what's the version and what's the environment details so
these are the things that are abstracted in the ammo so basically it's a description of
how this model needs to be run exactly okay okay cool and so when when you're building when you're
building these these models you mentioned that you don't there's this codeless way to do it
what are what kind of code do you have to bring if you're right if you're making your own model and
then when we talk about the the the um the uh ml flow models why is it that those ones are special
yeah awesome awesome so so let's take the the question first where what kind of if you want
to write code what kind of code you need to write so this is a sample code right so here you see we
have a score dot pi we expect two two functions one is the init function the other one is the
run function the init function is run once when the model is loaded when your in container starts
and then the run function is run every time you do a scoring operation so here for typically people
do cache their model into memory and stuff like that and here when the request comes in you will
apply feature engineering and undo stuff and return your response right so this is you hit
control plus on this so people can so they can be a little bit bigger there you go so okay now
i'm seeing it so the first function basically loads up the model because it's always just a
file and that only happens once at the beginning and then the run is every time there needs to be
inference that's that's what's happening you got it set yes and you've got a bunch of dependencies
in this simple example i have a score.thai but a lot of our users can have like tens or 50 whatever
hundreds of files dependencies here so that's all perfect so this is a way of doing things with code
right then you you have an option where you don't need to do code what we call no code deployment so
if you bring if you have ml flow uh a case you you can you the ml pro supports different flavors so
you can you can bring in like scikit learn pytar tensorflow all of these models you can bring this
in and it's packaged as an ml4 format and when you deploy this we we take care of uh the scoring
logic so that the user need not write any any code there similar for our auto ml case as well you
just have a model and when you register the model uh you you set the model format as ml flow and
then when you deploy it everything is just taken care because ml flow and auto ml have a notion
of like how the model was built and so they have notions on how the models can be executed but if
you want to build your own model you would write a script like this run script that we're looking at
right here all right so so what are we looking at right now so here we are looking at uh so what we
want to do as i was saying we want to i'm changing some some of the code and i want to release a new
version of the deployment without disrupting the current version to make sure everything is good
right so here let's assume that i have done some feature engineering code changes and let's
add something here to debug this so i i want to debug this and see if everything is okay so here
i'm just adding one extra statement here to say that hey model is loaded correctly model loaded
right and all i need to do is uh i just need to start the debugging here so everything is loaded
all when you executed the command like the html deployment create you just added a flag for local
vs code debug and that's all we did now it's automatically launched the vs code and i have
a breakpoint set and i started the debugger and you can see that the breakpoint is right
now here i can interactively debug this so uh so here i can see it's executed this
and i'm going to do this then you can actually see the model is loaded
uh so each and every line you can track the the logs here and you can put uh break points in any
of your nested code and you can just interactively deepen so that's interesting one of the problems
i always have is as i i always get these things to run locally but then it fails to run
in the cloud because i might be missing something how is this helping me with moving it to the cloud
yeah awesome question because i know you you know like in the initial testing phases you've been
working with us very closely so the whole idea is that's why you have this concept of local endpoint
it's downloading the the whole doc the whole environment is packaged as a docker so you you
when you test it locally you have the confidence that everything is working and uh and update it
and then test it again once everything is done then you deploy it in the remote remotely you
can expect that everything will work so you save that time that few minutes every time to deploy it
remotely and wait so this right here that you're debugging is running as if it were in the cloud
in the same environment the same docker container is that is that what this is doing right now yes
oh my goodness oh so i and see this is the thing where because it all started up so fast
and i was blathering with you and then we ran it i thought because i run stuff on my local
machine all the time but this is running inside of a container exactly the same way it would run
in in the cloud correctly is that correct exactly said and in fact in your deployment that the ammo
if you mentioned like hey this is my model and this is my environment all of those are
packaged and available for you to run here so yes oh my goodness so now i can with confidence
know that it's running number one and number two know that it will run correctly if there's
any errors i can suss them out before they get up into the cloud is that right absolutely absolutely
even even even after we do this though i there might be some weirdness when i deploy to the
cloud the one thing that i think is important is how does someone do this safely like i and
that's another challenge i think is we don't want to just replace whatever endpoint there is can you
talk a little bit about that yeah absolutely so if you recall the last time we discussed we have the
concept of like endpoint and deployments multiple deployments can be underneath an endpoint uh and
then i'm just going to show that what happens here so let's uh why don't we go to the next
steps and i show and should have i talk so here we're done with this debugging everything is good
now let's uh let's kind of push this in and see what happens so we we just need to commit this uh
let's push it to our gate i created a new feature branch to do this now i can just do git push
and now let's open the repo and merge this code and see how the whole thing is going to
work so let's see if this is done okay all our changes are pushed let's create a pull request
and merge this code
oops i don't watch it uh we can do it again go down and reopen the reopen
the pull request on in uh there you go reopen pull request and then the logic
so now what's going to happen is uh this has triggered because we must remain
uh this this will trigger uh a run here so you can see this is github actions this is a ci cd
uh engine very similar to what if you use azure devops or any other
git lab or any other cac engine it's very similar to that and now you see that since we've merged it
a github action has been kicked off so this is going to do a safe roll out
and you can see just started this to a step-by-step rollout i'm going to show you since
it's going to take a little bit of a time i wanted to show you something that is already run recently
okay so what has happened is it is going to create a new deployment under the endpoint so
an endpoint is already running in production with one deployment that is taking traffic
now we are creating a new deployment here that's what this step of the the github action of the ci
cd workflow does and next step you can see that we do after we roll out we do sanity checking
without even opening live traffic you can actually hit the deployment directly to see if things are
working and you can test this and now if we open uh 10 percent of the traffic light traffic that
is coming in to to this deployment and after some time we can validate so you can open this and you
can you can stay take this for five minutes ten minutes some some of our like users wait for a
day before they do kind of a roll out to make sure all this are good what do you mean by all
this in this specific example i have two cases i'm checking i'm checking whether there are no errors
uh when i open the traffic there's no error to the deployment that that means there is no two three x
axis or four x http codes or x s right it's only two x axis there and similarly we make sure the
latency is within x millisecond whatever your threshold is if these two are true assume these
are like your validation gates release gates then it goes to the next step it opens 50 traffic and
it can wait for arbitrary number of time arbitrary amount of time and then this is opened up
and similarly we check where do the validation and then we open 100 traffic and wait for whatever
time that needs to be done and then finally when we are super confident everything is good we can
delete the old deployment so that's kind of the the flow that a a safe roll out looks like and how
this is made possible is by integrating with our metrics so when i say latencies within the next
milliseconds i we validate here there is a github action here that people can reuse that goes and
checks that for this uh the metrics that we that are published are within in the azure monitor are
within x milliseconds so that's how we we might validate all of this uh set i mean this is really
cool it basically what you're doing is when you're checking code is it re-running the entire training
process and then producing a model is it did i see that right so it can be configured in many
ways uh so why don't i show you the um yeah why don't we show you the some of the uh if we go here
so it shows the flow very yeah if is it okay if i do a quick recap of what we're trying to what
we all saw yeah absolutely because i think i think like the the process showing it actually running
was really cool but i want to make sure it's crystal clear what is actually happening yeah
absolutely sir so this is the concept of endpoint and deployment right so you end point you have an
end point and endpoint is a abstraction where the authenticate the security is there ssl is
terminated and you have a stable uri for clients to reference and then you have you can have more
than one deployment underneath an end point so in this example you have like two and one is running
cpu another running three gpu skews and you can have arbitrary amount of traffic between these two
but like if you go below let's look at how the whole safe road concept works so we we have like
version n of a deployment currently running in production that's taking hundred percent of the
traffic and slowly and then we launch like v n plus one the new deployment in our case which
we just launched in the with when i changed the code it created a new version of the deployment
and i'm testing with zero percent traffic then we open more traffic more and more traffic
um and also there is documentation on how we do this like using tags to actually specify to track
which version of the which deployment is broad and which is like a release candidate so you
there's docs on this i would encourage people to go through but if you look at the safe route to
answer your question said you know how how we have kind of laid out the the pipelines in this repo
um this is the key pipeline right this is a safe rollout pipeline and it can be
triggered it can be triggered in multiple ways so one way is a training pipeline runs and the
training pipeline itself can be triggered when a schedule based or a data set is updated and so on
and then the training pipeline can register a model that can kick a safe rollout or you
register or you directly merge to main so in the in our case if you notice i did not change the
model at all i just changed the code maybe some feature engineering code is changing some code
optimization is done even this can be a new new deployment right so this is what the the example
we saw but in this repo we have examples where you can kick this off and you can see this as well
and the safe rollout itself you can see the steps that we saw we first create a deployment and
sanity test it then we roll out 10 traffic to it and more and more traffic right but the key thing
is you can see this approval gate at every step uh we have an automated approval in this case but
easily you can add a manual approval as well if you wish so uh yeah that's the story this is this
is really cool because like ignorantly i was like oh you changed some code and we got to redo the
model but the reality is that in some cases you're changing the inference code there's no need for
for model changes so this kind of pipeline these kinds of pipelines enable you to rapidly iterate
that way as well am i getting this a little bit better perfect yeah that's perfect understanding
well this is this has been awesome where can people go to find out more about these cool things
so we have like uh three i think three click uh three call to action links that are going to be
shared uh as a part of this so one of them is this um this repo the link to the repo people
share people can actually go and for this they don't need to even download the code or
create a python environment anything locally they just need to fork this repo and fill up
use some environment variables to you you know to um to point to their workspace and stuff and
they can get going and then there's one more blog that we're sharing as a part of this
ai show where people can go and read about all the new features that i talked about like auto scaling
mlfo integration and all of that autoimmune integration there are tutorials that people can
go through for each of these capabilities awesome well i put them up below and then if you want to
get started there's a link right down below online endpoints get started set to this has been amazing
uh thank you so much for being with us my friend oh thanks for having me said always glad awesome
well we've been learning all about managed online endpoints some of the new goodness
thank you so much thank you so much for watching and hopefully we'll see you next time take care
you