you don't want to miss this episode of the AI Show where
Tracy Chen talks about batch endpoints stay tuned
hi Tracy hi Bea welcome welcome to the ash show i'm i'm super excited to have
you on the ai show as you know because i love your features so i'm really excited
about what you have to tell us today thank you i'm so excited to be here too awesome
uh well just to begin you know obviously i know who you are and i know what you do
but a lot of people out there may not so tell us who you are and what you do
sure so uh my name is tracy chen i'm the product manager for azure machine learning badge
all right so what is batch inference so yeah so uh batch inference and the newest release feature
called batch endpoint is a new experience we built for you to deploy a model in a turnkey manner
to run batch inference and basically the batch endpoints manage the computer resources for you
when you invoke the batch endpoint an offline job will be triggered the service auto bring up a
computer cluster to run batch friends workload for you and after the job is completed the computer
cluster is automatically taken down and you can find the inferences stored in the cloud storage
of your choice and the best thing out of it is you only pay the computer resources you actually use
all right so we have you talking about batch endpoints but people here may have heard of
online endpoints and i'm just wondering how they compare yeah a great question so these
are actually two different type of endpoints targeting for slightly different use cases
batch endpoint is for batch inference where the latency to get the inference is not critical
usually in minutes hours or even days while other endpoint is for real-time inference where it's
critical for you to get the influence right away like in milliseconds or seconds both share the
same concepts of endpoint and deployment but due to the nature of the different latest expectations
the invocation behavior is totally different when you invoke a batch endpoint you pass in
a data reference and it runs as asynchronous job to process the entire data at once computers only
provisioned when the job starts and the inferences are stored in cloud while for online endpoint
you pass in the data through the http request directly and the inference is sent back through
the response immediately compute used by online endpoint is provisioned as a time of deployment
and is always up and running to be ready anytime to receive a request so another slider difference
is batch endpoint only has one default deployment while online endpoint can split traffic among
several deployments okay so you're telling us that latency is really like the main difference there's
other differences but latency right is is really the key here that with online endpoints latency is
much shorter and batching points is much longer so why why would somebody choose an endpoint that has
like a longer latency yeah great question so there are actually several aspects
you can check whether batch endpoint fits your use case the first one is like you mentioned right
how soon you need the inference if your latency requirement is relaxed go with batch endpoints
and the second is how much computer power is actually needed and if cost is a big concern
you have the flexibility to use low end compute and run a bit longer if using batch endpoints
and the third is how often do you need to influence for example you may only need to get
inferences once every hour then you can accumulate the incoming new data to a certain amount
and invoke the endpoint one time when the data is ready and on maybe sometimes on time schedule
all right so there could be significant cost savings and if people have the right scenario
if they use battery points right yeah that would be amazing everybody likes to save money i like
to save money so these are in preview and before right okay so and then before about
10 points we had parallel step so some people watching uh this interview they may already
maybe they may be using parallel step today why should they switch to batch endpoints
yeah we do have a lot of users currently using parallel run step to run batch inference and
it was released last year primarily used through the python sdk we're providing while customers are
really happy about the experience the performance and the scale paragraphs that it provides
we also receive feedback that there might be a steep learning curve to use it for the first time
so basically you need to construct a pipeline at the paragraph step prepare the environment
write a scholarship create a data set publish a pipeline to get an
endpoint so it's a like quite a long journey but comparing to paragon step right batch endpoints
provide a command line interface-based experience which is more intuitive and easier to use also
there are fewer concepts you need to learn and basically you have your model ready wrap it with
computer resources and server configs in a yaml file and run simple commands and you are all set
nice that does seem a lot easier i mean having these pipelines and button points i do agree that
it's better quicker to just use button points um so so you said that like people
can use the v2 cli with batch endpoints right yeah but they can't use the python sdk is that correct
yeah no no for now no price sdk but you can use v2 coi or rest or ui okay so they do have like
three different options that they can use awesome all right so can we see a demo sure happy to show
uh so um so i'm gonna show you how to use a v2 coi to uh run batch inference using batch endpoints
the first step here is you need to create a batch endpoint the most convenient way to do
so is just to run this batch endpoint create command and provide a name of your endpoint
you can also write providing a simple yamo like this one and the key property here is the name of
the endpoint it is used to identify an endpoint and but and also like a requirement for this is
it needs to be unique across the entire region because if you like as i mentioned before we can
also use rest api to consume the endpoint and this is actually the scanner url right which will will
be used to trigger through the rest api and then here you can see it's identified by the endpoint
name plus the region so this endpoint name needs to be unique across the region and now the next
step is right once you have the endpoint ready the next step is create a batch deployment under the
endpoint it just created by running this command match deployment create giving the endpoint name
a yaml file defines the deployment and then you can add this set default parameter to make this
newly created deployment to the default one under the endpoint so uh let's see let me just so i was
just wanna add something here so people who have may have been using the this feature like
uh longer than two three weeks ago they were probably used to doing these two steps in one
right you could put the deploy the create the deployment and the endpoint in one step so i just
want to kind of make it clear if you've used this before forget what you've learned do this do what
tracy is showing you do it in two steps yeah yeah okay yeah uh maybe let's quickly look at right
what will be the yama definition looks like for the deployment so this is example yama i'm used
to run this command basically right you wrap the model it can be a local model or it can also be
a registered model and then this is the scoring script it tells us like how you actually use this
model through this script and then this is the environment definition to host the model and then
here is the computer resources where you want to run the actual batch influence workload and there
are also a bunch of optional advanced settings you can tune to get the best performance out of it
so after you run this command right and at add set default now you will have this this
deployment added to this endpoint is best friend here okay yeah then right it's like basically till
now with two command runs it's all set right you can directly invoke the batch endpoint
to trigger a job which is actually during the batch inference workload
so uh while you invoke the batch endpoint right you need to pass in a data reference pointing
to the input data for example here i'm porting to a folder containing ms mnist data with like
thousands of images and a good thing about this invocation that you can also adjust some of the
settings for example for this invocation i'm using more uh instance counters which is measuring like
how many uh computer nodes for this ex specific job run so i increase this number so that
i can speed up the entire processing for this specific batch difference job nice now uh after
the invocation right you will get a job name with from the response which is usually a good
then another very convenient command here is that you can just run html job show giving the name and
plus this magic parameter dash dash web okay get click it will magically open up
oh the portal it's a studio ui that you can use to uh monitor the job
that's cool yeah like this yeah yeah so the job may take some time to process so i'm opening up uh
already completed the job here so if you see this box marked as completed that means the
batch inference job is complete right then you can right click this box and navigate to access data
this is where as a storage location where we store all the batch inferences results
so i have a pre-opened the results here so basically you have this one file containing
all your inferences scores and you will like to put an input identifier here something like a
file name to correlate with the score you get for that file so this is a sample result you can see
nice and that file name um you have to do it yourself right you have to write the code for
that yourself in the score file right yes okay uh this is awesome i love it uh you have another demo
for us right yeah exactly so with this command line interface it makes you very easy to set up a
ci cd pipeline so my next demo is about how to set the crcd pipeline with those simple coi commands
so what i have here is an end-to-end ci cd pipeline uh allows you to train a model first
and then after the training is completed a new model is generated they will and then it will auto
kick off a new deployment on the existing batch endpoints so you can add like different triggers
here for example you can just run the workflow by simply triggers here or maybe you can add other
triggers like um submit a new pr so uh yeah this workflow will take some time so let me
quickly open the previous i run the workflow especially for the auto deployment one
so here right yeah after the training there will be a new model produced you can like register
model in your azure machine learning workspace or you can download it to your local file and then
what you would like to do here is you will want to create a new batch deployment host the new created
model but without setting the default yet imagine you already have uh batch endpoints up and running
in your production environments now you want to replace the existing model with a new version
right you want to properly test it before switch it to the default one so yeah uh first step right
create a new batch endpoint deployment under the same batch endpoint you have without setting as a
default then verify the deployment and make sure everything works as you expected so if everything
goes well then you can have the third step here which is switch to default default deployment so
as i mentioned earlier one batch endpoint can only have one defaulted deployment and that default
development will be used when you invoke the batch endpoints so after this step this new depo
the new deployment which hosting the new model is replacing your old deployment
now you can safely delete the older one and after running these four steps you will have the same
batch endpoint with the same identifier which is an endpoint name but now running a brand new model
nice i love it i love it i think people do too uh so where can people go to find out more about this
oh yeah sure so there are several useful uh links the first one is uh this concept doc
it means it talks about what is batch endpoint what is deployment what are the major features
we built for you and then the second one is the how-to document it basically goes through the
step-by-step guideline plus a working sample as just as shown in the first demo and then uh third
is a sample ci cd pipeline report that i built and it's exactly the second demo i showed awesome
this is great thank you so much tracy it was really great to have you on the show
thank you in this episode of the ai show you saw how to create a deployment how to create a batch
endpoint and how to invoke the batch endpoint you also saw that you can use github actions to
streamline your development process thank you for watching and we'll see you next time you