you're not going to miss this episode of the ai show where we talk about high
performance serving with triton inference server in azure mail with shivani make sure you tune in
hello and welcome to this episode of the ai show we're talking about high performance serving with
triton inference server in azure ml we've got a special guest shavani how you doing my friend
i'm doing good how about you fantastic why don't you tell us who you are and what you do
sure so hi everyone my name is siobhani i am pm on the azure machine learning team
um i work on model deployment and survey um and currently i'm focusing on how to best integrate
open source softwares with our product fantastic so let's talk about serving or
inference um why don't you describe what it is and then tell us about the challenges that that are
around that doing that yeah but definitely so model deployment and survey is when you have
completed you have trained your model you have a final model that you want to deploy as web
service send requests to it and get some output from the web service um so one of the um problems
that customers are facing into that they have to worry about maintaining different frameworks
they have to worry about and they are looking into a high performance inferencing solution
and they want this one uh inferencing server that could cater to all their needs all their use cases
and of course just bring model to azure ml and deploy them seamlessly
i mean it it sounds easy but our research shows that it's hard
very hard like building a model is so easy but then getting it to actually provide business value
is super hard how is azure helping folks do that definitely so as you know we have this feature
called managed online endpoints where customers don't have to maintain their infrastructure and
they get capabilities like safe rollout security monitoring and lot more and we are integrating
our managed online endpoint within media striter inferencing server so trita inferencing server
is an open source software uh which supports multiple frameworks like tensorflow denser rt
pytorch onyx a lot more it is a high performance server inferencing server so triton runs model
concurrently on gpus to maximize your throughput and utilization uh hence it gives our customers
best of fourth volt and you only have to focus on your machine learning model and deployment
and that's basically what we want to do we just once we build the model we just want to
put it out it sounds i mean it sounds harder than it should be can you show us how this works
definitely so uh can you see my coding screen we sure can okay so in azure machine learning when
you are deploying your managed online endpoint you need to have two yamas your endpoint ammo and your
deployment yaml so i have my endpoint yaml here name of my endpoint is triton multimodal send
point automotive we're putting as aml token and then i'll hit acml online endpoint create command
to create my endpoint now one endpoint can have multiple deployments i am adding one deployment
to this endpoint called blue and to this endpoint uh to this deployment actually i am passing
uh my model and if you can see i'm saying here sample multi models so i'm actually
passing multiple models to one end point uh so one of my model that here i have is by daf
uh model so it's question and answering model you give them some context you ask a question
it answers you and another is dense net onyx model which is a simple image classification model
so i'm passing this two different models to one end point uh a important parameter to note here
is model format and striping once you put as model format as triton we understand it's no
good deployment so by no good deployment we mean is you don't need a scoring script you don't need
an environment triton expects your model to be in particular model repository format which we'll
talk in few seconds and that's it uh you give your model endpoint deployment name the skew
and you hit acml online deployment create command all right so just a couple of questions as you're
as you're doing that the first question i have is so this is basically multi a multi-model
deployment that's doing question answering and image recognition some sort of image recognition
that's first question am i getting that right yep yeah so we have two different models here uh one
is bite f9 and another is dense nitronics model fantastic the second question i have
and this is something that i'm i'm used to doing um don't i usually have to write like
a scoring file with these models you mentioned that it's no code how is that possible yep so
uh usually when you deploy there is a file called init and run function which one loads your model
and another takes inputs and outputs to score your model but with triton when we are integrating we
auto generate that for you so you do not need your scoring script you do not need your environment
and uh actually in a bit we'll go to the logs and see that our server has started as modulus loaded
so really it becomes easy to just bring in your models to azure machine learning another question
i have is how long does it take to deploy these things are we talking like hours days
oh no definitely not uh it takes a uh for this deployment it took me under 10 minutes seven to
eight minutes to complete the deployment hence i have like an endpoint running so that we can go to
that and see how it looks awesome so two questions uh two final questions the two yaml files
so the managed endpoint yaml file basically describes like like a like an http style endpoint
and then you're you're adding deployments to that endpoint with the other one am i am i getting this
right perfect yep see look at that i knew i i knew a thing or two about that all right can you show
us what the the outcome of this is yeah definitely so once i hit my endpoint create and deploy my
create command i'm coming to this azure machine learning studio where you can visualize everything
very nicely um so i have my end point i have my blue deployment um the multiple models that i uh
deployed to this you can see your environment is azure ml triton server ubuntu inference and
cd environment it's a mouthful but basically we generated this environment for you we use this
node deployment underneath so you don't have to do it we load the influencing triton inferencing
server for you i see can we scroll down because i i i want to make sure i understand so basically
what's happening is there is a pre-built environment i'm assuming it's a it's an image
or a container of some kind that just basically knows about the way triton things are deployed
am i am i getting this right yep exactly so we can actually go and look into this environment
and hopefully my intern is working nicely and we can oh it's still loading cool
here we go yep let's yeah you see it directly pulls your image from triton server so there's no
it's not like we're inventing a new thing this is something nvidia already does we're literally just
pulling down their thing yep yes so you don't so as a new user you don't really have to know that
you need to use environment for deployment uh we do that for you we just pull that nvcr image for
you uh so it's ready for your inferencing awesome and so how would someone if they want to now
call this end point how would they do that definitely uh so now we have just before
we go that you see in my deployment logs i can actually see that my model and my server is ready
those uh once this is i can go and score against my different models so just to keep in mind before
we go to the endpoint is this is my model one name and this is my another model name and
when you want to score or let's just say that we want to do a get request to this model triton
multimodal inferencing.com this you can get from our studio so it's right here your rest endpoint
and next you put is v2 models your model name and versions and ones so this everything
remains same just this will change whatever your model name is you would replace that
and when i do a get request i see what input the model expects for me to score
and that's all basically the triton format of doing things with models yup exactly so there is
yeah one caveat with triton is it does expect your model to be in specific modern repository format
and they have very beautifully defined in their uh github repo of different uh a modern repository
format expected by uh different frameworks models so denser rt onyx right now we have onyx models
so we followed this uh torchscript tensorflow all the frameworks that they support okay cool
all right and so basically once you've once you've decided as an organization to use the triton way
of of doing things then it's literally just saying here's my endpoint here's the deployment
that i'm going to do and that's it am i i mean it feels like it's a little too easy
so yeah that that's the goal that you can do your high performance inferencing on gpus load multiple
models or single model depending on your use case and just make it easy for you to deployment i mean
that's cool is there anything that i'm missing anything else you want to show us
uh no that would be all uh just to see that how easy it is for you to uh
deploy this model so we can actually look at our model's ui screen and just yeah and you
see this is the model that i deployed samples multi model that's my model name um format you
can see triton it recognizes uh if you remember in our yaml we did put model format as triton
and when i come to the artifacts uh this is the model format that is expected by triton
my model name version model.onyx model modeling version model.tonics that's it i just have to
have my model repository in the expected format and again this is this is really cool like we've
been talking about it's really hard to get models into business value quickly this feels like that's
one of the ways to do it a powerful way where can people go to find out more definitely so we have
links here for our docs documentation and samples you can go and try out and see how easy it is for
you to deploy all right so there's a link for the docs here's the link for the there's a blog
as well that they told us about make sure you go check that out and the samples are all here this
has been amazing thank you so much for being with us shivani thank you so much for having me and
thank you so much for watching we're learning all about high performance serving with triton
inference server in azure ml thanks for watching and hopefully we'll see you next time take care
you