you're not going to want to miss this episode of the ai show where we talk about managed
online endpoints now generally available my good friend sat through make sure you tune in hello
and welcome to this episode of the ai show we're talking about managed online endpoints
now generally available as of microsoft build 2022 i've got my good friend set to all right my friend
tell us who you are what you do hey sir my name is seth ramon and i'm a product manager in azure ml
uh driving the managed and points capability fantastic now it's probably one of my favorite
features of all of azure machine learning because getting models into production is probably the
hardest part of all of this for those that don't know about this feature why don't you give us the
elevator pitch yeah absolutely so manage endpoints gives you a turnkey way to deploy your ml models
in scale at production without you having to worry about the underlying infrastructure
in your previous sessions that we had in the ai show we talked about how you can do interactive
debugging how you can configure auto scaling and how you can roll out a new version of the model
using your cacd pipelines yeah it's been really cool to be able to do that because not having to
worry about all that stuff is is awesome so tell us what's the new stuff as of microsoft built yeah
so i have some exciting stuff to share today other than that we are launching uh general availability
in build uh firstly i want to i brought a demo with me to show how users can scale to really
really high uh requests per second in a limit in a short amount of time with with uh you know
with capability of our github friendly apis i love it let me roll it right now in this demo
i'm going to show you how users can self-serve their way to 100 000 queries per second this
will unblock large customers who are reluctant to onboard because of scale concerns shall we
let's look at this setup starting with the basics we have a 32 core vm to run the road
test here is the managed online endpoint and deployment here each box represents a node
also called as an instance let's take a look at setup of one node it's a eight core machine
in this example you can see only one worker process containing the inferencing components
to increase cpu utilization we can increase the number of workers this is our test setup
since it's eight core machine we configured eight workers to increase the cpu utilization while not
affecting the latency we will use scikit-learn's diabetes model from our default example
creating this deployment using our github friendly cli is very straightforward first we configure the
deployment properties in the yaml file here we have the instance type you can see it's a 8 core
f series machine and instance count currently we just need one node and then the worker count that
we saw is eight now we fire the create command using the cli passing the yaml as the input
now let's run the load test using a popular tool called wrk it outputs the qps and tail latencies
let's see it in a better format in our onenote test we got a pqps of 1600 and tail latencies
of p50 to p90 between 4.7 to 6.2 milliseconds now let's increase the scale up to 50 instances
under our test it is as simple as running the update command with the instance count set as 50.
in five and a half minutes we are open for business typically this takes somewhere
between 5 to 7 minutes though for our purposes 50 nodes are enough let's do a quick sidebar on what
it takes to scale up to 150 nodes we use the same update command now with the instance count as 150
and in this case we are all set within seven minutes
i run the load tests on not only 50 instances but also on intermediate node configurations
let's compare the slide shows the output for different test configurations starting from
one node 10 node and all the way through to 50 nodes first let's compare the tail latencies for
a single node configuration you can see the p50 to p90 latencies are between 4.7 to 6.2 milliseconds
for 50 node cluster you can see it is from 5.1 to 8.2 milliseconds this seems to be reasonably
close given the scale you can also see the distribution of the qps across these various nodes
now let's see a plot of them here the x-axis shows the number of the nodes and the y axis has the qps
the plot shows near linear scale of qps with respect to nodes let's get back to our original
goal to achieve 100 kqps with the same cost that is with 50 nodes can we achieve 100 kqps
we can try this if we are willing to trade off increase in tail latencies we can tune for high
throughput by increasing the work account in this case we'll push the boundary by changing
the work account from 8 to 48 per node now let's run the test with a 50 node cluster with the
increased work account you can see that we have in fact reached 100 000 qps yay let's compare
you can see that the new configuration has better qps but we have to trade off with the increase in
tail latencies this is good enough for our use case however we can always easily add more nodes
to reduce the latencies to summarize high-skilled customers can now self-serve their way to hundreds
of thousands of qps so just to a couple of because that first of all that was awesome
100k queries per sqps means kilos per second yes uh in under seven minutes able to
to deploy that fantastic you mentioned tail latencies and there was like i think it was
p 50 p 60 p what what does that mean so when you want to measure the uh user experience uh
of the the client that's making the request you know the response time people you know average
is one way to do that but it does not capture these stragglers tail latencies for example p90
let's say six milliseconds that means that ninety percent of your users will experience uh latency
of six milliseconds or less oh i see i see so it's basically the latency for the percentage of people
the best percentage the the best latency for that fifty percent sixty percent did i get that right
that is perfect the other thing that was maybe go ahead yeah so when you for like uh applications
which have very tight sla they typically may like measure let's say p90 p90 p95 or sometimes even
p99 latencies that's what they care about i see okay this makes a lot more sense the second thing
that was not altogether clear for me was that the increasing worker count increasing node count
causing issues can you explain that a little bit yeah absolutely so the overall goal of this this
demo is to first of all showcase to users hey um you you are able to kind of really scale up to
two very high requests per second or queries per second uh by using configuration our declarative
api and on that our system is kind of open so when we in each worker let's say in our case we
have like a 8 cpu machine and we put initially we started 8 workers or like eight processors right
so these are the inference containers and eight processes that are running that will kind of serve
the request and um if we you know there is still you know a lot of time these processes wait they
are not using the cpu all the time so when they are eight okay they are doing good uh because we
match one cpu code to this indirectly but when i want to squeeze more juice out of the cpu
i can keep adding more processors but then there's going to be the trade-off is going to be
you're going to you're pro you're going to be able to handle more requests but because of the lot
of context switching that happens there is going to be additional latency so that's the trade-off
so if you have very low latency application you don't want to you know squeeze a lot of
workers but if you're willing to trade off and you won't reduce your cost then that's that's a
good way okay so i think i understand basically each worker if you're on an eight cpu machine
if there's eight workers then they're pseudo map one to cpu but then the more workers that you
have the more contention there is between context switching etc that's right fantastic are any other
cool things announcements uh from microsoft build that you want to talk about absolutely so i have
two new features that we are public reviewing as a part of management endpoints based on you know a
lot of customer asks so i'll give you like a quick sneak peek of what what's coming here
let's do it
so the first one is mirror traffic seth if you recall in your last show we talked about safe
rollout in this we talked about how you can use percentage of live traffic uh diverted to a new
version of the model and then make sure that model is working properly before we divert more traffic
to that model and become 100 from blue to green right that's i remember that yeah so one of the
challenges with that issue with that capability is when you transfer live traffic let's say even you
transfer five percent of live traffic to a new modded test what if uh the deployment is faulty
there are some issues with that then your survey is going to be affected because five percent
of your request might fail now so what we are launching right now is a capability called mirror
traffic in which as you can see in the picture uh the v1 of the model will take still can you take
100 percent but now you give a percentage of that as a mirror traffic in this case i am diverting 10
tr traffic to the green but the response from green is not requested it's not sent back to
the client you know so it's like done internally but still you can go and measure your metrics to
make sure that hey i'm meeting my response time requirements am i uh are there no errors so you
can see all your metrics and make sure sla is doing good before you dive divert real traffic to
it so that's the mirror traffic capability that's really cool like i i can imagine scenarios where
you might not even want to let's just say you have an sla of you know three nines for example and you
divert five percent of your traffic to green when it doesn't work at all and you don't know
all of a sudden you're at 95 which is not good this basically just mirrors the traffic and then
you can monitor how it's handling the requests that is right that is right that's really cool
and and and to use this it's very very straightforward and uh you know you can see in the
bottom you have users who have seen this online endpoint update api already available you just
need to set this mirror traffic and you can set it to whatever percentage that is needed amazing
that's really cool any others or do you still have any other surprises yes one more thing uh
big ass from our net from our enterprise customers this is the network isolation support we worked on
this to make sure it's as declarative and simple for people to adopt uh rather than people go and
configure a lot of nsg rules in their v netting we want to reduce the setup overhead here so here
we we support both ingress network isolation for both ingress and egress through private endpoints
so the first command so it shows you how you can secure the ingress to a private endpoint uh
through to your online endpoint through a private endpoint works workspace private endpoint so all
you need to do is when you do the endpoint create you add this flag that's highlighted
like set public network access as disabled so this means there is no public id available for endpoint
and all the accesses have to go through you from your v-net through a workspace private
end point so this secures the ingress and to secure the outbound communication on the egress
from your deployment vms to the external resources like the storage and acr
you you know you can do that by just adding this when you do a deployment create all you need to
do is set the public network access as disabled here so this way uh you know we create private
and private endpoints to the from the deployment to the uh to the acr and block store for instance
and make sure that you know there is no internet access from the containers out so this way you
can secure both your ingers and egress and both are independent in a way that you can have like
a public endpoint with a private egress or like a pro like at the end point which is kind of private
and the the egress can be public so you can set any combinations that you want oh this is so
cool especially when you have sensitive uses uh that you can do this this is great stuff set to
where can people go to find out more uh so there is a link that's that that's going to be shared
in the bottom of this ai show and people can just go there and you know there's a blog
and there'll be links to uh tutorials on mirror traffic and network isolation uh this has all been
really cool satu thank you so much for spending time with us my friend thank you so much sir all
right we've been learning all about managed online endpoint now generally available as of microsoft
build 2022. thank you so much for watching and hopefully we'll see you next time take care
you