- [Instructor] This video walks
through a large-scale inference solution
on EKS.
Here is a summary of the problem.
We need to deploy a large number of models
for inference on AWS.
Each model needs to be individually accessible
through an API,
and each model needs to respond within 100 milliseconds
of receiving a request.
The solution needs to be cost-effective and scalable.
To solve the problem, we use accelerator-enabled EKS nodes.
We pack as many models as possible into each node.
This allows us to serve a large number of models
at sub-100-millisecond latency
while achieving sub-$100-per-hour cost
according to on-demand pricing.
To deploy this solution, execute the following steps.
First, clone the AWS-do-inference GitHub repository.
Then configure the project
by executing the config.sh script.
To enable deployment to Kubernetes,
you'll have to specify the registry setting.
For 3,000 models,
you will need to configure 160 model server pods,
which are spread over 40 inf1.6xl instances.
Next, build the base container
by executing the build.sh script.
This container image has all the necessary tooling
and will be used later to compile and package your model.
To compile the model and enable it to run
on Inferentia or GPU accelerators,
execute the trace.sh script.
The compiled model is saved in a PyTorch trace file.
Execute pack.sh to package this trace file
into a container image together with a model server.
Then authenticate with the registry by executing login.sh,
and push the container image by running pack.sh push.
Similarly, we need to build
and push the test container image,
which we're going to use later to verify our deployment.
Execute test.sh build,
login.sh,
and test.sh push to accomplish this task.
Next, we'll scale up the EKS cluster.
To do that, modify the corresponding node group
and set the desired capacity to 40 inf1.6x large nodes.
Here, we use the open source tool Qbox view
to visualize the cluster and its workload.
We are now ready to deploy our models.
To do that execute deploy.sh.
Kubernetes manifests are automatically generated
and applied to the cluster.
160 model server pods serving 19 models each,
equaling total 3,040 models,
get evenly distributed among the cluster nodes.
We are now ready to run some tests.
We will use the open source toolkit KL
to aggregate the logs of all model server pods
in the lower right.
Each model server log is displayed with a different color.
We will execute the tests
in the order they are described in the test script Help.
First, let's execute the sequential test.
In this test, a request is sent
to each of the models in the deployment
in a sequential order.
We can see that the responses from the models
are grouped by color.
This is because requests are being sent sequentially
to each of the models being served
by each individual model server pod
before moving on to the next.
We can see each request is receiving a response
within 21 or 22 milliseconds.
Next, we will run the random test.
In this test, requests are being sent to a random server
and a random model within that server.
The randomness of this test
reflects in the colors in the lower right,
as each individual server and model
responds to a random request.
We can see that the latency of the responses
is holding within 22 to 23 milliseconds.
Next, we'll run the benchmark test,
which uses Python multi-threading
to send random requests to the system
and measure throughput as well as latency
in the P50, P90, and P95 percentiles.
Once the test is complete,
we run the benchmark analysis job
to aggregate the results.
The results show that we are achieving a throughput
of more than 75 requests per second,
at latency of less than 36 milliseconds
on the P95 percentile.
Finally, let's clean up our deployment.
To do that, execute deploy.sh stop.
Once the model server pods are terminated,
scale down the EKS cluster by adjusting the desired capacity
of the auto-scaling group back to zero.
All of this source code and documentation is available
in the AWS-do-inference repository
on GitHub AWS Samples.
Thank you for watching.