one of the more powerful ways to use rust is to build a systems tool that talks to a GPU
fortunately because pytorch bindings work so well with rust what I'm going to do is piece together
a tool that can not only talk to the CPU and saturated via Pi torch but also can talk to a Cuda
enabled GPU saturated with pi torch and then use some of the advantages of rust which is the true
cores that allows you to spawn a pool of threads and then send data into a GPU to try to get the
most out of it in terms of a stress test let's go ahead and build that tool in just a few seconds
let's take a look at the architecture of how you could build a stress test tool for a Cuda
enabled GPU by using the systems programming capabilities of rust and The Helpful rust by
George bindings first up we have to have access to a Cuda enabled GPU with GitHub code spaces
that's one way to do it it could also be a AWS instance or gcp instance or an Azure instance we
then configure Nvidia SMI monitoring so next up we look at how the GPU is being able to be utilized
during our stress testing then we hook together the rest Pi torch bindings once those rust pytorch
bindings are enabled then I build together a rust command line tool and I have three different ways
that I execute my stress test I have a CPU a GPU and then I use the true thread and
the powerful multi-threaded but memory efficient capabilities of rayon and I shove data into a GPU
in a multi-threaded fashion to do a full stress test all right let's go ahead and get started
three two one okay let's go ahead and get open this environment here I'm going to go ahead and
launch it and we can actually very easily get started with building out this stress test and
some of the things to take care of right out of the way is to make sure that we've got the
correct cargo structure set up that's really one of the key takeaways here and it's probably easy
to go through and you know first run this to see how it works and then we can see what are
the changes we want to make in the future so here I have a translate app that's one and I also have
a stress that app I'm going to go ahead and use the stress one and this one is pretty interesting
because if we look at the cargo file we're using three dependencies first up we have the command
line tool second we have the pytorch bindings for rust and then we have rayon which allows us to do
multi-threaded code that's that's all I have to do to build this stress test tool next step
I again use this pattern my library code lives in lib.rs and my chameleon tool lives in Maine let's
go ahead and take a look at the library code first here inside of the library code we have functions
that are built and each function is quite simple so I have the rayon Imports I have the tensorflow
Imports but if we look at this you can see that I build a load test and all it's doing is sending a
vector into the device in this case it's a CPU and it's going to print out what's happening in
this case I build a GPU load test function so we just change the device so we say look I want to
actually iterate into this Cuda device and then finally all I need to do to change it to have a
multi-threaded version of this is just use this into par enter for each op and then throw that
into the Cuda device as well so what's really cool about this is we have three completely
different ways to actually utilize uh testing now all you need to do is go over to my main and again
it's the exact same pattern this is essentially boilerplate code here's the stuff that matters I
build out a CPU sub command I build out a GPU sub command and then I build out a t GPU sub command
for multi-threaded and look I just map each one of those to the library code so this pattern is
so powerful because you build just a few lines of code in a library and then you map that those
functions into here the only other thing you have to do to be aware of is make sure that these are
each Pub once you've done that you're good to go now all I need to do to run this is is go
ahead and run cargo so let's go ahead and build that we'll say cargo and then say dash dash help
and we can say cargo run help and this will go through and compile it again
and it'll give us a help menu which will correspond directly to each of
these commands the CPU the GPU and the QI GPU and that's exactly what we see
now one of the things that's really helpful is to do two kinds of monitoring First Step
let's go ahead and do each top and let's go ahead and take a look at the CPU here so if
we go through here and we look at the CPU and I run cargo run and I say CPU what's going to
happen is it's going to throw a bunch of data into the CPUs and look you can see
it's saturated right now if I go over to the GPU monitoring you'll see nothing's happening
right so this is a great way to understand what's going on with your code right we we know that
we're using the CPU device so no traffic is going to the GPU so next what I'm going to do is now
we're going to change gears here you can see it stopped and I'm going to go through and I'm going
to say okay let's go to the GPU now what happens when you go to the GPU we do a GPU stress test and
lo and behold it's going to send a bunch of data into the GPU we'll be able to see it by toggling
right here and you'll see this thing getting saturated Now it only really gets up to let's say
a quarter of the load and if we look at the the threads here as well they're still being utilized
so basically the the cores are still being utilized but some traffic as well is being sent to
the GPU now can we do better than this can we get this load higher if we really want to stress test
things well we can and that's where the threaded GPU comes into play and that was the code I wrote
earlier that uses rayon and now let's take a look at this there we go so we've been able to actually
over more than double people the load that's getting sent to the GPU right it was really around
like 20 25 so let's say it's been two to three times more load and now if we look at the CPU
check this out notice how it's actually reduced the load of the the CPU so you can see here once
you start putting enough traffic into a GPU it's actually a great way to offload uh some of the
resources on your system so what's really great about doing stress testing for a tool like this
is to really get a feel for what what's happening in your environment unless you do a benchmark you
really don't know what's happening especially in a distributed computing problem like doing Pi torch
training on a GPU and you can see here rust is the perfect tool to do training to do inference
to build portable binary tools and really rest is the systems tool that is an Ideal tool for mlops