Welcome back to the OpenVINO channel
Here is a great way to benchmark models, on multiple devices, multiple parameters,
calibrate to INT8, compare and analyze the results and even get a package ready to deploy
the inference Without writing a one line of code
In this video, Video 42, I'll show you the basic operation end to end
In the next video, Video 43 I'll show you how to compare multiple runs, multiple parameters,
calibrate to integer and more
DL-workbench is well documented.. You can follow this link to get all the details..
There are even recorded video instructions
But let me give you my quick overview..
These are the high level steps.. I'll demonstrate them one by one..
We will install and start using the workbench
We ll get a model, prepare it for inference Get our data-set, and benchmark the model
Then we have many options to analyze the results
and maybe rerun the benchmark with other parameters..
Then we take the preferred configuration and download a package that could be used on an
edge device for deployment..
Let s start
There are many options to get the DL-workbench. You can install it with the OpenVINO package,
and in the future even on the dev cloud.
The easiest way is probably to get it from Docker hub and run it in a Container..
Let s check if I already have an image.. Yes, I have the OpenVINO latest
Let s see If I have a container
no no containers..
But it all does not matter when you run this command..
And you can find it in this video description just copy and paste
Run it with Sudo permissions.. If you already have an image it will create
you a new container, if not it will download the image for you
Give it a new name The tag is the workbench latest
And if you have an integrated GPU or MYRIAD device like NCS, please enable them using
these flags..
And it s running and all you have to do is to open this link and you have the workbench
open in a browser.
First thing is to get a DL based model. .. You can bring your own model, or get it
from the OpenVINO open model zoo
Let s open the link in a web browser Create a new configuration..
And import a model I can load my original model,
You can see the formats accepted formats For example, tensor flow, version 2 choose
a file, and so on.. But I will take a model from the model,
I'll choose an SSD-mobilenet model And let s get it..
Then, sometimes the model need some preparation
So the Model-Optimizer is built into the DL-workbench..
INT8 calibration is possible in a later stage
and only if you supply a data-set so I'll demonstrate it only in the next video..
Video 43
If your model is already in an IR format, or in ONNX format, it needs no conversion
But, Our model do need it, as it is now still in a Caffe format?
So I ll choose FP16, and convert it
Please note that until the model is ready, we cannot choose a target device,
as not all models are supported by all devices..
Once the model is converted, I can choose the target device..
I have on this system a CPU, an integrated GPU and an NCS2 VPU stick..
I will choose the CPU
Now we need a data-set. You can bring your data-set, in multiple formats
or generate a synthetic one..
For this example, I will not use a data-set And I will just generate one..
The tool already knows the expected image sizes,
And I will generate 100 synthetic noise images..
It does limit me, because I will not able to calibrate to integer and other operations,
But sometimes, this is good enough..
And now we are ready to benchmark the model..
So we got the model We got the target device, CPU in this case
And we got the data-set, synthetic in our case
Now we can create the first benchmark point
And we are getting one point on the graph, latency of almost 10 ms
And throughput of 101 FPS
Let s create another point.. I ll try batch 8, and execute
As you can see, I can choose batch size and number of parallel streams for execution..
And we have a graph, Now I can choose my preferred performance
point and tradeoff latency with throughput in this case..
Next stage is to analyze the results. We have few tools that will give you insights
on the performance of your neural network..
We can choose one of the performance point. Explore the execution time by layers
And we can visualize the model Here is how the IR graph looks like
And here is how the nGraph, the actual execution
graph looks like We can even color code it, to easily detect
which layers took more time to execute And I have here a table of all the layers..
Let s sort it by execution time.. Layer Conv3 took most of the time..
It was executed sixth, and took 6.1 mS You can see here all the information on this
later, how it was fused from several IR operations into a single execution convolution
And in the next video I ll talk about more optimizations like calibrating to Integer
and others?
So we can analyze the results, Tune the benchmark parameters, and re-run
with a higher batch or different number of streams, until we are happy with the results..
Once the inference performance meet our requirements..
We would like to take the model with the exact parameters into our target device, for inference..
So we can go to Pack Specify the target device, here it s only
CPU? Do you want the model included, and so on..
And the deployment package is generated for us..
It is expected to be 55 Meg And the only limitation is that is only supported
on Ubuntu 18..
Deploying the inference your target machine requires few more steps,
You can find them in this link.
So, we have seen a great way to benchmark models, from installation to deployment,
Using a web application. The demo that I showed you would have taken
me a LOT more time if I had to operate all these scripts and options by myself..
Next video, Video 43 I ll show you a much more complex example,
on multiple devices, with integer calibration and more..
See you there..