Welcome back to the OpenVINO channel
In the previous video, video 42. We have talked about the steps
to follow when operating the DL-workbench​ And I showed you an end-to-end simple example..​
Now I’ll demonstrate many more options, ​ And if something is missing,
please check out the full documentation of the workbench..​
OK, let’s open a new configuration,​ And load, for no particular reason, ssd-mobilenet​
And convert it to FP16​
Once it’s done, I can choose the target device..​ I can even connect to a remote target device..​
I need to specify the credentials..​
But I will run on my local machine​ I’ll start with the CPU​
Now let’s load a validation data-set​ All these formats are supported.​
Directory structure and files format is important..​
I’m going to use a COCO format, so need a zip file with validation and annotation directories​
And I will load my data-set​ …​
I’ll choose to use it,​ And we are ready for benchmarking​
Next ​ I will benchmark
the 3 devices I have in my system and generate few performance points for each one of them..​
And I execute,​ And I have one performance point for the CPU​
That’s good but I want to try a wider range of parameters..​
So let’s do group inference​ We can configure multiple parameters sweep​
Let’s run 1 stream batch 1,​ 2 streams batch 2, 3 batch 3 ​
Or any kind of parameters you suspect will be good
for this model for this device​ And execute,
it’s supposed to take almost 3 minutes​
And I have my 4 performance points on the graph t compare and choose from..​
I would like to generate such a graph for each device I have in the system​
So I create a new configuration, ​ Same model, but now run on the integrated GPU​
One point is automatically generated,​ But I’m going to create 4 of them​
Same configuration​ And execute and now I have a graph for the GPU..​
Same thing for the VPU, the Movidius Myriad-X in this case​
And a group inference and execute​
And I have 3 configurations, each on a different device​
Next thing.. To compare them..​
And now we can compare in pairs..​ So the CPU Vs the GPU​
In this case it is obvious that the GPU
is giving me better throughput,​ for the same latency, more than 2x​
So let’s remove the CPU configuration and compare the VPU​
Here too it is obvious that the GPU is giving better inference results..​
But do not forget you are comparing a 2W device to a 15W one..​
So it really depend on the specifications of the product you are tying to build..​
One more very interesting thing to do it to calibrate the model to integer format​
I explain this process in details in video number 68, just youTube “openvino 68”​
We go to “optimize”, choose INT8, and there are lots of options to choose from..​
You can optimize for performance or for accuracy​ and specify the maximum acceptable accuracy drop…​
I will choose the default method,​ And hit “optimize”​
Now this is a very long process where the tool tries
all kind of Integer and floating=point combinations​
and re-evaluate the accuracy drop again and again..​
It does take some time, ​
But at the end we have an INT8 model, which is a hybrid model
of floating point and Integer​ And we can compare the two​
We have only one point for he integer model but performance jumped from the 60th to the 80th..​
Next to check is the accuracy,​
And we can see here that the accuracy dropped by only 0.4 % while the throughput increased
%by more than 40 A good tradeoff..​
If we are happy with the integer model, we can just download it and use it.. ​
Here are the files..​ ​
And now we can choose one selected optimal point.​ And pack it into a deployment package​
Another thing we can do is visually test the inference​
Let’s select an image and press “Test”​
You can see that class #3 which is a “Car” was detected, here it is​
We can lower the detection threshold and get more objects​
Most of them are cars.. Class #3, ​
But we also have a one person detected, class #1.​
Here he is..​ pretty cool..​
OK, we have seen a full benchmarking flow on 3 totally difference devices, ​
We saw how to calibrate to Integer, pack a deployment code and test the inference results..​
Thank you​