Welcome to the OpenVINO channel
I have a model.
It is trained and converged.
Now, what is the performance?
Intel has so many systems, which system is good enough for me?
Which system to use? which device to use, and so on..
The best practice is to know in advance what are your targets.
This is usually derived from the overall application you are building and the product..
It starts with Frames per second, or actually inferences per seconds,
as you might have many inferences in one frame
No need to shoot for 100 FPS when your system need only 10
2) Respond time or Latency
Some techniques, like batching, will improve the efficiency of your model but degrade the respond time.
If you are running a batch of 64 images at once, it may be more efficient per frame,
But the first frame will take longer to infer..
so, if you have a delay sensitive application, and you need real-rime response, you might need batch =1
3..
What is the best data format?
Some devices support FP32 and Integer, some FP16
Which format is best for my device..
4) Which device
I have a heterogeneous system. I have a CPU but also and integrated GPU
and a Movidius VPU or an FPGA connected Which device will run my model best?
So the most simple way to do it is to use the benchmark_app
and try the same model on multiple devices or multiple systems..
Let's open a directory for the samples,
Initialize openVINO
CMAKE,
and make only the benchmark app
I'm showing the C++ sample, but there is also a python application, look for it..
Now I need to create a folder with test images,
You could use your specific data-set I will only copy the car image from the security-barrier-demo..
Now let's run the benchmark_app
-i is the input images folder -d can be any of the supported devices..
Let's run it first on the CPU
-m is the model .xml .. The IR file you generated using the model-optimizer..
I'll use a squeezenet model..
And the model is running.
You can see that the image is automatically resized to match the input sizes..
You can see that the latency is 5.1 mili-seconds And the throughput is 380 inferences per second..
Now I can build a table to compare the inference efficiency and latency for various devices and configurations..
Let's run with higher batch size
You can see that I'm getting disappointing results..
So for this model, running with high batch on the CPU makes no sense
Let's now run on the GPU
I'll need FP16 format..
And I'm getting 391 FPS.. Not bad
And on the GPU with batch 32.. 710 FPS
you can see that for my specific GPU, which has large memory, Moving to Batch 32 doubles the performance
And so on.. I can run INT8, different number of batches, and compare
Some other interesting flags..
-api allow you to run the benchmark in a Asynchronous or synchronous mode,,
In asynchronous mode the inference requests are being launched one after the other without waiting for the result
which will arrive when they finish the execution... Asynchronously ..
-nireq determines the number of infer requests launched in parallel
If you are using Intel CPU you can benefit from what we call "Throughput Mode"
This is a new feature from R5 2018, and it enables efficient parallel execution of multiple inference requests
It reuse the weights in convolution operations, which also happens in batching
But here, The CPU cores are evenly distributed between the inference requests,
the amount of cheduling/synchronization is reduced, the cache utilization is improved..
And of course we execute it on multiple threads..
So if you have a multi-core CPU you can expect much better performance results..
The rule of thumb is to infer up to the number of cores your CPU has..
So I will run the same model, batch 1,
I will ask for 4 parallal inference requests to be run on 4 threads
And you can see that we got to 404 FPS
Of course if you have a 28 cores machine, you'll see here a huge improvement...
If you'd like a detailed report of the execution, Just add -report_type detailed_counters..
Once the benchmark is finished, all the data will be stored in a .CSV file.
You can open it and see the parameters,
And you can find the execution time of each layer, for each of the iterations..
And of course the summary at the bottom..
So, if you have a model, use the benchmark-app
Use different devices: CPU, GPU, VPU Use different batch sizes, synch or asynch execution
different number of infer requests, of threads and so on to find the best configuration that optimize your inference..
Subscribe to our channel to get more videos like that
Than you!