Welcome back to the OpenVINO channel
This is a new feature for OpenVINO R2 2019..
Don't try to run it on earlier versions..
We already understand today that we need heterogeneous computing system
Just like our brain which is not a one monolithic entity, your future system is probably a combination
of CPUs GPUs FPGA and AI accelerators..
And we have a wide set of Intel CPUs
Integrated GPUs
FPGAs
And AI accelerators like Movidius
Looking at the video pipeline we described in other videos,
We would like to run each of these tasks on each of the devices..
Maybe run the decode on GPU, the pre-processing on CPU and the Inference on the AI Movidius
accelerator for example..
OpenVINO allows us to run Inference on CPU GPU FPGA or AI accelerators.
We showed you how simple it is to run your model on the various devices with one parameter change
All you have to do is change the target device..
On Video 29 you can even see how to run portions of the network on different devices..
You can use the hetero plug-in to decide that layer 1 will run on CPU and layer 2 will run
on GPU But you can also set it to run on FPGA for
example with fallback to CPU, this is the basic operation
so if the IE cannot find one layer implementation for the FPGA it will run it on the CPU instead..
Now we introduce a new plug-in that allow the IE to load balance the execution on multiple devices..
So basically the IE , during run-time is checking the utilization of the machine and determines
which inference to run on which device.
If the GPU is available it will execute it, the CPU or other device that you specify is
available it will be used..
So all the devices in your system can share the inference burden, if they can..
This way you can get to a better utilization of the machine, we'll see an example in a
minute..
So you can use it to better utilize multiple myriad-X running in parallel or any multiple
devices that you have in the system..
The API is simple, If you are using the older, plugin centric
API, this is the way to use the multi plugin..
If you are using the new API, it's even simpler..
You can even change these priorities during the program execution,
Example, during the beginning of the program give priority to CPU.. after some time change
the priority to HDDL..
Or switch to only one device when you want to make sure that for a section of the program
this device is fully available.. (or totally free)
You can even specify how many inference requests will be allocated to any of the devices,
checkout the full documentation for that..
Let's see an example for CPU and integrated GPU..
The middle window is just "Top" and you can see that the CPU is at 40% utilization
The bottom window is metrics monitor, please check video 45
You can see that the GPU is working at 300 mHZ and hardly working..
At the top I have the R2 sample executable, And I'm going to run the benchmark app on
a face detection model on the CPU only..
You can see that my CPU is almost at 400%, I have 2 cores, so 4 threads fully running
But the GPU is completely idle..
And I'm getting 180 FPS
Now let's run the same benchmark on the GPU.
Please note that the GPU and CPU both can now run FP16
And we see that the GPU is at high utilization running at 800 MHZ
CPU is using 200% just to prepare the inference for the GPU
And I'm getting 180FPS..
Now let's use the MULTI plug in and run on both CPU and GPU
The IE is loading the model to both the plugins..
GPU at 90+ % utilization CPU is at 200%
Both are running inference in parallel And we are getting 230 FPS, better than just
using only one of the devices..
Pretty cool.
So we talked about the MULTI plug in And how you can use it to execute inference
requests on multiple devices in parallel and by that better utilize your heterogeneous
platform
Check out the full documentation on this link Ask questions in the forum if you have any.
Thank you