Welcome back to the OpenVINO channel
We showed you in previous videos that we can use the same code for inference on different devices
The code could run on CPU, GPU, Movidius VPU and FPGA with just changing the target device name.
But can we break a neural network model and run different layers on different devices?
Yes, we can.
The inference-engine has a flexible plug-in architecture,
Different plug-ins are used for different devices..
For example, when the target device is CPU, the CPU plug-in is used and it uses primitives from the MKL-DNN Library to implement the layers of the network.
So the same neural network topology, could be run on different devices,
each plug in will bring the right library cells and create the specific implementation for that device.
There might be multiple reasons why we would like the same topology to be split and run on multiple devices,
The first one is when some layers are not supported on a device but are supported on another.
For example, this network has one layer that is not supported by FPGA, but is supported by CPU
So it make sense, until you have the FPGA implementation, to run it on the CPU..
Or if a certain layer is running faster on a specific device, we can choose which layer to run on which device..
Or if a full network cannot fully work on a device but if you curve out few layers it's possible.
Then you can run the curved-out layers on a different device..
Does it make sense performance wise?
Depends on the memory traffic overhead between the devices.. and depends on other parameters,, So maybe..
Anyway, OpenVINO inference engine has such a capability.
The "hetero" plugin Can use several plugins for running one network.
There are 2 ways to do it. And you can also use both of them together.
The First option is to Set affinity or target device directly to specific layer
so basically you can specify which layer to run on which device.
The way to do it is to set the affinity of the layer using CNNLayer::affinity
You can see here that I'm taking the qqq layer and assign it to run on the CPU..
The 2nd option is to Let the Hetero plugin run the network layers on multiple devices, according to specified priority
The first device will be the first priority device,
if a specific layer is not supported for this device, the hetero plug-in will fall back into the second priority and so on..
In this example, You can see that I'm asking to run this network on the FPGA..
In case you cannot find the implementation of a layer, please run it on the CPU..
And you could use here all the plugin names: CPU, GPU, MYRIAD, FPGA in your preferred priority
Please note that Some typologies are not friendly to heterogeneous execution on some devices or cannot be executed in such mode at all
Or it makes not sense performance wise to run them this way.
Please also note that different devices use different data types..
CPU uses FP32, while GPU can run FP32 and FP16,
Please read the documentation
some automatic conversions are done by the inference-engine,
For example If you want to execute on GPU with CPU fallback, you need to use only FP16 IR..
The Weight will be automatically converted from FP16 to FP32 by the hetero plugin.
So we talked about the hetero plug in
This will give you the flexibility to run layers that are not currently supported on devices but can fall back to CPU And other advantages
Thank you