Welcome back to the OpenVINO channel
Let's talk about Numbers representation..
Integers, floating point
What are the trade-offs to use the different data and weight formats
How to use the Model Optimizer to change the formats
What are the supported and recommended formats for each of the devices
Numbers could be represented in many ways
When talking about integer the more bits you have the wider range you can represent ,
one more bit give you 2x the range..
For example with 8 bits you can represent 2^8 numbers..
So 0 to 255
In floating point representation we allocate a bit for the sign,
few bits for the exponent, and the rest for the mantissa or fraction
This allow us to represent a very wide range of numbers.
If we assume that this is the range that could be represented by INT8 format
This is the range for Floating point 16bits
And this for FP32
And the distribution of weights looks like this,
It is obvious we cannot just squeeze the weights into a smaller format without loosing accuracy of the model.
We are getting what we call "quantization errors"
We will talk about re-assessing the model accuracy with the validation utility on video 33
We will talk about the calibration tool and quantization on Video 40
The trade-offs are obvious
The more bits we allocate to represent the data and weights, the wider range they could represent
and potentially the better accuracy of the model
But it comes with a price..
Because the bigger the data More space in memory is required to store it
Higher memory bandwidth required to transfer it around,
And of course, More compute resources and time it takes to compute
Not every device support all possible formats.
We need to talk about the weight format.. The model
The input of the data format and the output of the calculation format..
You can find this table in the inference-engine documentation
You can see here that FP16 for example is not supported by the CPU plug-in,
For the integrated GPU FP16 will probably give you better performance
and this is why it's the preffered format
And Movidius VPU support only FP16
FPGA support a wide range of formats, we'll discuss it in another video.
This table describe the data input formats.
And another one will describe the output formats..
Let's run the model optimizer.
Let's initialize the environment
Let's get a resNet model..
I go to the model_downloader..
And download resnet-50 model
,,, model is here..
Let's define the model path
Now, let's navigate to the model optimizer directory
Let's tun model-optimizer, The model name will be -FP32
the data-type is FP32 And running..
And let's create another IR file, with FP16..
The best way to know what is the best format, is to try..
As you could see, I have generated 2 IR files, and I can just try both of them
Pls look at video 32 for explanation on the benchmark utility.
So we have talked about: Numbers representation..
What are the trade-offs to use them
How to use the Model Optimizer to change the formats
And which formats are supported and recommended per each of the devices
Integer support is a deeper topic, look for it in Video 40.
Thank you