Welcome back to the OpenVINO Channel
In Video number 10, we talked about data formats.
Let's talk about integer 8 bit. What we call in short INT8,
About the new instruction-set called VNNI
The motivation to move to INT8 and the possible trade-offs.
And the concept of the OpenVINO INT8 calibration flow
On Video 41, we'll present a hand-on example with the full instructions on how to run a real INT8 example..
In addition, you'll need a system that support the following x86 extensions
SSE4.2 , AVX2 Or AVX512
You can type lscpu and grep for AVX or SSE4 to see if your system support it..
These are special x86 instructions for SIMD, single instruction on multiple packed data.
In addition, your model must contain at least one (ReLU) activation layer
I will explain why.
Obvious benefit number 1 to move from FP32 to Integer is the size of the data.
FP32 data takes the same memory footprint as 4 INT8 .
SSE4.2 can pack 16 bytes of INT8 and perform the same operation on all of them in one clock cycle..
AVX2 can pack 32 and AVX512 can pack 64 INT8 So the performance gain is obvious..
Benefit number 2, is VNNI
VNNI stands for : Vector Neural Network Instruction
It is four new instructions for accelerating inner loops on convolutional neural network
You can think about it as a fusion of few layers that can accelerate your convolution operations by up to 3X.
It will be supported by the new Cascade-Lake systems,
and the performance boost is of course depend on your topology,
the amount of convolution layers your model include and more.
With OpenVINO, you do not need to code with the new instructions,
the CPU plug-in will use the MKL-DNN library functions that already take advantage of these new instructions
Which is great. The same code will be now accelerated without any effort from your side..
At the current moment, 2019 R1.
The MKL-DNN library which is used by the CPU plug-in supports only the following 8bit operations
Convolution, ReLU, Pooling, Eltwise, Concat
This means that only these operations could be performed using 8-bit data
and the rest of the layers should remain FP32
So we have a hybrid execution.. some layers will be FP32 and some INT8..
In order for one layer to be executed in Int-8, the input data has to be converted to INT8
and the weights and biases must be also quantized transitioned from fp32 to int8 format.
And sometimes, if the next operation is FP32. We need to convert the output back to FP32..
So we want to move to INT8.
You remember the flow we described in video 3 (the full inference flow)
There is an offline part, done by the model-optimizer..
It starts with your framework of choice, we start with a converged model
and generate an IR, .xml and .bin files..
This is an offline process you do once..
Then the inference engine uses these IR files.
Calls the right plug-in which uses the right library cells for the implementation.
The flow to run in INT8 requires few more stages..
First for the offline stage..
After we generate the IR files by the model-optimizer for FP32, we need to use the calibration tool.
The calibration tool needs a small sub-set of the data-set in order to operate
The user will also need to define a threshold of accuracy drop he is willing to absorb
The Calibration tool starts with first running the model as is in FP32 on the data-set Using the validation-app..
And get the baseline performance.
Now the tool will try to convert as many layers as possible without degrading the accuracy
below the allowed accuracy drop that is defined by the user.
The way to do it is to run the model again and again, each time to convert one layer to INT8 and check the accuracy
So each layer, at its turn,is scaled based on the statistics of the weights to INT8,..
We'll look at the distribution of the weights, What are the max and min values and are there outliers values we can just remove..
And then the validation utility is used again and again to check the accuracy..
Then all the layers are sorted based on the accuracy loss they cause (when they move to INT8)
and the final configuration is chosen based on the allowed accuracy degradation
The result is a new IR files, with all the statistics and directives on how to run each of the layers..
That could be read and used by the Inference engine..
When you look at the IR .xml file, you could see that the layers hierarchy is the same.
However, the layers that are chosen to be executed in 8-bit format are marked with the appropriate profile attribute,
and their statistics is stored at the end of the .xml file.
This is the original xml file, in FP32 And lets just pick a convolution layer..
You can see that the precision is FP32..
Nothing is special about this layer
Now let's open the calibrated INT8 .xml file Let's look for the same layer..
An you can see here the statistics for this layer..
Going to the layer definition, you can see the quantization level=I8
So this layer will be operated on INT8 format..
In general this IR .xml file looks the same, Just with additions at the bottom..
Each calibrated layer will have here the statistics required by the inference engine to run it.
So we have a calibrated IR files..
Now let's move to the run-time procedure...
The CPU plug-in load the IR file and based on the directives knows how to execute each layer..
So for each layer it can find The directives, the statistics
how to scale the weights, the biases the input data from FP32 to INT8 and sometimes back to FP32
You can run your model with -pc and see for each of the layers if it was executed in INT8 or not
I'm running classification using this model.
Let's look at the log file, And look for our friend conv1
you can see that this layer was executed with AVX2-FP32
I ran the same classification with the calibrated model,
Let's open log.i8
And you can see that conv1 was executed on AVX2-INT8
So this is the high level flow of running your DL model in INT8
Please check for the full documentation on the web..
We have talked about the motivation to move to INT8
What is VNNI What are the limitations and what is the flow for running INT8
On the next video, Video 41. we'll show you a full example on how to run the full flow
See you there Thank you.