Welcome back to the OpenVINO channel Calibrating models to integer is probably
the fastest way for you to accelerate the performance of your models..
You can go and re-train your models with a more compact data-format,
but I guess that this is faster. First benefit it obvious,
we can squeeze 4 INT8 into a one FP32 So model weights and memory-bandwidth required
are smaller In addition,
we are getting the acceleration of SIMD (Single instructions multiple data) operations..
For example AVX2 can pack 32 and AVX512 can pack 64 INT8 into one data structure
And perform the same operation on all of them in one clock cycle..
So the performance gain is obvious.. Especially on Xeon machines.
And if you have a machine that supports Intel DL-boost,
You can enjoy a significant additional acceleration to all your convolution layers..
DL-boost is basically new instructions for accelerating inner loops
in convolutional neural network You can think about it as a fusion
of few layers that can accelerate your convolution operations by up to 3X.
In this video, video 68
I’ll talk about the concepts of the Post-Training Optimization Toolkit,.. Which is part of OpenVINO
In video 69, I’ll demonstrate a full example of quantizing a model to INT8 stage by stage..
I have to mention that the easiest, by far, way to operate the POT is using the DL-Workbench.
Please watch our videos 42 and 43 But in this and the next video,
I’ll explain all the details and we’ll operate the toolkit standalone..
This is the overall flow for the POT the Post-Training Optimization Toolkit
we start with a model that is already in IR – intermediate
representation in FP32 or FP16 format
And the target is to convert as many as permitted layers to INT8.
As inputs we will need the model IR files and the data-set with the annotations,
The POT is implementing a full inference flow, It has a reader that read the data,
annotations can be converted per need, The pre-processing stage can perform
all kind of manipulations on the data, Then a launcher is called,
basically inference-engine, The inference could be done
using OpenVINO but also by the frameworks themselves, Caffe, MXNET OpenCV, TF etc..
And the last stage is to calculate the metrics so we could check the accuracy
You usually do not have to worry about these internal stages,
But if you have a special model, you can customize any of these stages..
So if your data need a special annotation conversion or a special pre-processing,
you can build this specific stage yourself.
The POT can work in 2 major modes. The first one is the default quantization,
The target here is to perform the fastest possible quantization,
It will perform relatively faster, and it is more performance focused..
Despite the fact that it is not accuracy aware, in most cases, we saw that the accuracy drop
is not significant.. The default quantization algorithm
You can read about the full algorithm and the stages..
and all the possible parameters you can play with in order to control this process
The second mode is called Accuracy Aware Quantization
Here the algorithm will usually quantize only some of the layers,
until the accuracy drop to the level you allow..
It starts with running the default quantization as in the previous mode,
and then, using an iterative process, each layer of the model is ranked based on
the performance gain it can provide, if converted to INT8 and the accuracy drop it can cause..
At the end of the process, the optimal combination of Floating point layers
and quantized Integer layers is set. And here you can see the full documentation
of the accuracy aware quantization.. You can read the full description of the flow
and all the possible parameters you can play with in order to control this process
Let’s take a look and compare the neural network before and after the quantization..
On the left side you can see the original model You can see here all the layers one by one.
On the right side you can see the same neural network, after quantization
And let’s focus on the first convolution layer You can see that 2 new layers
were added before the convolution, These are what we call Fake-quantization layers
These are really directives to quantize to INT8 the 2 inputs to this convolution during run-time.
So if you look at the xml file, you will not see them as INT8 layers,
They are really just operations, done by the run-time during run-time
So the inputs will be Integer, The convolution will execute on Integer,
What about the output? that depends on the next operation,
If the next operation requires FP format, the runtime will de-quantize the output back to FP..
It all happens during runtime and per the requirements of the next layer..
And you can see many fake-quantization layers all around..
In the next video we’ll do together a full real example.
Now let’s talk about the process, There are basically 3 major configuration files
you have to edit in order to operate the flow.. The first one is more of a global file that
describes the data-set structure and location. You can define here multiple data-sets,
You have to know which data-set was used to train your model,
and you can follow this link for the full instructions on how to prepare the data-set.
You can see here for example that I’m using the image-net,
and I provide the location of the model, the annotation file and so on
The second important configuration file, is the model yml file
And it defines the inference pipeline per launcher,
For example here, you can see that the framework,
the launcher is dl-sdk, meaning openvino And you can see the details of the data-set,
the preprocessing and so on. The last one is a json file,
and you can find it in this location.. It describe the model that we are trying
to optimize, and the algorithms used Here we are using mobilnet V2 model,
and performing Accuracy aware quantization on a subset of 3000 images.
The performance boost we see after calibrating models to IN8 is huge.
Especially if you have a system that supports DL-Boost..
In the next video, video 69, We’ll do a step by step example.
See you there.. Thnx