Welcome back
to AI
Model Optimization
with Intel Neural Compressor.
With so many AI model
quantization techniques to choose from,
where do you even start?
Post-training dynamic quantization
provides
a good balance
of optimization,
accuracy, and low effort.
Quantizing parameters to INT8
data types
can reduce a model’s memory
footprint up to 4x
and can speed up inference as well.
But one of the challenges
with mapping to INT8 is calibration
- figuring out
the range of floating-point values
that parameters can span,
a concept
we covered in the video
in this series on
How to Choose Between
Quantization Techniques.
This can be challenging for activations,
which vary depending on the model
inputs,
whereas weights are fixed after training.
Dynamic quantization simplifies
the process
by treating weights
and activations differently.
After the model
is trained in floating-point,
it quantizes the weights to INT8,
but does not quantize the activations.
Instead it
inserts functions into the model
to quantize activations
to INT8 during inference
based on the observed ranges.
So now
the matrix multiplications are all INT8,
and accumulations are then either INT16
or INT32,
which are reduced to INT8
if the next layer is quantized.
While this adds
some runtime
overhead versus
full quantization of all parameters,
it increases accuracy
because it can dynamically
adjust based on the model
inputs at inference.
This approach is optimal for models
whose runtime bottlenecks
are loading
and storing weights in memory,
such as transformers and LSTMs.
So getting started
with dynamic quantization in Intel
Neural Compressor can be as simple
as specifying in the configuration
to use dynamic quantization,
then calling quantization.fit
and passing
in the model and configuration.
And this is the same
API across PyTorch, TensorFlow, and ONNX.
But you also have options to customize
this process,
so let’s
take a look at a real application.
If you search for Intel
AI Reference Kits, you’ll
find this collection of applications
you can use to get started.
Here we will use the Text Data
Generation kit.
This has detailed
instructions and explanations
to get you started,
but we will just touch on a couple
things.This is based on a GPT2
model fine-tuned using news headlines -
so given a word as a seed – in this case
we give it two seeds - it generates
a synthetic headline for each seed.
Just focusing
on quantization, our two lines are here.
You can specify rules
by operator type
or name, here it’s by name.
I’m not sure it’s necessary here,
because we’re using an eval function.
Even though the eval function
is not required
with dynamic quantization, it’s
good practice
in order to guide the weight
quantization process.
We will cover eval functions
in the video on static quantization.
Here you can see the tool measuring the
results and adjusting parameters
and if certain operators
need to stay in floating
point, Intel Neural
Compressor will do that automatically.
I tried quantizing
without the eval function,
and the output looked more like this.
You can download the kit and try it
yourself, and see what kind of headlines
it generates without an eval function.
The inference
script has a benchmark mode,
and these take a few minutes to run,
so skipping ahead to the results,
which show that for this instance
on Intel Developer Cloud,
quantizing dynamically
to INT8 gave us over a 1.6x speedup,
and of course reduced
the size of the model.
If you would like to learn more
about Intel Neural Compressor,
or to try out this Reference Kit,
they’re both linked below
or scan the QR codes.
And be sure to check out the other videos
in the AI Model Optimization series.