Hello, this is Munara from Intel.
When we say
Intel optimizations for Pytorch,
we mean optimizations developed by Intel
that were upstreamed
and already integrated
into stock Pytorch framework
and optimizations
implemented in Intel extension
of Pytorch.
Let’s focus on upstream
optimizations first.
In collaboration with Meta, Intel
regularly upstreams
software optimizations
designed to improve the performance
and scalability of PyTorch.
For example, Optimizations in the oneDNN
library leverage Intel hardware features
such as specific
instruction sets like Intel
Deep Learning Boost, including
Intel AVX 512 VNNI,
AVX512 BF16, and AMX.
Other optimizations
include customized data layouts,
kernel optimizations,
mixed-precision support,
and some graph transformations.
For a developer
to use these optimizations,
one doesn’t
need to make any changes in the code.
On the other hand,
IPEX is a package developed by Intel
as an extension for open source Pytorch
and includes Optimizations
that, by design, have not been integrated
to open source Pytorch yet.
IPEX contains optimizations
targeted for Intel's latest
hardware platforms,
including support for discrete Intel GPUs
and more aggressive optimizations
that may not be appropriate
for general use
but provide
even more performance
in some specific domains or cases.
For example,
for some
specific types of models
like transformer
based models or Large Language Models,
Intel implements
optimized custom operator kernels
that are heavily used in these models
and thus improve the performance
of these models significantly.
Intel Extension for PyTorch enables
Int8 quantization
for optimizing a model in size
and performance.
It could invoke Intel
Neural Compressor features
internally for quantization as well.
So, if you have access
and would like to try
running your application
on the latest Intel hardware
or if you would like to try out
the more aggressive optimizations,
you can install Intel Extension
for Pytorch
separately
for Intel Extension for Pytorch usage,
some minimal code change is required.
If you are running your model on CPUs,
we suggest trying out the stock Pytorch
first
and seeing if performance is acceptable.
If more performance is needed,
then try the Intel Extension
for PyTorch package next.
If you are running your model on CPUs,
we suggest trying out the stock Pytorch
first and seeing
if performance is acceptable.
If more performance is needed,
then try the Intel Extension for Pytorch
If you run your model on discrete
Intel GPUs,
you should use the Intel Extension
for Pytorch package directly first.
If you wonder
what type of optimizations
are being developed for Pytorch,
then, in general,
we can divide them into three
categories: Operator level optimizations,
Graph optimizations,
and optimizations affecting Runtime.
We aim to improve
the time
it takes to execute a single operator
by: Applying
vectorization
and multi-threading techniques.
By enabling the operator
to run in low precision,
such as BF16 or Int8.
And by changing the data
layout to favor cache locality.
We also perform
graph level transformations like:
We also perform graph level
transformations like constant folding
for reduction of compute time,
and operator fusion
for benefiting cache locality.
With Runtime Extension,
we provide Pytorch APIs
to control thread execution behavior,
such as binding threads
to physical cores and
enabling multi-stream runs.
We provide a launcher utility script
that helps users run
an application
using different
optimized configuration settings
when running performance tests.
We also support execution
with discrete Intel GPUs.
You can learn about this topic and more
by following the links
in the description.
If you like this video, please share it
and remember to subscribe to the Intel
Software YouTube channel.