[MUSIC] Hi everyone. Very glad to share my work
recently on High Performance Training and
Inference for NLP Models on GPU.
We know that these large neural networks
are pushing the boundaries of AI.
In particular, for applications
like machine translation,
automatic speech recognition,
and visual recognition recently.
All these applications are really relying on
this large-scale neural network.
In particular, one neural network very useful
for all these applications
is called a transformer models.
Right now, the transformer models
are reaching the of
the art [inaudible] for all these applications.
In particular for machine translation,
language understanding,
image classification, and the speech.
These models, like what I described on the left part,
this has common architecture.
They can vary in different number of
layers and the different number of parameters.
Normally, the basic version include
around 50-100 millions of parameters already.
That's a pretty big number.
If you trim that down a single GPU,
that requires a few hundred hours
of GPU time to train that.
More latest work using this type of a transformer model,
they are relying on a very large model.
Here I plot the model size forces
and the training time, in particular,
Swin Transformer here is
the latest transformer model
for computer vision and for image classification.
It gets the state of
either result for last year
and it's actually the best paper from ICTCD last year,
and mRASP2, it's getting
the best result right now
for multilingual machine translation.
The largest one is this GPT3.
Perhaps you have heard of this.
This is a language model with
close to half a trillion parameters and you
need millions of GPU hours to even train that model.
Well, if you compare
the energy needed to train that model,
it's equivalent to driving
a common passenger car for 146 years.
That's a lot of energy consumption.
Now, of course, that's very
expensive and at the university,
it's very difficult to do that as well.
My focus here is to develop high-performance library
for this specific type of transformer model family.
We developed this library called LightSeq.
Its an efficient training and inference library.
Comparing to some baseline frameworks like TensorFlow,
Pytorch, both are popular deep learning frameworks.
You can get 14 times speedup in inference and
45-250 percent speed up
compared to Pytorch and TensorFlow.
It's also very convenient.
You can still use TensorFlow,
Pytorch, or Hugginface, etc,
to describe the model and put that model to LightSeq,
and then use LightSeq to further train and inference.
It's also deployed in industry-scale products.
It's already deployed in VolcTrans.
One of the client of VolcTrans is TikTok.
It's serving one billion customers every day.
In comparison, I'm listing
several libraries for transformer computation.
It includes one library from Nvidia, FasterTransformer,
and from Tencent,
TurboTransformer, and from Microsoft, DeepSpeed.
They are designed for all different types of purpose.
FasterTransformer and TurboTransformer are
rarely used for inference.
The portables are brought GPT3 type of model,
and for a transformer as well.
But they do not support
more advanced model like VAE or Multilingual
Transformer with more diverse inference techniques.
DeepSpeed instead, it's focused on
training if [inaudible] really
support any particular inference method.
Our LightSeq, in comparison,
it supports all different types of model and
all these various inference technique
and supports training as well.
It can integrate with PyTorch and TensorFlow as well.
In terms of performance,
we compare this LightSeq with
several other frameworks on different model size.
We have six layers of encoder,
six layers of decoder,
that's the standard transformer.
We have larger ones like 12 encoder, 12 decoder,
and even a larger one with 24 encoder and 24 decoders.
Different types of GPUs,
hung in as well.
This is the training performance
in terms of the number of tokens per second.
In comparison, this is the same plot,
but we plot it in terms of speedup
comparing to the basic PyTorch.
FairSeq is a PyTorch library-specific sequence learning.
If we combine FairSeq with the Apex library,
which is again developed by Nvidia for quantization,
you can achieve some little bit speed-up.
But if you combine FairSeq with LightSeq together,
you can achieve
almost three times speedup in terms of training.
In terms of inference,
we also compare inference of this transformer model.
This is a machine translation of creation
on T4 and P4 GPU.
Usually, we don't really need this A100 or V100,
those powerful GPUs for inference.
We can use less powerful GPUs like T4.
In both cases, the LightSeq achieves over
10 times of speedup compared to PyTorch and TensorFlow.
It's also faster than
the FasterTransformer library from Nvidia.
There are three major techniques
that's very important for the LightSeq improvement.
They are the kernel operation fusion,
and dynamic memory management on GPU,
and this is optimizer acceleration.
In the computation of a transformer,
there are two types of computation.
The matrix multiplication, which
can be realized by making cuBLAS library or cuBLAS,
and there are other operations.
The one technique we used is to
fuse all the rest operations together,
all the other operations other
than the matrix multiplication,
and fuse them into just one operation.
Then also we optimized the number of times we need to do
the reduce operation by using algorithmic formulation.
With that, we can basically improve
the portion needed for
operations other than matrix multiplication.
Comparing to the Baseline and
the LightSeq approach, after optimization,
the matrix multiplication occupies
the majority of the time here.
We can also improve the memory usage as
well by dynamic memory management,
comparing to the basic PyTorch implementation where we
can reduce the memory consumption by almost a half.
With that reduction, we can increase
the batch size for transformer training.
With that increase of the batch size,
not only we can improve the speed for training,
but also we improve the accuracy as well.
Because usually, these large batch size will
stabilize training and also improve the performance.
Looking forward, right now
we still use handwritten optimization library.
But in order to tailor it to
a different hardware as Professor Ding mentioned,
in looking to the future,
we still need to do
adaptation or automatic compilation
to different hardware,
and auto-tuning according to
the different data distribution.
I haven't talked about very large models so far away.
We can only optimize that for Kernels on single GPU.
We still need to
optimize the training and the inference for
very large models that
cannot fit on a single GPU. [MUSIC]