>> You're not going to want to miss this episode of the AI Show,
where we look at ONNX and specifically the ONNX Runtime.
How you can make smaller models that are
sometimes faster and maybe will
even fit on a phone. Make sure you tune in.
[MUSIC]
>> Hello and welcome to this episode of the AI Show.
We're going to talk about something really
cool called the ONNX Runtime.
I have someone special with us.
Emma, why don't you introduce yourself,
tell us who you are and what you do.
>> Hi, everyone. This is Emma.
I'm a Senior Program Manager in the AI Framework
Team under Microsoft Cloud and AI group.
Our team aims to make Machine Learning Engineer more
efficiently as through optimize the library tools and communities.
I'm focusing on AI Model Inference
and [inaudible] Acceleration with
ONNX and the ONNX Runtime for open and inter-operable AI.
>> This is fantastic. For those who don't know,
what is ONNX and then what is the ONNX Runtime?
>> Yeah. So for people who are not
familiar with ONNX and ONNX Runtime,
here is the graph which explains
what a role ONNX and the ONNX Runtime play.
>> You have to pause right here, there's a window on your screen.
>> Oh, I see.
>> Yeah, we have to move that over.
Did you see that [inaudible] ?
>> I did now, let me take a look.
Sorry, I didn't catch it,
but I'll go full screen here and Emma,
just reintroduce this slide and we'll be okay.
>> Hit it.
>> Should we start off from the beginning?
>> No, I'll ask you the question.
For those who don't know what ONNX is or what the ONNX Runtime,
how would you explain it to them?
>> Yeah, so for people who are
not familiar with ONNX and ONNX Runtime,
here is a graph which explains what a role ONNX and
ONNX Runtime play in the AI Software Stack.
There are a lot of front end framework
outside like PyTorch and TensorFlow.
Each framework has its own format of the model graph.
The graph serves as IR or
shorter for Intermediate Representations.
ONNX stands for Open Neural Network Exchange is
a standard format for representing
on both [inaudible] and the traditional machine learning models.
As our standard IR,
it allows model on-trend on different frameworks,
it can be converted to your ONNX format.
Then ONNX model can be inferenced with
ONNX runtime on which leverage a variety
of hardware accelerators to get at
their optimal performance for different hardware.
As you can see,
we build our comma inference stag on its
ONNX runtime for Microsoft,
which is a highly optimized across the machine learning of
inference lifecycle of a wider range of Cloud and edge device.
>> This is really cool and I really liked this.
ONNX feels like it's a PDF format
for models and ONNX runtime is what runs the actual thing.
My question to you is,
how do people run these ONNX models?
How do they operationalize them?
>> Yeah, that's a good question.
Since the beauty of ONNX is our framework into our probability,
you can turn on model from any frameworks and services,
which is of course, ONNX format.
This model can be combined either to
archivists are using ONNX converter tools,
they're not inference with ONNX runtime for
high performance on different hardware.
As you can see, ONNX Converters and
ONNX Runtime are the major pieces in this workflow.
>> This is really cool, who's using this?
Are there people using this already?
>> Actually, I have some more content.
>> Okay. Let's go full screen,
Cam. Okay keep going, sorry.
>> Yeah, in terms of model commercial,
there are a lot of
popular frameworks that are so-called commercial for ONNX.
For some of these, like PyTorch,
ONNX format, export a spewed in natively,
and for others like TensorFlow or Carers are there are
separate installable package that a can't handle the conversion.
ONNX runtime is our high-performance inference engine
until around ONNX models.
It is suppose CPU, GPU,
and VPU offers APIs covering a variety of languages,
including Python, C++, and Java Script and others.
It's also a cross platform and
compatible on the new Windows and Mac.
You get easily plug it into your existing, server now pipeline.
The highlight of ONNX runtime is open and extensible
architecture for easy optimizing
and accelerating machine learning inference.
It a marriage viewed in graph optimizations,
and on a lot of
hardware cellular ration capabilities for different hardware.
>> This is really cool because it feels like it's an end to
end thing for machine-learning models.
Are there many people using this already?
>> Yeah. ONNX Runtime has been
integrated into a variety of platforms.
Intimal platforms like our web ML, Azure ML,
and the some external platforms are like Oracle, navy towel.
Allow ONNX runtime runs on millions of devices and it's
processing billions of requests in our production everyday,
and then from production point of view,
ONNX Runtime has been now powering many flagship products like
Office collagen services across all kinds of machinery models,
including CM, RM and the transformers.
For model ship in our production,
we have observed with our two points,
my average performance improvement
compared to the previous solution.
>> This is cool. I've already known about ONNX for some time.
I'm sure there's people that have known about
ONNX and then ONNX Runtime.
What are some of the newest features
that you're most excited about?
>> Yeah, thanks for asking that.
Previously, we have been working with
first-party measuring for Cloud side inference with ONNX Runtime.
Nowadays, on device inference becomes appealing due to
growing awareness of their privacy and the data transfer cost.
We get more and more requirements for
extending existing scenarios of from Cloud to client.
But the limited memory and computer resource
makes this very challenge to run heavy models on client.
To solve this problem,
we're glad to enable INT8 quantization in ONNX Runtime for CPU.
On quantization approximates floating-point numbers with
lower bits with numbers
automatically reduce the model size memory for the print,
as well as accelerating and performance.
Quantization might introduce accuracy loss
because low bits limit the precision and the range of the values.
You're needed to make the trade off on your scenario.
ONNX Runtime use INT8 quantization.
Compared to FP32,
INT8 representations reduce storage by four times.
In terms of inference performance,
integer computation is more efficient than a floating point math.
>> This is really interesting. So basically,
what you're doing is you're snapping
floating-point numbers to INT8,
which is going to make it better for space and for computing.
But how much of a speedup do we get doing this?
Is it a lot better?
>> Yeah, much better.
We have a benchmark we can go through.
So we applied ONNX Runtime quantization
on hacking phase transformer models,
and observed a very promising result
on both performance and the model size.
Performance varies with hardware.
We benchmark performance for BERT base,
RoBERTa base and GPT-2 on two cans of CPU machines with
AVX2 instruction and AVX of
512 and we underline instructions are separately.
So the orange in the models are [inaudible] PyTorch,
we can model them to ONNX then apply ONNX Runtime quantization.
On the performance gain compared to the orange
in our model is pretty significant.
We saw that ONNX Runtime INT8 quantization can
accelerate inference performance by
about three times on a big machine.
On smaller balanced steel loadable performance improvements
on AVX2 machine, around 60 percent.
In terms of model size,
ONNX Runtime is able to
quantize more layers and
reduce the model size by almost four times.
The final model is
about half as much as the quantize the PyTorch model.
Accuracy is also important in the evaluation.
Smaller and faster is great but we also need
to make sure the model is returning good result.
Given the accuracy is Pascal specific.
We took advantage on the BERT model,
for MPRC task and for accuracy benchmark.
MPRC task is a common NLP task
for non-bridging on parallel classification.
This model is fine tuned using bolt
based model in a Hugging Face transformer for MPRC tasks.
Compared to PyTorch quantization,
or even with a small model ONNX Runtime quantization
surely they seem accuracy and have a slightly higher F1 score.
In this task, the INT8 model can
preserve the similar accuracy as FP32 models.
>> This is really cool because quantization
apparently makes the model much
smaller but we still maintain good accuracy.
Overall, it's cool. I would love to see how this
actually works in practice.
Do you have something you can show us to get
a sense for how we would implement something like this?
>> Yeah. So let me take this notebook to
demonstrate the end to end workflow
of ONNX Runtime INT8 quantization.
You can use PyTorch,
TensorFlow to turn our model with FP32 format.
Then convert it to ONNX with our converter tools.
The converted model is STR FP32 format.
To get INT8 model,
we provide a quantization tool to
quantize ONNX model from FP32 to INT8.
Once you get a quantized model,
you can inference it in
ONNX Runtime the same way you would normally take.
So according to a model characteristics,
there are three modes enabled in
a quantization tool: dynamic quantization,
standard of quantization and quantization while training.
This notebook shows how to quantize Hugging Face,
both model trend with PyTorch using
dynamic quantization to reduce
the model size and speed up inference.
To save some time,
I've already run the notebook ahead of time.
Firstly, we need to set up environment
by installing all the necessary packages,
including PyTorch, ONNX Runtime and the Hugging Face transformer.
Then we can fine-tune Hugging Face BERT model
for MPRC task or with PyTorch,
by using GLUE data set.
So you can download the data set and run the fine tunes deck.
PyTorch also published this model of Directory usage.
Here we can download the model directory.
In order to get baseline,
we are wrong on the model or with PyTorch quantization firstly.
Firstly, we set some global configurations
and then we apply on PyTorch quantization.
Here is the model size of quantized PyTorch model.
In terms of performance evaluation,
we use the Tokenize and evaluation function from Hugging Face.
Here is performance and
the accuracy results with PyTorch quantized model.
>> That's cool. This is a PyTorch quantize model.
We saw it go from 400 down to 170 something,
but we haven't done the ONNX
Runtime quantization model is that right?
>> Yes.
>> That's okay, let's take a look at that.
>> Yeah, so let's go to the key part of ONNX Runtime quantization.
As shown in this diagram,
there are three steps on [inaudible] ,
the orange model to
ONNX FP32 model on quantize eight and with our quantization tool,
then inference into aid modal with ONNX runtime.
Since PyTorch support ONNX format latently,
it provides a built in export API.
It shows here for exporting PyTorch model to ONNX format.
Together the optimal performance with ONNX runtime for
FP32 model that is on offline transformer optimization tool.
You can now reach to get or further optimize the bottom model.
Here is the code how to use our optimization tool.
>> That optimizer is doing the quantization for us to [inaudible] .
>> No, we haven't go to the quantization step yet.
Yeah, this process is still based on FP32 from our datatype.
>> I see, so there is an optimization step before we even do
the quantization step that is
provided in the ONNX Runtime. Is that right?
>> Yeah, but I will point out on this step,
that is only for transformer models.
>> Got it.
>> Other types of models,
you don't need to run this step.
After this step, we can apply
quantization for these for the optimized model.
Here you can call on
its runtime quantization API to
quantize sine model into eight format.
Here you can enter the model size after ONNX runtime quantization.
>> Yeah, and that's like a quarter of the size.
That it was originally, which is pretty cool.
>> Yeah. Okay. Then let's
evaluate the performance and accuracy with
ONNX Runtime using the same way we
normally inference ONNX FP32 model.
You can create inference session always there is session options.
Then you call our session [inaudible] onto score this model.
Here is accuracy and the performance result.
Always ONNX Runtime quantization.
>> Yeah, it's roughly the same for both, which is really cool.
>> Yeah, so actually we can compel on
PyTorch result always ONNX Runtime result.
You can see for model signs,
here we can get better results.
ONNX Runtime can achieve with even smaller model.
For accuracy of ONNX Runtime quantization
is slightly better than on PyTorch quantization on F1 score.
>> That's cool.
>> Yeah.
>> Sorry. Overall we were able to get
a better size model with
the quantization from the ONNX Runtime than with the PyTorch one.
It seems like in this case,
the F1 score was even better. That's cool.
Here's another question as we move along.
Are there any other new features
that you're actually really excited
about in the ONNX Runtime that
you'd like to share before we call it a show?
>> Yeah, there's another exciting feature I would like to share.
As we mentioned, there is a strict constraint of
these core footprint and
the inner memory size for on-device inference,
especially on smartphone and Edge devices.
Or with quantization, we can reduce the model size.
Either way we can reduce the Runtime size as well.
ONNX Runtime model is a new feature on minimizing
the Runtime size for usage in
a mobile and embedding the scenarios.
On this capability extends ONNX Runtime to support
optimize our model inference from Cloud to client.
Now this feature is available in
preview with ONNX Runtime release 1.5.
There are two major techniques enabled for ONNX Runtime mobile.
One is introducing new optimized format
we call ONNX Runtime format,
which support our model inference with
ONNX Runtime with less dependencies.
The other one is building ONNX Runtime with
operators only needed by predefined models,
so that at the Runtime size can be further reduced
by getting rid off those unused operators for your scenario.
>> This is really cool because not only are you trying to
reduce the size of the models,
but you're also reducing the size of the actual Runtime.
Because you're only going to export the operations that you need.
For example, if you're doing like only [inaudible] ,
you don't need to do a softmax,
that's actually really cool, because now, you have less
of a Runtime footprint, which is really cool.
How much does this actually help with speed, etc?
>> The speed would not to be impact on.
>> I mean size because it's the size matter, sorry.
Because obviously, it's going to do
the multiplications and do the functions,
but you want to squeeze
as much as you can in a little bit of space.
How much of an improvement are we going to
see with space. Sorry about that.
>> Yeah. Here is the benchmark of
the Runtime size with ONNX Runtime mobile.
The size of the Runtime package varies from
the number of models you want to support.
For different platforms, here is the result.
You can see they compress the size
of core ONNX Runtime mobile package
can achieve a less than 300 kilobyte
on both Android and iOS by default.
Which a makes own device inference,
more feasible to meet a straight
to a memory and the [inaudible] .
>> Did you say, 300 kilobytes?
>> Yeah.
>> That's like the size of a picture.
I mean that's very impressive, that's really cool.
So obviously, we spent a lot of time
talking about the ONNX and the ONNX Runtime.
I know we're running short on time.
Where can people go to find out more and maybe try this out?
>> ONNX Runtime is open source project.
All the tutorials and examples are available in the GitHub repo.
Feel free to take a look and have a try.
>> Awesome. Well, thank you so much, Emma, for spending
some time with us and thank you so much for watching.
We were learning all about ONNX and the
ONNX Runtime, two amazing features.
Thanks so much for watching and we'll see you next time.
Take care.
[MUSIC]