>> You're not going to want miss
this episode of the AI show
where we're getting into the nitty-gritty
of the ONNX model format
as well as the ONNX Runtime.
It's pretty amazing.
And in the end, you can actually run your own AI pretty quickly.
Make sure you tune in.
[MUSIC]
>> Hello, welcome to this episode of the AI show.
We're going to learn all about ONNX,
the Open Neural Network Exchange.
I'm pretty excited to be here with Venitra.
How are you doing, my friend?
>> I'm doing great. Thank you, Seth.
>> Fantastic. What do you do?
>> So I work on the ONNX engineering team as part
of the AI Frameworks team here at Microsoft.
>> So I've heard about ONNX and I've even converted stuff to ONNX,
but can you explain to our audience what exactly is ONNX?
>> Right. So ONNX actually stands for
the Open Neural Network Exchange and it's
specifically a format to represent deep learning models.
>> Okay.
>> The benefit of using ONNX is that you're able to
translate from a variety of frameworks into ONNX,
and it's the standard for deep-learning libraries,
it specifies a set of operators.
>> So I understand like,
so basically if I like PyTorch or someone else likes TensorFlow,
the outputs are obviously different,
ONNX allows you to have this format where it's all common.
So I get that part but why would someone want to do that,
what's the benefit of doing that?
>> So the benefit of doing that is multifold.
One of them is that ONNX is able to
leverage graph optimizations and you're
able to use different runtime inference
frameworks to take advantage of those.
>> I see, I see. So you're saying that, okay.
So now I'm understanding this because
like for some reason I'm like,
"Okay, it's all the same format.
I can work with other developers."
But now you're saying from the inference perspective,
if you have something in a common format
and everyone agrees to it,
then hardware vendors can optimize for that, number one.
Then number two, as you're doing conversions,
you can find errors and maybe graph things that can
be simplified and is this
something that you do when you're converting as well?
>> Yeah. So there is an optimizer in
the conversion tools but the main advantage to using
ONNX is you're able to take
advantage of whichever hardware optimization comes next.
So you're able to translate between these frameworks with ease,
and then choose whichever inference engine you would like.
>> So let's talk about the actual converter.
This is actually super helpful because now I understand it's not,
so I think of it like as the PDF
of deep learning which is kind of cool.
But it's only cool if there's a lot of
different PDF readers and
you're saying that now that we have a format,
then a lot of vendors can take advantage of that.
Is it hard to go from what I'm already doing into ONNX,
and that's the part that maybe I'm asking
for help because someone might be thinking,
"This is going to add a little bit to my workflow."
Is there something we can do to help with that?
>> Yeah. So again,
the advantage of ONNX is that
it's supported by a large amount of industry partners.
That means that a lot of these frameworks
have built-in exporters for example,
PyTorch and a few others,
and some of the other ones are able to
leverage converters in on XML tools.
So scikit-learn, TensorFlow, Keras to ONNX,
all of these are just simple covert commands
that convert your model,
that's a saved protobuf file or something you load in
whichever framework of your choice into the ONNX model format.
>> I see. So you're saying it's as easy
as making an extra caller in
your framework of choice
or loading up some tool that does the conversion?
>> Exactly.
>> So there's not really a lot of
extra work that you have to do hopefully.
>> Yes. Not at all.
>> So what if someone already has
this and they want to experiment with converting to ONNX,
is there something we could do to help even then?
>> Right. So if someone already
has ONNX and they want to experiment,
we've created this ONNX ecosystem Docker file
where you have all the ONNX converters loaded,
you have ONNX, you have all of the different frameworks,
and you also have ONNX runtime.
>> It's pretty easy to do.
>> Yeah, because I set up my Conda environment in
a very special way and I don't want to install,
I tried installing some of these stuff once and
I was missing something or something rather happened.
But you're saying now with a Docker file,
you can just use it. Can you show us how to do that?
>> Yes, of course. So let's take a quick look here.
This Docker file has already been uploaded to
Docker Hub and what we're going to do here is,
right now I've already pulled this Docker file.
This is all you have to do. You just have to have Docker,
and then after that you can run
this command and it launches the Jupyter Notebook server.
>> Okay. So this is where I was confused, I'm like,
if you run this Docker, is it going to look
for all of your [inaudible]?
No, it's loading a Jupyter server
where you can actually do some work.
>> Exactly.
>> Awesome.
>> So in this Jupyter server,
you can see that there are
some converter scripts from all of these different frameworks,
and all of these environments have already been loaded.
So if you have a model file,
it's as easy as uploading it and here,
I've recently just uploaded
an ML model file which is an Apple Core ML.
>> Right.
>> Machine learning framework file,
and all that I would have to do,
go through the script, in this case,
all of the imports already exist,
change the name of the model which is
going to be "DocumentClassification".
In this case, this model just takes a set of
articles and classifies what kind of document it is.
>> Got it.
>> Change our model name for the output file,
finish that, and we have our model being saved.
So now we open up.
It's as easy as Pip.
>> That's pretty cool. The thing
I like about this the most is that,
every time I do something in Python and
I need a Pip install or Conda install,
I'm like, "I don't want to mess up what I'm doing."
With this Dockerfile, you're able to pull this image down,
run the container, and then experiment with
the ONNX stuff yourself and download it if you want, all right?
>> Exactly. So we can then
confirm that the model works by running through this checker.
>> This is also in the Dockerfile.
>> This is all in the Dockerfile.
>> Okay.
>> So your entire environment
is installed, you're welcome to use it.
You can check that our model is checked,
and see we're passing in the same name as our previous output.
We can start an ONNX Runtime inference session.
>> Okay.
>> So this is something else I
wanted to talk about. I'm not too sure.
>> Yeah, I do want to talk about this because this is interesting.
Because I actually did something a little bit ago where I
ran an inference session in C sharp, which is kind of cool.
Because now that we have this exchange format,
everyone's agreed on what it looks like,
you can run these inference sessions in
any type of [inaudible] that you want. Is that right?
>> Exactly. The benefit of it is,
it actually takes advantage of
hardware optimizations from partners like Nvidia and Intel,
to make inference for ONNX models even faster.
>> That's amazing. So now, I'm starting to get the full story.
I think initially what I did,
I was thinking of this like a,
but now it's just a format. I said that this is good.
Now everyone has the same format.
But now you're saying, not only does that
enable hardware people to do like,
"Oh, now that we know this is the standard,
then we can make our hardware super-fast for that."
Number one, it also allows people that make frameworks or tools
or whatever ingest these ONNX models in a predictable way.
>> Right.
>> They're irrespective of length. Because I'm used to doing
all this stuff in Python like maybe some of you are.
But I've been able to load up models directly in C sharp,
and it ran beautifully.
All I had to do was look at the model,
know what was going in and what was coming out, and it was great.
So is the Runtime something that we provide,
or where does Runtime come from?
Is everyone making their own Runtime?
>> Right. So the ONNX Runtime is actually from Microsoft itself.
It's meant to host three main ideas.
The ONNX Runtime is important
because it's an open source scoring engine for ONNX models.
The benefit to using it,
is that it's guaranteed to run any ONNX model.
Right now, it's compatible with the most recent release,
which is ONNX 1.4 offset 10.
It's high performance and that it
often gains even more performance
than native Runtimes that you're running on ONNX.
So if you upload it in
ONNX model in some other frameworks or runtimes,
you would see a better performance
with ONNX Runtime for the most part.
>> I see.
>> It's cross-platform. Meaning that,
all you would have to do to install
is just a simple Pip install on
Mac or on Linux or on Windows.
>> You'd mentioned something interesting and hopefully,
you can get us into this.
You talked about the version,
but then you talked about the offset.
What is the difference between
the ONNX Runtime version or the ONNX offset?
Can you describe what those differences are?
>> Right. ONNX versions and
operator sets are essentially similar things.
There's continual development from Microsoft, Facebook,
and a couple of the other large industry partners
that continually contribute to the ONNX spec.
Every time that gets refreshed,
every new version that comes out,
they update the operator set.
>> Is it like a matrix multiply operator?
Or like a log
of something operator? Is that what we're talking about?
>> Yeah, exactly. So we're talking about
all the different operators that can be
represented in deep-learning lessons.
Usually, what happens is that
the ONNX spec has to represent a set of operators.
In those operators, you can
represent the translations from other libraries.
>> I see.
>> So from Core ML, from TensorFlow, from various others,
you can convert to ONNX,
and then convert from ONNX back to any other library.
>> I see. So what's happening is,
there's a set of operations that,
like for example, W transpose X
plus B is your standard dense layer.
>> Yeah.
>> That's going to be a single operator
inside of the ONNX definition.
So whenever it sees a dense layer
in either Keras or inside of PyTorch,
is to be like this operator.
It turns out that the offset changes with every version,
and there's more operations that are added.
>> Right.
>> I got it. Okay, that makes sense.
Because for example in sequence models,
there might be a different operation
as opposed to just a dense layer.
>> Exactly.
>> Okay. So now, I'm understanding
and thank goodness you're doing that.
So anything else you want to add about ONNX?
>> It's great. There's a pretty wide industry adoption.
We're excited to incorporate it with ONNX Runtime.
ONNX Runtime is Microsoft's own inference engine,
but you can deploy ONNX models on a variety of devices.
>> My understanding is that the ONNX Runtime is
part of the whole Windows ecosystem. Is that right?
>> That's correct. ONNX Runtime is included
on every single machine that is running Windows.
>> That's cool.
>> Yeah. Over one billion devices now currently have ONNX Runtime.
The benefit of using it is that,
it leverages execution providers speedups.
Let's say you have an Intel chip that's
running MKL-DNN or end graph,
or an NVIDIA processor that has TensorRT or GPU capabilities.
ONNX Runtime can be specified to leverage
those speedups and make your model run faster.
We've seen speedups kind of across
the board with a lot of different frameworks.
But also, especially in comparison to
native Runtimes like TensorFlow, PyTorch, etc.
>> Awesome. Now, let's hopefully get into the nitty-gritty.
Is there an example where this is being leveraged in
a real actual problem where you're seeing substantial benefits?
>> Yes. I'm really excited to bring this amazing story here today.
The Bing multimedia team is responsible for image search.
Specifically, you can understand that there's some queries
that are really hard to search for images.
So for example, pencil erasers that look like tools.
Right? How would you be able to identify
that we're looking for
erasers that are not tools, that are pencils,
and that all of these images are being mapped to
the same semantic embedding space as our Word's.
>> So let's go to the screen so people can see.
They're out there watching.
So you're saying that when I'm searching for something,
I know how to word search stuff,
but we're searching for an image of something.
There's certain kinds of
image searches that are very difficult. This is an example of one.
>> Right. It definitely is. There are a couple of others.
For example, if you take a look at this image
over here, how would you describe it?
>> A big whale and a man swimming, I guess.
>> Exactly. But you would also expect it to
come up with a variety of others,
for example, endangered animals,
blue desktop backgrounds, National Geographic.
>> Oh yeah, that's a problem.
>> Right? So how do you express that
this image has a man and a whale,
a whale is an endangered animal,
and it's also mostly blue.
That's a lot of information to be put in one vector.
>> Definitely.
>> So a solution that the Bing multimedia team came up with,
is called Semantic Precise Image Search.
>> Okay.
>> It's in relation to project of Dirk,
which is a deep image ranking stack.
This is thanks to the Bing Multimedia team,
and Edward [inaudible] team.
So if we take a look at this, we have a query.
In this case, gold butterflies HD background,
which is what the user types in.
We have the text on the page which is,
let's say something like HD wallpapers
golden butterfly, something like that.
We also have this image.
So we want to be able to represent
this image in the same vector space
as the queries and have
some semantic embedding in terms of this model.
>> So it's just like, for those that are watching,
is basically a way of taking the image,
embedding it into a vector space that you can
query against in the same vector space as the query.
>> Yes.
>> So basically, you're trying to put these in the same space,
and then whichever ones closest, we pick those ones out?
>> Exactly.
>> Okay. Cool. Now, I imagine that then you'd have to,
for every image that's ever uploaded into Bing,
you'd have to put it into this latent space.
>> Exactly. Which is a daunting concept.
>> Yeah, there's a lot of pictures that are uploaded.
>> Not only that, you'd have to process it in terms of the query,
in terms of the text on the page,
and you'd have to find which images
make the most sense and then order them.
>> It's a lot of stuff.
>> Well, it's a large model, large embedding.
>> So how does ONNX help here?
>> ONNX Runtime helps by providing a speedup for this model.
Ideally, what happens is that
this specific model came to us with a cafe model.
We converted it to the ONNX model format,
and deployed it with ONNX Runtime for two times speedup.
So we reduce the latency by 50 percent on
CPUs using Intel's MKL-DNN as an execution provider.
With a model like this,
especially for Bing, it's already heavily optimized.
So these kind of out-of-the-box optimizations
that are now in production have really enabled
Bing to be able to search in this way.
>> Because now, there's
two models that have to be run simultaneously.
First, when the image is uploaded,
so you can save it in the latent space,
and then the model that does the comparison
between the query and the image in the latent space.
So you have two models that need to be running all the time.
>> Right.
>> So you've seen a hundred,
like is two times speedup is what you've seen?
>> Yeah. Two times speedup.
Then across the board,
we've seen a lot of other really great examples
with even more of a speedup.
>> I see. Okay.
>> So ONNX Runtime is really taking
advantage of this first-party equals
third-party attitude that our senior leadership team
has started to employ.
The idea here is,
you can use ONNX Runtime models,
and it is being used at Microsoft across all walks of,
>> Of just models everywhere, right?
>> Exactly.
>> So here's the thing.
Where can people go to find out more about this.
Honestly, I feel a lot better,
that I feel like I'm rounded out a little bit more on
the ONNX model and the Runtime.
Where can people go to find out more
or maybe even play with some of these things?
>> Right. So one of the best ways
to play with these things is by using Azure Machine Learning.
So we can now deploy
pre-trained ONNX models using ONNX Runtime for inference,
using Azure Machine Learning.
There are some great tutorials,
and we'll include some links at the bottom below or on the screen.
>> Hopefully, there are some on the screen right
now that we can look at.
>> So if you go to aka.ms/onnxRuntime,
you'll see this great tutorial about how to deploy
ONNX Runtime models using Azure ML for inference.
You can always go to GitHub.
We have open-sourced.
Both of these, both our contributions to ONNX as
well as ONNX Runtime as it's growing engine.
>> So what if I don't have any models though?
Is there a place where I can get a model to play with?
>> Exactly. There is.
I'm so glad you asked.
If we go to ONNX website,
we'll see here there are a couple of links.
One is "Tutorials", which
allow you to look at some conversion tutorials.
Or you can just go to the ONNX Docker page for those.
If you want it pre-trained models,
which it seems like you're asking for,
you can go to the ONNX "Model Zoo".
In here, what you'll see is a set of
state-of-the art models all categorized in various categories,
and you can just pull these immediate models,
deploy them using ONNX Runtime
and hopefully on Azure Machine Learning.
>> This is pretty cool.
Basically, everyone out there watching
right now can go to the Model Zoo,
download it, download the Docker container,
run inference against it,
and have it work right away.
>> Yes.
>> Okay. This is pretty amazing.
I'm pretty excited about it.
Thanks so much. Venitra for this.
This has been amazing.
I've had a good time because I've learned
a ton about the ONNX model.
Thanks much again for your time.
Thanks so much for watching and we will see you next time.
Take care.
>> Thank you, sir.
>> [MUSIC]