Hey, I'm Ragesh and this is Edge Innovation Series for Developers!
This is a series where we give you updates on tools and resources from
Intel to help AI developers like you drive faster innovation at the edge.
Swapping out our usual host, Steph. Today, we have with us Devang Aggarwal. He is a product manager
with Intel’s OpenVINO™ AI framework engineering team and is doing some work around deep learning
and computer vision and will be talking with us today about OpenVINO™ Execution Provider
for ONNX Runtime. Welcome to the show Devang! Hey everyone! Thanks for having me on this
episode, Ragesh. Super excited to be here! So Devang, many of our viewers watching
might be new to OpenVINO Execution Provider for ONNX Runtime, can you first help to set
the context by talking about ONNX runtime? Yes, ONNX Runtime is a cross-platform machine
learning model accelerator. It can be used with models from frameworks like PyTorch, Tensorflow or
Keras, TFLite, scikit-learn etc., to deploy deep learning models on range of Intel compute devices.
Typically, how ONNX Runtime itself works is that users import the onnxruntime library into their
code and start an inference session which is a runtime call pointing to the model location.
Users just have to provide some input data and ONNXRuntime will
run the inferencing and generate the output. Great! So, how can developers creating ONNX
models accelerate inference with OpenVINO™ toolkit? Can you introduce us to the product?
Sure! As we all know OpenVINO™ Toolkit does all the magic aimed at boosting the performance of
your Deep Learning models using some of the most advanced optimization techniques
specifically optimized for Intel® hardware. Now developers can leverage the power
of the OpenVINO™ toolkit through ONNX Runtime to accelerate inferencing of ONNX models.
So, how does OpenVINO Execution Provider actually work? Users continue to import
the onnxruntime library after installing our package from PyPi but now with a simple
modification to the Inference Session line of code, where they set the provider as OpenVINO
Execution Provider, they can utilize the power of OpenVINO while using the
ONNX Runtime API. All it takes is that simple modification to the line of code.
Intel and Microsoft actually joined hands to create this
product which delivers better inferencing performance on the same hardware compared
to generic acceleration on Intel® compute devices.
This is awesome. How can developers get started with this and would also love to see a demo.
We have created a few samples that show how you can get performance boost with OpenVINO
Execution Provider for ONNX Runtime. The Object detection sample you see here,
uses a YOLO Deep Learning ONNX Model from the ONNX Model Zoo. The sample involves presenting
a frame-by-frame video to ONNX Runtime, which uses the OpenVINO Execution Provider to run inference
on various Intel® hardware devices and perform object detection to detect up to 80 different
objects like a person, car, motorbike, and much much more from the COCO dataset.
The first step for us is to pip install the onnxruntime-openvino package as well
as all the other dependencies. Next, we go ahead and import the modules. After this,
comes the inferencing, where we set the provider as OpenVINOExecutionProvider for the inference
session. After setting the provider as OpenVINO, we can run the inference. Here is the output.
This is great to know. Is it also possible to share some insights into the architecture
diagram? I'm sure folks would love to see that. Sure, I can give a quick overview of the
architecture. So, how it works is that the ONNX model which is coming into the ONNX Runtime
Application gets converted into an In-Memory Graph representation and goes into a graph partitioner
or a get capability module. So, this module queries all the backends that are available,
and it asks which parts of the graph it can support. So, the providers that we have
enabled have to reply and say that these are the subgraphs that we support, and we have to provide
handles back to ONNX Runtime. When it’s time for the subgraphs to be executed, ONNX Runtime
will send back to the Execution Provider and it will do the inference and send the output back.
Thanks Devang – this is super cool to see. Thanks for being here today.
Thanks Ragesh Before we sign off today – do checkout
these two reference implementations from Intel® Edge Software Hub. First one is designed to detect
and track driver behavior and driver actions to ensure safe driving. And the second one is for
vehicle event recording to monitor the exterior of a vehicle and send events to the cloud dashboard.
And did you know we are publishing new articles on Medium? We are constantly
publishing new ones. Our Medium channel explores OpenVINO toolkit. We have very useful developer
guides and how-to articles. Check them out! Thanks for watching. See you next time. Bye.