>> Make sure to stay tuned to
this episode of the AI show where Manash Goswami,
Principal Program Manager for AI Frameworks,
tells us all about the ONNX Runtime,
what's new, and even some amazing hardware announcements.
Hint: there's partners involved,
FPGAs, special guests. Enjoy.
[MUSIC]
>> Hello, and welcome to this
on-demand session about ONNX Runtime.
Today, I'm going to talk about our work with
Xilinx to integrate ONNX Runtime with their FPGA solution.
I'm Manash Goswami, a Program
Manager in Microsoft's AI Frameworks team.
The Open Neural Network Exchange or ONNX is
an open source format for representing ML models,
both traditional ML and DNNs.
There are converters available to convert from
different training frameworks to ONNX.
By having a common model format,
we provide data science teams with the freedom to
choose their tools for development and training,
while ensuring that their model can be represented in
a common way that can be easily deployed and executed.
We achieve this by using
the ONNX Runtime inference engine to execute ONNX models.
On the other hand,
having a Standard Model format
allows hardware platform makers to focus on
innovation for optimizing
neural network computations on their platform.
This common format for the machine learning model
and the ONNX Runtime inference engine allows us to decouple
the data science world of model development from
the fragmented hardware execution environments
in the Cloud and the Edge.
ONNX Runtime is the open source inference engine
to execute ONNX models.
It was designed with a focus on performance, extensibility,
and scalability in order to support heavy workloads and
traffic as a production solution on the Cloud and Edge.
The runtime is optimized to implement
all the operators defined in the ONNX spec,
and therefore, it can run any valid ONNX model.
The ONNX Runtime supports the notion of
a custom operator that can run alongside official ONNX operators,
which means that as operators get introduced into the spec,
the ONNX Runtime is capable of running then even
before the spec is standardized.
The runtime is backwards and forwards compatible so
that as models are updated to newer versions of the offset,
the ONNX Runtime can execute them.
The ONNX random architecture supports
this extensible login interface
called the execution provider interface,
and this is used to run
models on different hardware optimized environments.
We will talk about the
ONNX Runtime Execution provider in a subsequent slide.
Finally, ONNX Runtime is integrated inside WinML,
which is the ML execution environment in Windows 10.
Today, we are releasing an update for
the ONNX Runtime Inference Engine version
1.3 This update includes support for ONNX spec version 1.7.
We are announcing the general availability for
the direct ML execution provider on Windows.
This provides acceleration for
ONNX model inference across
different platforms on Windows devices.
We are releasing a number of
other execution providers in preview to support
ONNX models across different platforms expanding our ecosystem.
The Python package for
ARM64 devices is going to be published starting this release.
Additional details about the
version 1.3 release can be found in the ONNX Runtime GitHub page.
The ONNX Runtime enables ML model execution on
different hardware platforms through
the execution provider interface.
This allows hardware platforms to
integrate with the ONNX Runtime by building
an execution provider for that specific compute endpoint.
We have a number of execution providers
from our hardware ecosystem partners across
the breadth of options from CPUs
to GPUs and custom neural accelerators.
This execution providers are used to execute models in the Cloud,
as well as on Edge devices.
Today, we are happy to announce a new partner
in our growing hardware ecosystem for ONNX Runtime.
We have partnered with Xilinx to build
the execution provider for the Vitis AI software library.
This execution provider is used to execute
ONNX models on the Xilinx U250 FPGA platform.
Developers can now take advantage of
this execution provider to build and
execute models on this Xilinx FPGA platform.
The Vitis AI software library is
the Xilinx development platform
for AI inference on Xilinx hardware platforms.
The U250 FPGA is available in private preview in Azure
as the NP VMs queue for
users to use with the Vitis AI software stack.
The Vitis AI toolkit consists of
optimized IP tools, libraries, models,
and example designs for developers to use with the FPGAs.
This allows FPGA developers to infuse
AI inferencing and acceleration on the FPGA.
Azure provides a wide variety of
infrastructure options specialized for your big compute workloads.
The Azure NP VM with Xilinx,
LVO U250 FPGA accelerator is in private preview now.
This queue offers from one to four Xilinx U250 FPGA devices as
an Azure VM backed by
powerful Xeon platinum CPU cores and fast NVMe-based storage.
The NP series will enable true lift and shift and
single dark-end development of
FPGA applications for general purpose Cloud applications.
This VM is based on
a board and software ecosystem for the Xilinx U250 card.
The Vitis software stack and
ONNX Runtime will run on this Azure VM,
as well as on other on-prem and Edge devices with the Xilinx FPGA.
This solution enables the bleeding edge
of accelerator development to
harness the power of Cloud without additional development costs.
Xilinx and Microsoft collaborated with Peakspeed,
an innovative startup that provides
geospatial analytic solutions to
integrate ONNX Runtime and the Vitis AI stack.
This solution seamlessly integrates
Peakspeed's deep-learning object classification models
and uses ONNX Runtime with
the Vitis AI execution provider to accelerate
the pipeline on the new Azure NP Series FPGA platform.
To discuss more about the solution,
please welcome Oscar Kramer, Peakspeed's chief scientists.
>> Thanks Manash. Here at Peakspeed,
we're very excited to be one of the first users of
Microsoft's new NP Series Azure VMs
for high-performance computing.
We've been focused on building
the world's fastest geospatial imaging solutions,
and Azure NP will enable us to meet some critical demands.
Today, I'll be showing Peakspeed's TrueView
and identify applications that
leverage the power of the FPGA for
performing compute intensive operations.
Here, we're using Esri's ArcGIS Pro application to illustrate.
This image over Estes Park in
the Colorado Rockies was provided by Maxar.
This area of about 100 square kilometers has high terrain relief,
resulting in significant distortions
due to the sensor's perspective.
Let's fly over to the Stanley Hotel
and look at this curved road here.
We can see a big offset between the map and the image.
To get the image to line up,
it needs to undergo a geospatial correction or
orthorectification that resamples every pixel
from the satellite perspective to a map projection.
Until now, this was being performed on CPUs, but Peakspeed,
using the FPGA on
an Azure NP instance has
managed to accelerate that process considerably.
Let's look at the performance numbers.
Here are Peakspeed's DevOps,
pipeline outputs that we use for bench-marking.
These numbers compare TrueView running on an Azure NP hosting
the Xilinx U250 FPGA against
the same algorithm running on a Xeon platinum CPU.
We're looking at 4 minutes 21 seconds on
the CPU versus only 1.7 seconds on the FPGA.
Let's go back to ArcGIS Pro
and look at the output produced by TrueView.
Here's a same image after orthorectification.
We'll zoom into the same spot,
and you can see that now,
it lines up perfectly with a base map.
Let's zoom back out and switch between the two images.
This gives you an idea of
the highly non-linear transformation involved.
Now that we have an ortho image,
we're ready for object detection.
This is where the ONNX Runtime running
Xilinx's Vitis AI library come in.
We'll be using Peakspeed's identify training models
to detect the cars in the satellite image.
The production pipeline is implemented in a Python script,
and we can easily flip between CPU, FPGA,
or GPU via a runtime parameter,
depending on our needs.
In this case, using an Azure NP instance,
customers will see a significant savings in
TCO and faster response times.
The ONNX Runtime enables Peakspeed to
target a variety of acceleration platforms in Azure,
and that helps us provide the best value to our customers.
Thanks, and back to you, Manash.
>> Thanks, Oscar. Peakspeed was able to
deploy their deep learning models on the Azure NP VM,
hence a great performance.
The YOLOv3 model showed improvements on
the VM relative to GPU instances,
and the end-to-end geospatial analytics pipeline
ran 4x faster than on a GPU.
ISPs and customers can now infuse
their applications with deep-learning using
ONNX Runtime and the Vitis AI stack and
accelerate on the FPGA endpoints in Azure,
as well as on-prem with the Xilinx U250 hardware.
Developers can take advantage of this integration by
building the Vitis AI execution provider
with ONNX Runtime from source.
You can do that from the ONNX Runtime, GitHub master repo.
Additionally, very soon,
Xilinx will release a VM image in Azure Marketplace with
all the pre-built software libraries integrated in
one place for easy deployment into the Azure NP VM.
Thank you for listening,
and I hope you have a great day.
[MUSIC]