CARLOS GARCIA SIERRE: Welcome everyone to
Microsoft build session.
My name is Carlos Garcia and I'm
the Product Manager for DeepStream.
Today, I have David Edwin from Fractal Analytics,
and Joe from Microsoft.
We're going to cover Azure-enabled Vision AI solutions
within NVIDIA for Enterprise and Jetson.
If I have to describe
the three most important challenges
for real-time streaming applications,
I will go with scalability,
multisensor, and driving automation systems.
Scalability, this is because we have billions
of transactions across millions of locations.
For multisensor, we have great solutions with
DeepStream where we have support for streaming cameras,
USB cameras, MPPI cameras,
lidar and other sensors.
However, there's a clear need for
additional sensors for environmental systems.
Driving automation, there's a clear need to integrate
into applications in industrial use cases.
We want to control PLCs
and robotic systems in an efficient way
that articulates all the information
gathered by the AI systems.
Let me expand a little bit on what I mean with
the opportunities around industrial inspection.
Up today, the real-time situational awareness
is what DeepStream enables to our developers.
We can have Queue management board,
different venues, biomarker automation or airports,
and drive through automation and improving
quality of deliberate and of course,
intelligent traffic systems in cities.
However, there's a new opportunity
around industrial inspection that requires
the deterministic systems and
use of multisensor capabilities.
This is unlocking a new opportunity for
use cases around quality control,
where there's a lot of important use cases
around making sure lines do not go down.
Starting and building a product in
an automated way and predictive maintenance.
To support these use cases,
we recently introduced a GXF-runtime accelerator
that in conjunction works with GStreamer.
That is the foundation of DeepStream.
In this case now you can have applications that
run in GStreamer at full speed
with cameras and millions of cameras in place
and running accelerated extensions
for real-time use cases.
Everything runs from edge,
workstations and Cloud and
benefits from all these solutions we have developed in
DeepStream for zero memory copy between extensions and
plug-ins and developing applications in
rock composer C++ or Python.
To give you an example of what I mean with these,
ambition this canonical representation
of an industrial use case.
You can have production line
with worker operating and collaborating with that robot
where all that information is captured by lidars and
cameras and streaming to
the DeepStream blogging shown in green.
But now you want to use that information with GXF
to control our PLC and maybe the robotic arm.
GXF allows you to do that functionality as it has
scheduling capabilities that can be
triggered by different scenarios.
Finally, one important advancement in
NVIDIA is the introduction NVIDIA AI enterprise.
This allows us to bring
AI enterprise grade support for their deployments.
There is a capabilities for retraining your model,
synthetic data generation,
and optimizing your models without.
All the applications can be deployed with
Triton Inference, DeepStream or Riva.
Now I'll pass it to David so they can talk
about the DeepStream accelerator.
DAVID BROWN: Hi everyone.
I'm David Brown.
I work closely with Edwin at Fractal.
We're a consulting firm and we helped
build the Azure DeepStream accelerator.
This is a package of code available on
GitHub that enables a custom model parsing path,
as well as 30 pre-built NVIDIA models to
run on Azure with a click to deploy button.
This extends the Azure control plane into the Edge,
enabling customers and partners to easily and
simply deploy what Carlos
has talked about onto Azure hardware.
Our custom paths also support edge hardware,
meaning we can take
Jetson devices as well as Azure stack edge
devices and enable
supportive those devices through this architecture.
This allows us to take advantage of
all of the software that Carlos had mentioned,
particularly NVIDIA TAO toolkit,
as well as all of the frameworks
around DeepStream, including GStreamer,
in order to scale
Cloud analytics to the edge
and enable seamless interoperability.
What this enables is a host of use cases.
These use cases scale widely across industries,
particularly industrial
manufacturing and quality control.
Use cases are exciting for us
because of the recent releases
that Carlos had mentioned
earlier around industrial inspection automation.
Higher fidelity cameras are supported with
the latest version of Jetson
and DeepStream than ever before,
meaning we can do higher fidelity industrial automation,
leveraging vision AI than has ever been done in the past.
This in turn, lights up a whole range of possibilities,
including NVIDIA's omnivorous suite,
as we create integrated digital twins
that are operating off of
manufacturing processes in order
to perform yield optimization tasks,
taking waste and turning it into good product.
This, combined with
the physics simulation capabilities of omniverse,
lights up a whole new suite of AI,
specifically deep reinforcement learning,
as it allows for us to
take defect detection to the next level.
In the example packaged
within the DeepStream accelerator,
we're doing defect detection on boxes.
This is a simple showcase of what can be done from
an implementation perspective using this accelerator.
In fact, Joe and team have also created several models,
including people identification,
queue management, hazard zone detection,
and PPE, personal protective
equipment detection that are packaged up and
available through Azure's cognitive services that
work with the Azure DeepStream accelerator.
Our team is
working with Joe's to offer
white glove services for
any customer that requires assistance for onboarding.
Similarly, Fractal has an in Marketplace
offer designed to offer support for this whole lifecycle,
including provisioning and management of
hardware through our Azure Marketplace listing.
With that, I will turn things over to Edwin Webster,
who will walk you through the code base of
the Azure DeepStream accelerator. Edwin.
EDWIN WEBSTER: Starting with
the public GitHub release
of the Azure DeepStream accelerator,
I just want to take some time to focus on
the overall conceptual architecture that we have here.
At the heart of the DeepStream accelerator
is the AI pipeline container.
This allows for connecting to a variety of
different video sources and
processing those using your DeepStream pipelines.
This could be a bring your own model,
this can be something that you would train
using the other NVIDIA tools,
such as the Tau toolkits and so on and so forth
within integrations
with Azure Machine Learning, for example.
And then that's further extended
through suite of additional modules for applying
your own custom business logic for
making sure that the events that we're trying to detect,
that we're taking the appropriate action
based off of that and we can
then communicate the various insights
that are generated from the Edge to the Cloud.
With this, we provided a sample application through
the GitHub example here and this has
a number of different assets that are included here.
Once again, you'll see some of
the graphics that we just saw.
But it also contains
an end-to-end workflow hosted
through this Jupyter Notebook
for taking the data from our pre-labeled dataset
running a trading and custom model using the Tau toolkit,
exporting that in a format that is natively supported
for inclusion with this DeepStream accelerator.
What I've done is I've gone ahead and already
run through this notebook in advance and deployed that.
What you'll see here is that I deployed
this tool in NVIDIA AGX or in DevKit.
If we take one step back here,
you can see we have the modules,
they're all running here.
This is all controlled through one module
called the controller module.
If I click on the controller module,
I could control its activity
in the DeepStream accelerator
itself by modifying the twin properties
that are present on that twin module.
And it will orchestrate all of the changes accordingly.
In this case, I packaged
the model from running
the notebook and uploaded that to a blob storage.
Here I need to specify where my blob storage is located.
The zip file that has the assets with
my model and any configurations that are
required and then this is the path within
my zip file to my primary inferencing engine,
configuration as well as any
secondary or trackers that
we would like to apply as well.
And from there, we applied the module twin and the model
will then be downloaded
and the pipelines will be spun up.
I will also point that within
this configuration is where we can
define things like regions of interests,
as well as the links to the individual video streams
and the sources that would be present there.
Just to take a look at what one of these looks like,
I can pull up the Azure IoT Explorer
and start listening to the telemetry events.
What we'll see is that we start to get the telemetry
and some frames will have
detections for events and others there may not be.
But here you'll see that we
have this concept of halves and
of defect on these inferencing events.
What this is doing is it's
detecting those and then we can then apply
that business logic that was mentioned earlier and
then that gets sent and
routed to the Cloud for further analysis.
What that looks like at the end of the day
is that gets uploaded so both the video
for the event and interests and
the JSON representation of the inferencing events.
We can go ahead and take a look at one of those
and all of the detections that occurred
within a period of time is a JSON representation with
a standardized inferencing schema and then
we bring finally the capability
of visualizing that and
integrating that through a widget.
What the widget looks like is it's with
some very minor configuration
to connect to the blob storage as well as
the Edge deployment itself.
What we're able to do is we're able to define
those regions of interests with
the real viewport from your camera,
as well as playback inferences
from specific periods of time.
If we wanted to select today and then we
can make sure that we have the day selected.
We should get the list of
inferences or events that
were detected throughout the day.
Here, this is just a few minutes ago.
What we can do is we
can play back these inferencing events
detecting the defect boxes versus
the boxes that are passing the analysis here
and you can then take this and
integrate this with other systems that you might have.
You might have custom business logic
for auditing purposes or
downstream inspections or validation as
well as future model training and improvements there.
That's just a glimpse into
the end-to-end capabilities of
this Azure DeepStream accelerator.
We just scratched the surface
on the overall capabilities.
But this should give a great overview of what's
possible with this accelerator
that's available on GitHub.
With that, I will hand it over to Joe for an overview
on the latest with the Microsoft Cognitive Services.
JOE FILCIK: Awesome. Thanks, Edwin.
It's exciting to see how
the Azure DeepStream accelerator
can enable these new scenarios
and how it turns on capabilities from
Azure Cognitive Services for Vision
inside the DeepStream environment.
I'm going to walk through
some quick additional details on how
the Azure Cognitive Services Vision capabilities
work and how they went light up in these scenarios.
Azure Cognitive Services for Vision is part of
this broader Azure AI platform that includes
scenario-based services like
form recognizer and video analyzer,
as well as cognitive services that enable
you with pre-trained models and
the ability to customize for your scenario.
Vision includes a set of
capabilities around OCR to understand text,
face recognition and detection,
image analysis to understand content,
and then spatial analysis to understand video.
All of these are built on top of
a core set of investments
around foundation models, like Project Florence.
Foundation models have been
pre-trained on vast amounts of
data to enable basic understanding
of the world around us.
They come in lots of different flavors,
including Florence for vision,
which can identify almost any object.
Z-code covers language and
translation as it's trained on hundreds
of different languages and GPT,
which can understand and generate
language and code seemingly like magic.
In vision, we've
revamped the Azure Computer Vision service,
accelerating it with a set of
Florence capabilities like captioning here,
that it's able to identify the animal, the vehicle,
and the people in the scene and
enables scenarios like workflows and
notifications and search to be built on
top of the data that it generates.
When we think about integrating this with DeepStream,
we look to hybrid configurations
where we leverage the Edge for everything it can do,
getting the maximum value out of the Edge,
and then utilizing these foundation models
in the Cloud to add
additional insight and actionable intelligence.
Just one example here
shows that when we upload video to the Cloud,
we can enable the user to summarize the video,
getting a text representation of the video,
highlighting key activities that occurred,
like people climbing a ladder,
moving boxes around this warehouse and then we can find
specific frames based on natural language in that video.
This is just one example of how
these foundation models can
enable enhanced capabilities using
video captured from DeepStream devices.
You can try this out yourself at aka.ms/VisualStudio.
Additionally, we'll have a
built session that goes deeper on this
called build apps with state of the art computer vision.
And then there's additional resources covering
the previous topics that we
covered around the Azure DeepStream accelerator,
here you can find the links and the NVIDIA
DeepStream SDK and Jetson Orin device.
With that, I'll transfer back to Carlos,
and he can close this out.
CARLOS GARCIA SIERRE: Thank you,
Joe. Before we finalize,
I want to be sure that you guys can check
the NVIDIA virtual booth after
show where we have
additional demos and information regarding the session.
There is information on how to
become a developer at NVIDIA and join
the developer program and
additional NVIDIA sessions during the show.
Thank you for your attendance and happy DeepStreaming.
DAVID BROWN: Thank everyone. Bye.
CARLOS GARCIA SIERRE: Bye now.