Inference is the last stage of the machine learning pipeline. This is where you take
the model that you trained and deploy it into services to make predictions. What comes out
of the machine learning pipeline and frameworks are computational graphs that are incredibly
complex. These are gigantic computational graphs and there are so many different types
of neural network architectures that the computer science of compiling these computational graphs
into a target machine to run as fast as possible with all the different types of numeric precisions,
incredibly complex problem. We created an optimizing compiler TensorRT.
We're now in our seventh generation. This generation can handle CNNs and transformers
and now we can handle RNNs as well. TensorRT 7 includes over a thousand optimized kernels
and in fact our developers could do the same for themselves and create all kinds of custom
kernels and custom neural networks. We support precisions from FP32, FP16 all
the way to 8-bit and 4-bit integer. TensorRT has been a huge success. The number of developers
that use TensorRT increased 10X, the world's top 300 internet services now have NVIDIA
GPU in their data centers to do inference. It just kind of shows how many industries
deep learning and AI is going to impact. One of the most important applications that we
can now enable is conversational AI. It is one of the most challenging inference tasks,
because conversational AI requires interactive performance.
The elements that make up a conversational AI pipeline has achieved tremendous breakthroughs
recently starting from ASR, automatic speech recognition to natural language understanding
to text to speech, speech synthesis. And now it's possible for us to imagine for the very
first time having an interactive, low-latency conversational AI pipeline. However, the application
is very complex still. The models are state of the art.
Training those models take a tremendous amount of computation and the ability to put together
all of the models end-to-end on top of a computing platform fully accelerated, is something the
world's never done before. Until now. Ladies and gentlemen, we're announcing today a new
application framework we call Jarvis. And because it has the ability to connect into
Omniverse we have the ability to create an interactive 3D chatbot.
One of the AI models that makes this entire experience possible is what we call audio-to-face.
It takes an audio input and has learned how to animate a geometric mesh, to animate facial
expressions while talking. That then drives our computer graphics system in Omniverse
and creates an interactive chatbot. Let me now show you our AI model audio-to-face, The
speech here is coming from a rapper, One of our employees, John Della Bonna, and
it is really fantastic. Take a look.
Living for the now Long as time allows
I'ma keep on switching different styles keep creative on a cloud
Sweat is on my brow Cause I'm running on these tracks
Just to keep them running back You the know drill the court and back
And I been practicing my craft Dedicate this play to Kobe
What could be a bigger legacy Than making it his-tory
Every day gets better No doubt
So we gotta hold out I got a dream
To pack the show that's sold out It's all about the progression
The daily grind is a test and Someday it's less of a grind and more like
A day with a blessing If you having real problems I feel bad for
you son there's like 99 problems many share the same one
Yeah so keep your head up
and get up I bet tomorrow is better
Striving to be B who I C Follow that to the letter
Once we have audio-to-face, then we can connect it to our conversational AI model where you
can ask it a question, it would understand the question, answer the question and then
synthesize speech from that which goes into our audio-to-face model, which animates and
drives a 3D model. We've created a demonstration we call Misty, our first interactive 3D chatbot.
Hello. Hi, I'm Misty.
How are you? I'm doing well. Thank you for asking.
What are you here for? I'm here to give you the latest weather information
around the world. Tell you interesting facts about meteorology and keep you company.
Should I bring an umbrella tomorrow? It is not expected to rain in Santa Clara,
California tomorrow afternoon. What is the hottest city in the world?
The hottest city in the world is Mecca in Saudi Arabia where temperatures can reach
122 degrees Fahrenheit. What is the coldest city?
The coldest city in the world is Yakutsk, Russia with temperatures around minus 40 degrees
Fahrenheit in the winter months.
Tell me something about lightening.
We see lightening first because light travels faster than sound.
How come you know so much about weather? I know the weather information from the Internet.
I have lived for a long time and have been talking about the weather for as long as I know
Where are you from, Misty?
I'm from a place called Omniverse. Who made you?
I am created by the wonderful scientists of NVIDIA.
What powers you? I'm powered by NVIDIA GPU and the CUDA neurons
inside of them. Thank you Misty.
Thank you. It was a pleasure talking to you.
As you could see, the ability to have an interactive
conversation requires the conversational AI to process the speech, the natural language
understanding, synthesize and also render the graphics as fast as possible and you have
to process the entire pipeline end to end from speech recognition, language, language
understanding, text-to-speech as well as driving and generating the computer graphics in just
a few hundred milliseconds in order to feel like you're having an interactive conversation.
And that's what NVIDIA Jarvis is about. A multimodal conversational AI service framework
that simplifies the creation and the development of conversational AI services.
It includes state-of-the-art models that has been pre pipelined into these Helms charts
optimized to run on NVIDIA's Triton Inference Server, which runs on top of our GPUs and
the performance is interactive and the entire pipeline end-to-end is only a few hundred
milliseconds, but there's more. Jarvis comes with pre-trained state-of-the-art
models. These state-of-the-art models have been trained with a great deal of data and
a lot of computation. In fact, what's available in NGC, the NVIDIA GPU Cloud represents several
hundred thousand DGX training hours. If you had one DGX, it would take you something like
10 to 20 years to be able to process all of the data necessary to train these models.
We've done that for you. And then it comes with a new tool called NeMo, which takes the
pre-trained model and augments it with your data.
Your data could, because of a particular domain, it could be in healthcare or insurance or
financial services, or maybe it's a call center for a particular business and there's special
language or special vocabulary that Jarvis needs to learn. And so you would take that
data and you would train, retrain the Jarvis models with this tool we call NeMo. This end-to-end
pipeline from pre-trained models, retraining with NeMo, and then the application of all
of these state-of-the-art models all put together in a Helms chart running on a service on top
of Triton this entire end-to-end system we call Jarvis.
Conversational AI is going to automate conversations and it's going to make it possible for us
to deploy automated services in all kinds of new applications, whether it's video conferencing,
call centers, smart speakers. One of my favorite applications is video conferencing. When you
have a group conference one person can talk at a time, it would be nice sometimes if we
can kind of hear multiple people talking. So wouldn't it be amazing if whenever anybody
talks it's picked up and close captioned or translated in real time or the end of a conference
a simple summary and transcription is done. Video conferencing with a conversational AI
agent is going to be really transformed. We know that the number of people who are calling
into call centers these days is growing and many of the call centers have people wait
half an hour to much longer. Jarvis conversational AI is going to be one of the most important
real-time inference applications.
That's NVIDIA AI. It is now end-to-end, fully accelerated. From data processing with the
amount of data growing exponentially, Spark is more important than ever and now it's accelerated
by NVIDIA. Second, the deep learning and the model training pipeline. We have accelerated
for a long time. One of the particular applications recommender systems, is so complicated. We
decided to create an application framework just for it so that we can democratize recommender
systems. We call it NVIDIA Merlin. And then lastly Inference, TensorRT 7.0 is here. The
number of people who are downloading it and using it is really growing and we're super
excited about that. We decided to create an application framework for every company to
be able to create state-of-the-art, conversational AI models we call that NVIDIA Jarvis, this
end-to-end acceleration platform, and some of the reference applications
is what we call NVIDIA AI.