AI is writing software that achieves results no human-written software can.
The scalability of this approach has started a race to create ever larger, more complex and more capable AI.
The chart shows the number of days to train a model on a 1 PetaFLOPS computer.
Computation required to train state-of-the-art models has increased 30,000 times in less than 5 years
That's doubling every couple of months!
The capabilities of AI are incredible:
NVIDIA's Natural Language Understanding AI took the RACE reading comprehension test,
consisting of 30,000 passages and 100,000 multiple choice questions.
The average human scored 73%.
Expert humans score 95%.
NVIDIA's Megatron-BERT scored 91%.
Facebook AI Research developed a Transformer-based chatbot
with knowledge, personality, and empathy
that half of the users tested actually preferred.
Researchers at Caltech developed a reinforcement learning-based drone flight-control-system
that flies smoothly through turbulence
and changes in terrain.
Other researchers at Caltech,
used deep learning in quantum chemistry
to predict molecular-protein reaction energy.
Sped up simulation by 1000 times!
TUM - Technical University of Munich, Google, NVIDIA, and Oak Ridge National Lab trained language models
that predicted the 3D structure of a protein just by reading its amino acid sequence.
The 3D shape is used to predict whether a chemical compound will bond to the protein.
Researchers at NVIDIA used deep reinforcement learning to train the control system
of a 4-legged robot to adapt its gait across multiple terrains, even if the surface is in motion.
The progress of AI is incredible and like magic.
The magic of AI also comes with its incredible computing challenges -
enormous models and training datasets that push the limits of every aspect of the computer
- compute, storage, networking.
NVIDIA is democratizing AI.
There are three pillars to NVIDIA AI:
First, data processing, feature engineering, and training, at any scale.
One GPU, multi-GPU, to multi-node. Any framework, any model, in any cloud.
Second, Inference is a great computing challenge.
NVIDIA TensorRT is a graph-optimizing compiler that targets an NVIDIA inference computer
while optimizing for a bunch of competing constraints accuracy, throughput, response time, memory size.
We are releasing TensorRT 7.2 over 2000 kernel- and layer-type optimizations.
TensorRT has been downloaded 1.3M times,
used by 16,000 companies,
operating over 300 services around the world today.
NVIDIA Triton Inference Server is used by data centers to deploy AI models.
NVIDIA Transfer Learning Tool lets customers take pre-trained models and refine them for their own datasets
to optimize for their usage domain
like refining a general language model for healthcare or engineering speak.
Third, AI application frameworks, pre-trained models, transfer learning, runtime engines
for some of the most challenging AI applications
like self-driving cars, robotics, drug discovery, conversational AI and recommender systems.
Thousands of NVIDIA engineers are working on NVIDIA AI today.
And thousands of companies around the world run on NVIDIA AI.
The software stack for AI in accelerated computing is complex.
The stacks for each domain are different
Rendering is different than genomics,
different than data analytics,
and different than deep learning.
The stacks for each computing environment virtualized or scale-out or microservices are also different.
NGC is our cloud for containerized stacks always up to date, always optimized.
NGC runs on all NVIDIA-certified OEM systems and clouds.
NGC has been a huge success -
1 million downloads in just a couple of years,
250,000 users,
growing 4x year over year.
Cloud-Service-Providers have asked us to integrate NGC into their clouds.
So today we are announcing that NGC will be also be available in Azure, AWS, and GCP marketplaces.
NGC will be easily discoverable in CSP clouds and always up to date with the master registry.
With this, you can be assured to get the best performance and the lowest cloud compute cost.
And your cloud and on-prem stacks will be consistent.
Engineering is required so we'll get there as fast as we can.
In the meantime, the optimized stacks are on NGC.
Billions of people are interacting with cloud services hundreds to thousands of times each day.
Each interaction launches a query, a recommendation, an ad, a piece of news, and often all the above.
Trillions of AI services are processed each day.
Processing the AI on NVIDIA is the most performant and cost effective.
This year is the 10th anniversary of the first NVIDIA GPU in the AWS cloud.
Since AWS, every cloud now offers NVIDIA GPU,
and the aggregate compute throughput has increased 10x every 2 years.
This year, we will ship over 166 exaops of inference compute to CSPs more than 6x that we shipped last year.
We estimate that the aggregate NVIDIA GPU-inference-compute in the cloud now exceeds that of all cloud CPUs.
With this trend, in a couple of two-three years, NVIDIA GPUs will represent 90% of the total cloud inference compute.
Any AI application and service can now rely on NVIDIA Inference.
We are past the tipping point.
NVIDIA AI Inference is operating services for companies in a broad range of industries from cloud, financial, and consumer services,
autonomous vehicles, robotics, healthcare, agriculture, and retail.
Take for example, Square, they are using NVIDIA to process conversational AI for businesses to schedule appointments with customers.
Twitter is using NVIDIA AI to understand the torrent of live videos.
Ebay does millions of item image-searches a day with NVIDIA AI.
GE Healthcare uses NVIDIA AI to infer measurements on cardiovascular ultrasound.
Zoox, the robotaxi group of Amazon, uses NVIDIA AI to do perception for cameras, lidars, and radars.
Today we are announcing that Microsoft is adopting NVIDIA AI on Azure
to power Smart Experiences in Microsoft Office.
The world's most popular productivity application, used by hundreds of millions, will now be AI-assisted.
The first features will include smart grammar correction, Q&A, text prediction.
Because of the volume of users and the instant response needed for a good experience,
Office will be connected to NVIDIA GPUs in Azure.
With NVIDIA GPUs, responses take less than 200ms
and our throughput lets Microsoft scale to millions of simultaneous users.
The inference pipeline consists of NVIDIA V100 in Azure, running the ONNX runtime, on NVIDIA Triton Inference Server.
Cybercrime costs the global economy nearly a trillion dollars a year about 1% of the $140 trillion global economy.
American Express does 8 billion transactions a year,
totaling $1 trillion dollars for their 115 million cardholders
Huge numbers!
Using NVIDIA AI, American Express processes tens of millions of transactions daily, each within 2 milliseconds to detect fraud instantly.
AI fraud detection is going to save the financial industry hundreds of billions each year.
NVIDIA AI makes this possible.
Modern cloud services are composed of microservice containers and orchestrated by Kubernetes.
Microservices like Automatic Speech Recognition, Natural Language Understanding,
text to speech, recommenders, database queries
are reused by many applications all at the same time.
These microservices are running on servers throughout the data center and scaled up-and-down to meet dynamic workloads
while delivering good response time.
Modern data centers are scalable, composable, and accelerated
with servers of different types like CPU, GPU, and storage each optimal
for certain types of microservices.
Let me show you one of the most advanced AI applications running end-to-end.
All of the technology components are state-of-the-art and ready to for you to customize and deploy in the cloud or on-prem.
This is SpeechSquad, a benchmark we developed to measure the response and throughput of conversational AI.
SpeechSquad asks a question,
understands the question, searches for the correct answer, and generates a natural human voice response.
It consists of speech recognition, Q&A using BERT, and text to speech.
SpeechSquad is open-sourced on Github.
First, let's run SpeechSquad on a CPU
When did the Germanic tribes claim territory in north and west Europe?
In the middle period of classical antiquity
Each of the ASR, BERT, and TTS services are assigned to its own CPU.
The response time is over 600ms. CSPs want response times in about 300ms.
For a CPU, that's not enough time to use a neural network model
so we are using Kaldi for speech recognition and MaryTTS for speaking.
That's why CPU sounds mechanical
Now let's run SpeechSquad on Jarvis on GPU.
What is the highest court in European Union law?
The European Court of Justice
Again, we assign each of the ASR, BERT, and TTS microservices to GPUs.
The latency is now 300ms, twice as fast as a CPU.
And GPU can deliver a much more human-quality speech.
We're using NVIDIA TacoTron speech synthesis and NVIDIA Waveglow voice encoding
100x more computationally intensive than Kaldi and MaryTTS.
However, it sounds a lot more natural.
Let's scale this service to support 1,000 streams.
NVIDIA Triton scales up 15 A100s.
Notice we are getting super linear scaling.
This is because the Triton inference server is doing load
balancing of the microservices.
It takes over 100 CPU nodes to support 1,000 streams.
Scaling to 112 A100s deliver nearly 7,200 streams.
This would require 1000 CPUs.
So, with A100, conversational AI is half the latency, a third the cost, and much more natural sounding.
What did I tell you? The more you buy, the more you save.
I want to thank GCP for letting us run the experiment on their cloud A100s and CPUs.
GCP is a fabulous cloud infrastructure.
NVIDIA Jarvis, our state-of-the-art conversational AI application is now in open beta.
Jarvis is an AI with conversational skills and trainable only with large AI infrastructures like DGX SuperPOD.
Its neural text to speech is human-like.
Its natural language understanding is our reading comprehension record setting Megatron-BERT
And the application is fully customizable.
Live video calls are one of the highest traffic on the internet today.
For work, social, school, virtual events, doctor visits video conferencing is now the most important application for many people.
Today we are announcing NVIDIA Maxine. a cloud-native streaming video AI platform for applications like video calls.
AI can do magic for video calls.
Using AI, we can perceive the important features of a face, send only the changes of the features, and re-animate your face at the receiver
This reduces bandwidth by a factor of 10.
AI can reorient your face so you're making eye contact with each person on the call, individually.
Your face is regenerated to make eye contact with each person.
AI can realistically animate an avatar based on just the words you are speaking.
Yes, there was recently a sign of extraterrestrial life on Venus. What else do you want to know?
AI can remove background noise, super res, see better in low light, replace your background, and even re-light your face.
And with Jarvis conversational AI, we can do real-time language translation and closed captioning
one person is heard but everyone can
simultaneously talk what they say is closed captioned or texted.
With Jarvis and Maxine, we have an opportunity to revolutionize video conferencing of today and invent the virtual presence of tomorrow.
NVIDIA AI inference applications are coming from every industry.
Natural language understanding
Microsoft Office
Instantaneous fraud detection
Conversational AI
AI video calls
Robotics
NVIDIA AI Inference is in every cloud, growing at 10X every couple of years
and have even eclipsed the total inference compute of CPUs in the cloud.
We've reached the tipping point of accelerated AI inference every AI service and application can now be GPU accelerated.