Let me highlight three great uses of DGX.
Transformers have led to dramatic breakthroughs in natural language processing.
Like RNN and LSTM, Transformers are designed to inference on sequential data.
However, Transformers, more than meets the eyes, are not trained sequentially,
but use a mechanism called attention such that Transformers can be trained in parallel.
This breakthrough reduced training time, which more importantly enabled the training of huge models with a correspondingly enormous amount of data.
Unsupervised learning can now achieve excellent results, but the models are huge.
Google's Transformer was 65 million parameters.
OpenAI's GPT-3 is 175 billion parameters.
That's 3,000x larger in just three years.
The applications for GPT-3 are really incredible.
Generate document summaries.
Email phrase completion.
Grade quizzes.
Generate live sports commentary.
GPT-3 can even generate JavaScript and HTML from plain English -
essentially telling an AI to write code based on what you want it to do.
Model sizes are growing exponentially - on a pace of doubling every two-and-a-half months.
We expect to see multi-trillion-parameter models by next year, and 100 trillion+ parameter models by 2023.
As a very loose comparison, the human brain has roughly 125 trillion synapses.
So these transformer models are getting quite large
Training models of this scale is incredible computer science.
Today,
we are announcing NVIDIA Megatron for training Transformers.
Megatron trains giant Transformer models.
It partitions and distributes the model for optimal multi-GPU and multi-node parallelism.
Megatron does fast data loading, micro batching, scheduling and syncing, kernel fusing.
It pushes the limits of every NVIDIA invention -
NCCL, NVLink, InfiniBand, Tensor Cores.
Even with Megatron, a trillion-parameter model will take about 3-4 months to train on Selene.
So, lots of DGX SuperPODs will be needed around the world
to create specialized language-model services, to support international languages,
and to create specialized applications,
all the while repeatedly retraining to avoid model decay.
Inferencing giant Transformer models is also a great computer science challenge.
GPT-3 is so big,
with so many floating-point operations,
that it would take a dual-CPU server over a minute to respond to a single 128-word query.
And GPT-3 is so large that it doesn't fit in GPU memory - so it will have to be distributed.
Multi-GPU, multi-node inference has never been done.
Today,
we're announcing the Megatron Triton Inference Server.
A DGX with Megatron Triton
will respond within a second!
Not a minute -
a second!
And for 16 queries at the same time.
DGX is 1,000x faster and opens up many new use cases,
like call-center support,
where a one-minute response is effectively unusable.
Naver is Korea's No. 1 search engine.
They installed a DGX SuperPOD and are running their AI platform CLOVA to train language models for Korean.
They will use the language model for search, AI assistants, content creation, and summarization.
I expect many leading service providers around the world to do the same - use DGX to develop and operate region-specific
industry-specific language services.
NVIDIA Clara Discovery is our suite of acceleration libraries created for computational drug discovery -
from imaging,
to quantum chemistry,
to gene variant-calling,
to using NLP to understand genetics,
and using AI
to generate new drug compounds.
Today we're announcing four new models available in Clara Discovery.
MegaMolBART
is a model for generating biomolecular compounds,
otherwise known as small molecule drugs.
This method has seen recent success with Insilico Medicine using AI to find a new drug in less than two years.
NVIDIA ATAC-seq,
denoising algorithm for rare and single cell epi-genomics, is helping to understand gene expression of individual cells.
AlphaFold1 is a model that can predict the 3D structure of a protein from the amino acid sequence.
GatorTron is the world's largest clinical language model that can read and understand doctors' notes.
GatorTron was developed at University of Florida, using Megatron, and trained on the DGX SuperPOD gifted to his alma mater
by Chris Malachowsky, who founded NVIDIA with Curtis and me.
Oxford Nanopore is the third-generation genomics sequencing technology
capable of ultra-high throughput in digitizing biology -
viral genomes,
human genomes,
cancer genomes,
plant and animal genomes.
One-fifth of the SARS-CoV-2 virus genomes in the global database were generated on Oxford Nanopore.
Last year, Oxford Nanopore developed a diagnostic test for COVID-19 called LamPORE, which is used by NHS.
Oxford Nanopore is GPU-accelerated throughout.
DNA samples pass through nanopores and the current signal is fed into an AI model, like speech recognition, but trained to recognize genetic code.
This model was developed from NVIDIA Quartznet.
Another model called Medaka reads the code and detects genetic variants.
Both models were trained on DGX SuperPOD.
These new deep learning algorithms achieve 99.9% detection accuracy of single nucleotide variants.
This is the gold standard of human sequencing.
Pharma is a $1.3 trillion industry where a new drug can take 10+ years and fails 90% of the time.
Schrödinger is the leading physics-based and machine learning computational platform
for drug discovery and material science.
They introduced the first virtual screening application 20 years ago.
Their FEP+, Free Energy Perturbation calculation,
achieves lab-level accuracy.
Schrödinger is already a heavy user of NVIDIA GPUs,
recently entering into an agreement to use hundreds of millions of NVIDIA GPU hours
on the Google cloud.
Some customers can’t use the cloud,
so today we are announcing a partnership to accelerate Schrödinger’s drug discovery workflow
with NVIDIA Clara Discovery libraries
and NVIDIA DGX.
The world’s top 20 pharmas use Schrödinger today.
Their researchers are going to see a giant boost in productivity.
Recursion is a biotech company using leading-edge computer science
to decode biology to industrialize drug discovery.
The Recursion operating system is built on NVIDIA DGX SuperPOD
for generating,
analyzing,
and gaining insight
from massive biological and chemical datasets.
They call their SuperPOD
the BioHive-1.
It is the most powerful computer at any pharma today.
Using deep learning on DGX,
Recursion is classifying cell responses after exposure to small molecule drugs.
They are processing 80TB of new images weekly.
It is the world's largest biomedical dataset.
With Biohive-1,
Recursion expects to turn drug discovery
from sequential trial-and-error into a parallel search problem.
Quantum computing is a field of physics that studies the use of natural quantum behavior -
superposition,
entanglement,
and interference -
to build a computer.
The computation is performed using quantum circuits that operate on quantum bits - called qubits.
Qubits can be 0 or 1, like a classical computing bit, but also in superposition - meaning they exist simultaneously in both states.
So, one qubit is in two possible states,
two qubits are in four possible states,
and three qubits are simultaneously in eight possible states -
this superposition is the physics that gives quantum computing its natural parallelism.
The qubits can be entangled
where the behavior of one can affect or control the behavior of others.
Adding and entangling more qubits lets quantum computers calculate exponentially more information.
There is a large community around the world doing research in quantum computers and algorithms.
Well over 50 teams in industry, academia, and national labs are researching the field.
We are working with many of them.
Quantum computing can solve Exponential Order Complexity problems,
like factoring large numbers for cryptography,
simulating atoms and molecules for drug discovery,
finding shortest path optimizations, like the traveling salesman problem,
or searching through exponentially growing data.
The limiter in quantum computing is decoherence,
falling out of quantum states,
caused by the tiniest of background noise -
which can be electromagnetic, thermal, or vibration.
So error correction is essential.
It is estimated that, to solve meaningful problems, several million physical qubits will be required to sufficiently error correct.
The research community is making fast progress,
doubling physical qubits each year,
so likely achieving the milestone by 2035 to 2040 -
well within my career horizon.
In the meantime, our mission is to help the community research the computer of tomorrow with the fastest computer of today.
Today, we're announcing cuQuantum -
an acceleration library designed for simulating quantum circuits,
for both tensor network solvers and state vector solvers.
It is optimized to scale to large GPU memories, multiple GPUs, and multiple DGX nodes.
The speed-up of cuQuantum on DGX is excellent.
Running the cuQuantum benchmark,
state vector simulation takes 10 days on a dual-CPU server
but only 2 hours on a DGX A100.
cuQuantum on DGX can productively simulate tens of qubits.
And Caltech,
using Contengra/Quimb,
simulated the Sycamore quantum circuit
at depth 20 in record time
using cuQuantum on NVIDIA’s Selene supercomputer.
What would have taken years on CPUs can now run in a few days
on cuQuantum and DGX.
We can now reasonably simulate 1000’s of qubits
in Tensor Network Solvers with cuQuantum and DGX.
cuQuantum will accelerate quantum circuit simulators so researchers can design better quantum computers and verify their results,
architect hybrid quantum-classical systems -
and maybe one of these days a quantum computer will be an accelerator to a GPU computer -
and discover more quantum-optimal algorithms like Shor's and Grover's.
They can even accelerate the quantum chemistry codes
that are in use today for drug discovery -
like Quantum ESPRESSO,
VASP,
CP2K,
NWChem,
BerkeleyGW,
TeraChem,
and many more.
cuQuantum on DGX is going to give the quantum community a huge boost.
I'm hoping cuQuantum will do for quantum computing
what cuDNN
did for deep learning.