[MUSIC] Rob is a Chief Platform Architect of
NVIDIA's data center products and we
all extensively collaborate with Rob.
Rob works with the global hyperscale datacenters
in terms of his influence
around AI technology for
future NVIDIA products and roadmaps.
Rob has an extensive background
in processor and system architecture,
multiple patents, and is
an expert in AI and data-centered hardware.
With that, let me pass it on to Rob to take it away.
Thanks very much. I am going to start with a slide,
which you've basically already seen this a few times.
This is a slightly older slide from NVIDIA.
This is our view of how things were
evolving a little over a year ago.
Very similar message, except we tend to use CEO math.
The complexity and cute resources that are being consumed
by meeting edge AI is growing about 10x per year,
so really easy number to remember.
That means it's doubling every few months.
The reason this is happening is
because it works so very well.
The models get bigger, the data gets bigger,
the data sets get bigger.
The consumption is growing in order of magnitude a year.
In response to that,
we do in fact,
have a very rapid hardware evolution.
We do take advantage of silicon advances.
We are constantly working on software optimizations.
But even so, we can offer on the order of
3x year-on-year improvements through those mechanisms.
Which is actually pretty
astonishing when you step back and think about it.
But in order to accommodate 10x year-on-year,
we're seeing, it's very interesting that Broth showed.
My trajectory shows
about three-and-a-half x
year-on-year cluster growth size.
Very similar number.
This is in effect the problem statement of AI,
as much as we're doing,
as compelling and awesome a job as we're doing to
make the AI throughput more efficient and better,
it's still not nearly enough.
I also wanted to set up a few cartoony slides
here and try to help you
see some of the problems as well.
This was prepared for a darker workshop about a year ago.
Because of the position in the ecosystem,
I managed to talk to all of
the hyperscalers and an awful lot
of researchers in the world
and get some anecdotal information about what
the common training size is,
what the clusters look like,
and what they're doing.
I happen to keep track of that over time.
This is anecdotally the way it's looking.
You can see the green bars are
the common GPU count in a single run for AI training.
You can see that going up over time, and in fact,
one of the 100 platform,
which I'll show you later, it has been incredibly
popular for scale cluster training in the Cloud.
One of the Cloud vendors has told me their most common,
their average single training run size is 512 GPUs.
That's not the biggest.
They are certainly getting well over 1,000.
That is their average single-run training size.
We're seeing that progression of
cluster size just compounding really rapidly.
We're in talks with people deploying 2,000,
4,000 GPUs for single training runs.
If you look at that, that's on the order of
megawatts per training run. It's a big problem.
Then I want to
also mention that it's a bit counter-intuitive.
But because the heartbeat of
these neural networks as Broth has talked about,
there are different roof lines that we hit into.
Every different neural network
has a different heartbeat in this,
and you will hit compute roof lines,
memory bandwidth roof lines,
and communication roof lines.
You cycle through those and
the transition between some of those
has some very rapid current changes.
Over time, the transient power
as a present of
the TDP power has been increasing over time.
Now I want to say this is very useful.
It's valuable and incredible amount
of work gets done in this time period.
But if you're talking about a microsecond transient,
that's going to be absorbed by the device
in the peripheral logic, in the board level.
But if you're talking about
milliseconds or tens of milliseconds,
which is really what's going on,
that rolls back into
the infrastructure and the power supplies thorax throws.
If you start thinking about
an average or typical run being 512 GPUs,
you've actually got a row of
compute infrastructure that is running synchronous,
highly aligned and these transients are massive.
Logically and algorithmically, we're doing
things to reduce those transients aggressively,
but there is value to them.
I just want to point that out that in fact,
it is more efficient to let those transient happen.
It's just very hostile to the data center.
Another cartoon Broth touched on this as well,
is that if you take
the current trends and we're not seeing a slowdown.
If you take the current trends,
we're talking about 1,000-megawatt
training around in a few years,
so gigawatts, and that's
just hard to believe that's sustainable.
Nonetheless, that's the direction we're going.
Then, the interesting thing is,
is when I did talk to some researchers, they're no,
no, 40,000 GPUs for a single run,
that will be sufficient.
Even the people who are fairly sanguine think it's
an incredible number of
GPUs that will be involved in a single training run.
Again, this sets the problem statement.
I wanted to take a second mostly just because this is
cool to help you visualize what these look like.
On the top right is an A100 GPU.
It's essentially a full radical,
seven-nanometer FinFET device and
the real device would have
stacks of HBM down either side.
At bottom-right is an HDX board.
This is called the Delta board
and if you buy systems or if you go
to for example Azure or Google Cloud or AWS,
or Oracle Cloud, this is what
you'll be using this board exactly.
There are six of the GPUs with heat sinks,
two are shown there.
8,100 GPUs total, those are 400 watts each,
then the back row with
the heat pipes are in the switches.
Those are incredibly high bandwidth
and interconnect between the GPUs.
It's about two-and-a-half terabytes per
second between the GPUs and then out the back,
the connectors are NVs,
they can exit the board, the same bandwidth.
We have full bisectional bandwidth exiting
the board that can be connected to other systems.
On the bottom left is
the PCI interface to the host nodes.
This is a very heavy board.
It consumes a lot of power,
but it does very efficient compute.
If you look at the left is a diagram of
the canonical reference architecture
for a system built like this.
The board on the right is
just the green boxes labeled
GPU and below it the NV switches.
It's also connected to a PCI switch playing with nicks
and local storage and high-end dual-socket CPU.
What we found ourselves going from
a gaming card company to an HPC cluster company.
This is SELENE.
It's our first-generation reference design SuperPOD.
We've ended up having to do this both
for our own internal research
and development and software development tuning,
but also because our customers are demanding it.
The clusters are becoming so complex that they
want a reference design
that's efficient and high-powered,
that they can rely on,
they can drop software on,
and predict their performance.
This is the number one MLPerf benchmark machine
that's commercially available.
It's the Number 6 on the top 500.
It's the most powerful
privately owned supercomputer in the world.
Depending on the configuration,
it's Number 2 to Number 10 on the Green500.
Half year ago it was Number 1.
In spite of the power this SuperPOD consumes,
it's literally about the most efficient way
you can do complex computing on the planet.
In front elevation,
so you get an idea, this is what we call a scalable unit.
It's 20 nodes that you saw the picture of.
Then we take seven of
the scalable units that include the compute,
the power, networking, and storage.
We combine them and you get 1,120 GPUs
with a fully connected fabric.
Then we can stitch four of those together
to easily get to a 4,000 GPU POD.
Then those 4,000 units can be started to stitch back to
back to get 8,000 or more GPUs in a POD.
This is a boilerplate design today.
For example, Microsoft Azure is a logical clone.
It is an exact copy of the SuperPOD architecture.
With that, I want to give you guys some key takeaways.
I think really the challenge,
the problem statement is that AI is
consuming resources at about 10x per year rate.
It is unbelievable, but we're not seeing a slowdown,
if anything, we're seeing an increase
in that growth rate.
Also as Roth pointed out,
every neural network is different.
The state of the art changes every six months.
Each new neural network architecture
has very different metrics.
Some are compute-heavy,
some are communication-heavy,
some are memory-heavy,
and some are all of those,
but at different points in the training run,
they hit a limit on each of those.
Flexible platforms are needed as
appealing as it is to
optimize hardware around these problems.
I don't think we're there in the evolution of AI yet.
It is changing stills so unbelievably rapidly.
Cluster scales, it's now
really common to have hundreds or thousands of
synchronous GPUs operating at once on a single problem.
That number is likely to grow even as
the capability and power dissipation
of those GPUs continues to grow.
It's optimizing throughput per
watt or throughput per dollar or throughput per system.
It's not just teraflops.
It really is both bandwidth
and latency for networks and memories.
It can be memory capacity
and as well at times it's teraflops.
Again, there's real value that you can
extract from those difficult power transients.
You actually get overall a more efficient throughput,
but it is a real challenge for infrastructure
in power provisioning to accommodate those things.
I think that's it for me.
I don't know if we have time for questions or not.
We do Rob for a question.
Maybe we can go back at the one
that we had earlier for Catherine
and I think you may be able to take
a crack at it from an AI context.
With current state-of-the-art data centers,
how do you look at energy consumed by compute
memory versus rest of the cluster?
How do you see this changing going forward?
Now that is a great question.
I would say traditionally,
it's been a fairly stable ratio
between CPU cores and network bandwidth,
so it's been fairly constant.
What we're finding is, as clusters grow,
it's no big surprise that
the cross-sectional bandwidth that's
required also grows super linearly.
We're very quickly seeing
the fabric be a major contributor
to the overall power consumption.
That is very concerning.
We ourselves are trying to do a few things to
minimize the amount of
energy going into the communication,
both algorithmic ally and physically,
but it is not enough.
There are tremendous opportunities there.
One of the biggest issues is
we cannot compromise latency.
Latency is very critical.
I think as a percentage,
it is growing pretty aggressively.
Roth, there's also a question in the Q&A section.
I can read it if you can't see it right now.
Go ahead, Mark.
The question is around how you talked
about 10x power growth rate
and how is that divided on
the training side versus the inference side.
It's not entirely the same,
but we are definitely seeing
the same trend on inference. We've gotten
used to talking about very small devices for
inference, especially with embedded.
Our mobile phones can do things automotive applications,
but the reality is is some of
these neural networks are so immense
that we have several customers who are
actually using the full DGX system I showed,
that full huge board.
That is a single inference machine,
one inference, one at a time.
Obviously, we're working to try to optimize that,
but I think that's the long-term trend is
that inference is also going to
have a lot of complexity in capability consumption.
[OVERLAPPING] Not to mention
the sheer number of inferences that happened.
It is a brutal workload,
I think on data-centers.
[OVERLAPPING] Is there any water cooling?
I mean getting 4 kw off a board is pretty impressive.
When does water cooling become necessary?
[LAUGHTER] When we can't do air cooling anymore.
I think conceptually everybody loves liquid cooling,
immersion cooling is really awesome.
They're just really awkward
to deploy in the data-center,
and as a result,
everybody is preferring to avoid it as long as possible.
But I think we're right on the cusp.
Honestly, we were expecting
our next-generation to require liquid cooling.
I think we've dodged that bullet for the moment,
but there are a lot of data-centers that simply can't
accommodate this kind of cooling and
they will migrate to liquid or immersion.
We're getting some fantastic experience
with immersion as well.
It's a great way to cool.
It's incredibly awkward to build a data-center for it.
Thanks.
Dodging the bullet sounds like the right word.
[LAUGHTER]
Yes.
That certainly sounds like the right word.
Thank you. Rob. [MUSIC]