- It's pretty clear
where NVIDIA's priorities lie these days.
We're here at the Computex booth
of one of their partners, Gigabyte,
and this is the entire gaming showcase.
That's because they, like the rest of the industry,
understand that the future of computing
lies in the data center.
That is where the Grace Superchip comes in.
Under each of these gigantic heat spreaders
are 72 of NVIDIA's Grace CPU cores,
connected together using what NVIDIA calls
the NVLink chip-to-chip interconnect
for a total of 144 cores,
except that's just one of the nodes.
This server from Gigabyte accepts not one, not two,
but four of these modules in its four separate nodes.
That is an absolutely mind bending
576 cores in a 2U server rack.
But these are not the types of CPUs
that you have in your gaming PC at home.
Those processors from the likes of AMD and Intel
are based on the x86 architecture.
So, similar to what Apple did with their M series:
M1 and M2 processors,
NVIDIA is making use of a different
processor architecture called ARM.
And we actually did get permission to do this,
you're gonna be taking a closer look here.
Ohh.
It doesn't look much like it,
but this is the same style of processor
that you might find in your phone.
ARM Processors have a lot of advantages,
first and foremost being that they're typically
more power-efficient thanks to their relatively
lightweight instructure set.
So much so that NVIDIA claims
these Grace CPUs have twice the performance per watt
of the latest x86 chips.
But the disadvantage is they also require software,
like your operating system
and all the programs you need to run,
to be coded and compiled specifically for ARM.
Now for the PC market,
because 86 has been the standard for so long,
it's difficult to justify switching over to ARM.
It would cost you so much
in terms of backwards compatibility.
But in the data center, the types of customers
who are gonna buy a processor like this
are usually developing their own software anyway.
Like, let's say Google, to run the algorithms
that power Google search, or YouTube recommendations.
For them, switching over to ARM isn't as big a deal,
and in fact, companies like Amazon,
who are developing their own ARM-based CPUs,
are already doing it, and very effectively.
I mean, hey, if my next gaming CPU
could be half the power draw
and the same performance of my current one, I'd be stoked,
but this is even better.
Imagine if instead of one computer,
you're talking thousands or tens of thousands.
The savings start to become so large
that it's less a question of can we afford this migration,
and more a question of can we afford not to make it?
Now, I didn't ask permission for this part
but nobody seems to be stopping me
or even really paying attention to me.
So, let's take apart Grace Superchip.
On each Grace Superchip is up to 480 gigabytes
of LPDDR5X ECC memory per CPU.
And what's really cool is that that can actually be accessed
by either CPU over the NVLink interconnect.
That's how fast this new NVLink is.
The only downside to this approach,
since we're making comparisons to Apple,
is that just like with your M2 MacBook,
you better decide how much memory you want in your server
right at the time you buy it,
unless you wanna replace the entire compute engine
while you perform a memory upgrade.
Given that the rumored price
of their H100 GPUs is $100,000,
I don't even want to know what this thing costs.
But hopefully you get a bit of a discount
when you buy it together with the Grace Superchip CPU.
Let me show you this.
Can't believe they're letting me
take this off the wall. (chuckles)
Okay.
Success.
We have dropped nothing important so far today.
This is Grace Hopper.
On the one side we've got the same 72-core
Grace ARM CPU that we just saw,
but on the other side,
the ooh, shiny, latest NVIDIA H100 Hopper GPU.
You can probably see where this is going.
Just like with the dual-CPU Grace module,
these two are also NVLink chip-to-chip interconnected,
meaning that the CPU and GPU
have a whopping 900 gigabytes per second
of theoretical bandwidth to talk to each other.
So first in perspective, a GPU,
using a full 16-lane Gen5 PCIe slot,
would only have about 64 gigabytes a second
of peak throughput.
That is 1/14 as much as this,
and that's far from the only mind-bending number
that this thing is capable of.
While the CPU side uses the same,
up to 480 gigabytes of LPDDR5X,
for the GPU side they need much faster HBM3 memory
that runs at a whopping four terabytes per second.
That's about four times faster.
That's why the memory needs to be right on the package
right next to the GPU.
Now, all of that is great and cool and all,
but HBM is very expensive,
and as you can see, there's only so much space here.
So the H100 only gets 96 gigabytes of memory.
Okay, yeah, for gaming, that certainly sounds like a lot
but AI data sets can involve terabytes of data,
so it can get used up very quickly.
That's where the interconnect comes in.
It allows the GPU to access the CPU's memory
in a very direct and transparent way,
giving the H100 Hopper GPU a functional memory capacity
of nearly 600 gigabytes.
In practical terms, according to NVIDIA,
that puts Grace Hopper
anywhere from about two and a half times
to nearly four times as fast
as an x86 CPU paired with their last generation A100 GPU.
And where things get really wild is in the data center
with an NVLink switch system,
you could connect up to 256 GPUs together,
giving them access to up to 150 terabytes
of high bandwidth memory.
I mean, you guys remember that crazy Mars Lander demo
that we showed off on the "Petabyte of Flash" array?
You could load that entire one billion point data set
into memory in that configuration,
and still have 50 terabytes to spare.
Now, this module, little bit more power hungry
than the dual CPU version.
A thousand versus 500 watts per module.
But I mean, that's for CPU, GPU, and RAM for both of them.
And with this kind of performance
of course not everybody wants to move
to an ARM hybrid CPU, GPU architecture.
So NVIDIA is still going to be supporting their
old fashioned configurations, be they H100 GPUs
and a PCIe form factor, or their HGX H100
with up to eight SMX5 GPUs.
Each of these draws a massive 700 watts,
making an RTX 40 90 look like a child's play thing,
and supports NVLink between these GPUs,
and NVSwitch to additional servers.
This is the G593-SD0,
and Gigabyte was very proud of the fact
that they are the first NVIDIA certified HGX H100
8-GPU server in a 5U chassis.
Man, that is a lot of compute in a tiny space.
Jake's in my ear here telling me
I should pull one of the power supplies,
but if you've noticed it getting darker,
it's because they're actually shutting down the pre-show
and they're trying to get us outta here.
But there is one more thing that we wanted to talk about.
Where'd it go?
Dang it, Jake.
No. Oh my God!
Okay, well this is, no, wait.
This isn't the one I wanted.
Okay. It's a ConnectX-7,
this is an even faster network card.
So this is probably the first NVIDIA developed
Mellanox network card,
given that the acquisition was what, about two years ago?
- [Jake] Yeah, ConnectX-6 was already out.
- Yeah, but NVIDIA didn't buy Mellanox
just to make faster ConnectX cards.
(chuckles) No, it was to make these.
This is a BlueField-3, so it has networking on it.
This is a 100 gigabit one, but it's available
at speeds up to 400 gigabit.
But what's really special about it
is that it has up to 16 processing cores on it.
Why, you might ask?
Well, just like in the old days when we started offloading
TCPIP processing to our network cards
rather than having our CPU handle them,
this is gonna offload all kinds of interesting things
like encryption of your network traffic,
or say for example, handling, managing your file system.
Because when you're someone like an AWS
and you wanna squeeze as much revenue as possible
out of every CPU in your data center,
you don't want it handling stupid BS
that you could just offload to your network card.
So the idea here is to free up CPU resources
that can be leased to customers
by putting them onto the network card itself.
And this is especially true for software
where the developer sells you a license per core.
That's why even though these are gonna be wildly expensive,
a lot more than the 4060 Ti,
NVIDIA is going to sell shed loads of them.
Just like I sold this segue to our sponsor.
Pulseway.
Are you sick of feeling like a prisoner
chained to a desk, managing IT systems?
Unleash your inner IT hero
with Pulseway's remote monitoring and management software.
Pulseway's platform gives you the power
to manage your IT infrastructure from anywhere,
even from the comfort of your own couch.
And with real time alerts and notifications
you can be the first to know
about potential issues before anyone else on your team.
It's accessible through whatever device is close to you
thanks to their convenient apps,
allowing you to control your IT systems like a boss,
even if you're lounging in your PJs.
So say goodbye to the boring routine
of IT management and hello to the fun
of being an IT hero with Pulseway's advanced technology.
Don't wait.
This is your chance to become a legend in the IT world.
Just try Pulseway for free today, and experience the power
of simplified IT infrastructure management.
Click the link below to get started.
(upbeat music) If you guys enjoyed this video
why don't you check out, oh, "The Petabyte of Flash."
- That was a good one.
Well, at the Gigabyte booth, come on.
The G RAID one.
- Yeah, G RAID. - Actually, no, new Whonnock.
New, new, new, Whonnock 3.
Whonnock 4, I mean.
Damn it.