- I bet most of you forgot about the $1,000,000 PC here,
and honestly, I don't blame you.
It's kind of been a while since we touched the thing,
but I swear it's not our fault.
Right before we were gonna do the big demo, it broke.
Then it broke again and again.
It's just been the worst kind of problem to troubleshoot.
Sometimes our full petabyte of NVMe SSDs gets detected,
and other times, kapoof, some of them don't.
But, I wouldn't make a video just to tell you
it's still broken, would I?
The solution ended up being shockingly simple,
and something that I bet you've actually seen before.
But that's in the past.
This is the now, and it is finally time.
When this video is over,
we are going to have the largest,
and fastest storage server on YouTube.
At least until we have to find all the boxes,
pack it up and send it back to them.
- Did I tell you they threw away the boxes?
- Sorry, what? Oh my god.
- They're not behind me.
- How are we gonna deal with that?
How are we gonna segue to our sponsor?
G.Skill! G.Skill's Trident Z5 Neo DDR5 memory
is built for AMD Ryzen 7000 series processors,
and one click memory over clocking.
Learn more at the link down below.
(upbeat music)
- The culprit was this guy.
- But, not this guy entirely.
Four of the drives, specifically out of the 12,
had this erratic behavior,
where we would fire up the whole system
and they wouldn't be there.
Then we'd re-plug the cables, and they'd come back,
and we could benchmark it, and test it and validate it,
and then they would be gone.
We tried everything,
from moving those drives to different bays, to...
- Different servers even.
- Unplugging and re-plugging the cables,
and it just kept freaking happening.
And, in the end, it was one of the most basic
troubleshooting steps in the book.
Have you tried unplugging it and plugging it back in?
But not the drive, not the back plane.
- Or the cables.
- The CPU. 'Cause what it's hard to wrap your brain around,
is that these are NVMe, or PCI Express drives,
and through the connector on the back,
through the back plane, through the motherboard,
to the CPU socket, they actually connect directly
all the way back to your processor.
Which means, that a loosely seated CPU could actually
cause a broken link in that chain and...
- Here's the thing, after the servers had been running
for a few minutes, that's just enough thermal expansion
for that CPU pin that's just not quite touching.
Bam. Now it's touching.
- Boom. Intermittent problem.
- Yeah. Of course, with the system already on,
it's not gonna pick those drives up again,
so when we were rebooting it, after reseating the cables,
they'd show up again.
It makes perfect sense now, but oh my God.
Trying to figure this out.
You gotta think outside of the box and also-
- But also,
- in the box - deep inside the box.
(both laughing)
Okay. You got it?
- Yep. - You got it.
Everybody's got it. Oh, this is not screwed on, Jake.
- [Jake] What is not?
- This top cover.
- [Jake] Oh, whatever.
- Where are we working on this?
- In there.
- Oh, sure.
- You have a screwdriver, right?
- I have a highly erect server, that's for sure.
Yes, I have my high quality,
ratcheting magnetic screwdriver. lttstore.com.
- Now, if we were smart folks,
we might have looked at what drives they were,
and traced it back to what CPU it was.
- Yeah that's not a CPU, but you know.
- What I was trying to say, is we're just gonna reseat
both the CPU's, dammit!
- Sure. Oh-
- I don't have a screwdriver yet, so-
- I went and I put a Phillips bit in here.
- So you're just gonna have to do this all yourself.
- That's fine. Crap. I don't have a torx set.
You know what? Sometimes hex works.
- [Jake] Oh my god, don't do that.
- [Linus] What?
- Don't do that, that's awful.
You're gonna strip this and then we're-
- I'm not gonna strip it.
- You're gonna make this a very long video.
- I'm not gonna strip it.
- You could go grab another one.
You could make yourself useful.
- I could.
- [Jake] The funny thing is-
- [Linus] Oh God! - Oh my God.
- I'm kidding.
- Oh my God.
- Oh my God, he's freaking out.
- Why did you even take it out? Just-
- I'm just-
- You- Don't, no, don't clean the thermal paste.
We'll just stick it...
- Oh my God. Really?
- Yeah. Re seat it, baby.
- No, we should re goop it.
- No.
- All right.
- Thermal paste is expensive. In this economy?
- [Linus] Right. I mean left, correct?
- [Jake] Left?
- [Linus] Correct, yes. No. Uh, no.
- I was never much good at math.
Why are these so finicky, dammit?
- [Linus] Ta-da! - [Jake] Beautiful!
- [Linus] Beautiful.
Watch the server not boot now.
- I know, right?
That is a very real possibility with AMD's thread ripper
and epic CPUs, because they're so large
it is pretty easy to accidentally install them
a little bit cockeyed which can cause these sorts of issues.
Whether it's PCI express devices
having intermittent problems or RAM.
- This isn't even on yet. This is like the boot sequence.
This is the the IPMI.
- For those of you who haven't seen the previous parts
of these series or who, understandably, forget them
at this point, what you're looking at here
is a one petabyte of flash storage server
which is a non-trivial thing to build because,
aside from just having enough bays to put that many SSDs in,
if you wanna get anywhere near the full performance
of these Kioxia drives, you need a ton of compute.
So inside each of these 1U servers here
is 12 15-terabyte drives.
So that's for a total of 72 drives.
Each of these CD6-R enterprise drives is capable
of a whopping 5.5 gigabytes a second reads,
four gigabytes a second writes.
All of these work together using a file system
called WekaFS, that's designed specifically
for NVMe drives to achieve unbelievable performance.
Of course, in order to measure this performance
you actually have to put some kind of load on the system.
That's where this guy comes in.
It has 2 64 core epic CPUs, eight of NVIDIA's A100 GPUs.
Those are critical to generate the load
and it has a whopping eight 200 gigabit per second
network connections.
And, of course, those run through this 32-port 200 gigabit
network switch to each of our servers down below.
That's a lot of high speed connectivity
but with great performance comes great power consumption.
And to run this thing full bore,
we needed to plug extension cords into five separate
15 amp 120 volt breakers and a separate 208 volt,
30 amp breaker.
- And did I mention they're over clocked?
- I mean, I feel like I could tell
from all the heat coming off of it.
It is flippin' warm back here.
- It's very uncomfortable to stand here.
Another thing we ran into last time
just to get the array started, these servers were
so out of sync, just from shipping.
- Like, the time.
- The time on the servers was so out of sync.
The real time clock, you know, it could be five, ten
seconds off, whatever, that the array would not start.
And the easiest way to get them to sync up again
is NTP network time protocol.
And of course, without networking
- This wasn't set up to actually connect to the internet.
- All of these have static IPs that are assigned.
So, in our router, I had to create a VLAN that has the same
subnet information and route it from the server room
all the way over here to here and plug it into the switch.
Had to go in the switch and tell it. "Oh, 10 gig is fine.
It doesn't have to be 200 gig."
And then they turned on no problem.
But I want to access the IPMI
so we can see the power consumption.
- Sure.
- So we've gotta plug this bad boy into all the IPMI ports
really quick.
- Oh, okay. So we need some patch cables.
- [Jake] The patch cables are right in the front.
- All right, let's do it. Oh, that's wild. Yeah.
The management port for this one is actually on the front.
- Oh, man. It's so much nicer in here.
- [Linus] I wanna see all 72 drives is what I wanna see.
- You might be asking for a lot.
- No, no. Bare minimum.
- When those four drives are missing,
the array just rebuilds. Oh, god.
Like, just give it a second. Oh! Sick, it's working.
Okay. Woo! The cool thing though, when those four drives
are missing, the array, like, rebuilt itself,
like it was nothing. Takes like five minutes 'cause they're
so fast. - So fast.
Yeah, but we want the capacity.
- We want the performance.
- That's the whole point. A petabyte of flash.
- Okay. I hate to burst your bubble.
For some reason, they only configured it with
a 500 terabyte, like, drive.
- Only 500 terabytes?
- I think we can expand it later, but, for the purposes of
the first demo, I think let's just leave it.
I don't want to break it. I'm SSHing into the node.
Oh, look at that. How many is that?
1, 2, 3, 4, 5, 6, 7, 8.
- [Linus] Good. That's the correct number.
- That's a lot of GPUs.
- That's a lot of of freaking GPUs.
- Is there NVIDIA SMI on here?
- Each of these is, like...
- Oh, look at that! 400 hundred watts.
- 3,200 watts of just a GPU.
- We're not connecting to these things over anything
like SMB 'cause that would be way too slow.
We're, like, directly connecting with the Weka interface.
- Yeah.
- And we're gonna be using GPU direct storage
which is super cool.
We'll talk a little bit more about that later.
- [Linus] Yeah, this visualization is super cool.
The green things I'm looking at, I guess are drives.
- [Jake] Those are the drives.
- [Linus] And then the gray things are cores. Or, no,
the gray things are drives, the green things are cores.
That makes way more sense actually.
- [Jake] Yeah, yeah, yeah.
- And then blue is our network interfaces.
That is a pretty neat way
to visualize it. - It's so cute.
Oh yeah, you see this? Look, it's, they're pinning cores
to specific drive slots it looks like.
And there's some special sauce going on here I think.
- There is some manual ass shiz that went on
in the configuration of this system.
Seriously though, a lot of tuning because,
like I said before, making a one petabyte flash server?
Easy. Making one that performs near even actual...
- We're still not gonna be hitting the peak.
You'll remember when we did the honey badger server
we got like a hundred gigs a second.
- [Linus] Yeah.
- That was directly writing to individual drives.
There's no file system. No raid parody.
- No networking.
- Nothing was going on.
- Yeah.
- This is a usable file system across six servers.
All right. You ready to see some big numbers?
- Yes.
- This one's not as exciting as you'd expect.
You have to, you run it and then it shows you,
it like gives you a file with the numbers.
That's not that fast.
- [Linus] That's not very fast.
- [Jake] Just give it a sec. Ooh!
- All right, it's faster. You said this was gonna be cool.
- It's trying different sizes.
This is probably, like, 4K. Right now, this script
is starting with GDS or GPU direct storage.
It sounds very complicated and, realistically
in practice, setting it up was probably
very complicated for somebody.
But the main difference is, rather than taking the data
from those NVMe drives, putting them into the CPU's memory.
- Yeah.
- And then into the GPU's memory.
We're skipping that middle step.
It goes right from NVMe to the GPU's memory.
Ooh, there we go. 15 gigs a second.
- [Linus] Was that CPU first then?
- No, it's doing GDS, but, like I said,
it's doing different block sizes.
You ready to see some big numbers?
Okay, so, this is GDS.
It says GPU direct storage only read tests
106 gibibyts per second.
So that's like 114 gigabytes per second.
- Over a network..
- That's like two...
- With the file system.
- Blu-rays per second, like, full quality, big boy.
- Like the whole thing
per second. - Blu-rays.
Holy (bleep)
- And again, we've seen numbers like this before,
but not with a file system loaded on it.
- No, we could take the...oh, you know Weka?
- We could actually use this.
- You wanna, like, SMB for a second here?
- I mean, well, we don't have a client that's anywhere
near fast enough, which I guess is a perfect opportunity
for us to talk about what this would be for.
Wanna show us that visualization demo?
Humanity's desire to look beyond the stars.
Is nothing new. - It's not!
- And one of the major stepping stones in the space travel
is finding a way to transport people.
In all seriousness though, sending people
to another planet is an incredibly complex challenge,
but one that modern computing can help us with.
And one of the current proposed plans from NASA
involves sending a six person manned craft
all the way to Mars.
After the 10 month journey, the crew would transfer
into a landing pod that is 16 meters in diameter.
So roughly the size of a two story house.
Due to the sheer size of the lander, as well as
the thin atmosphere on Mars, it's not possible to use
a parachute to slow down, like they have on past landings
like the perseverance rover.
So instead, NASA wants to use retro propulsion
otherwise known as creating thrust in the opposite direction
to slow the lander down from 12,000 miles an hour
to, you know, the ground, in less than seven minutes.
Simulating this landing is a very important part
of the research and development for the project.
And with the power of the summit super computers, 27,000
NVIDIA GPUs over the course of a week, they were able
to build a 100 plus terabyte model of the landing that is
over 1 billion points each with seven numerical values.
Density, vorticity, pressure, that sort of thing.
And I guess we have NASA security clearance now
because we have that exact model loaded up
on the million dollar server and here it is.
And Jake's got it running right behind me.
Now, in the past, engineers would actually take this data
model and render it out frame by frame into a video.
Still super cool and very useful.
But if you wanna tweak a parameter and see how
it affects the model, guess what? You're waiting hours
for a new render. With new technology like super fast
NVMe storage, NVMe speed file systems, like WekaFS
and the power of GPU direct.
Well now we can have the graphics cards render the data
directly from the storage. Bypassing the CPU,
allowing us to do this.
- [Jake] In, basically, real time.
- Real time manipulation. That is...
- I mean, it's a bit cinematic,
but we're getting like five FPS.
- Five FPS? That's, I mean, a lot better than...
- Okay, hold on, hold on, hold on. Five FPS?
Each of these frames is over 14 gigabytes of data.
- Yeah!
- That's before it becomes a frame.
The GPUs have to process that.
- So, each second we are streaming between 70 and 90
gigabytes from that storage directly to those GPUs.
So that's one to two blu-rays each second.
Can you, like, move it?
- Yeah, I can. Here, let's...
- Yeah, move it, play it, play it. Trust us.
This is really neat.
- I Wish there was a way for me to, like, translate
how much is happening in the background.
I guess we could look at how much power it's drawing.
- This is what you guys are gonna wanna see.
How many amps are going through our PDU here?
- And that's just for this top server.
- Yeah.
- Why does it need 17 amps and 4,000 watts right now?
What the hell?
- Oh, hey. This is interesting. Look, so one of our GPUs,
you can see it's refreshing every two seconds.
- [Linus] Yeah.
- [Jake] They're kind of drawing a little more power now
but still not anywhere near 400 watts.
- Yeah, that's interesting.
I guess this is just more storage dependent than it is
actual GPU rendering dependent, which makes sense.
That's... - They're pretty OP GPUs
- Sort of why they sent this demo. It's about the storage.
- [Jake] Like, moving it around a bit.
- [Linus] Oh yeah, they get a little spike here.
- You can see the utilization on the GPU is like 50, 60%.
Probably a lot of this is more the memory usage
and less the actual core itself.
You can imagine the type of workflow improvement
you would have from being able to just mess around with this
in real time rather than like rendering it, waiting three
days for it to finish rendering 'cause you're at
a university and you got a small budget.
- Works just like this segue to our sponsor.
- Vessi. Do you hate wet socks as much as I do?
Vessi footwear makes lightweight, breathable
and most importantly, water resistant shoes.
So no more squelchy socks.
Their Dyma-tex material not only keeps your feet dry,
but keeps them warm in the winter and cool in the summer.
How does that work?
The stretchy design shows that comfort is at the forefront.
At times, making you forget you're wearing shoes.
Vessi makes cruelty-free products, right down to the glue.
Their shoes are 100% vegan.
Whether it's a rainy city or a rocky trail,
the herringbone tread design is there to help stop you
from slipping around.
Your feet deserve a little treat.
So go ahead and click the link below and use promo code
LinusTechTips to save 25 bucks on your first pair today.
- If you guys enjoyed this video,
maybe check out the previous parts and, I don't know,
maybe subscribe to Floatplane.
Maybe we'll shoot an exclusive of packing them up.
No, I don't think we'll do that.
- I want, like, I gotta wonder if there's anything else
we could do. I thought about video editing.
- I would love to use these for the machine vision
cat squirter turret projects.