- This is the second fastest server
we've ever gotten our hands on.
It's for scientific applications and machine learning
and it might actually be better in some ways
than the fastest one, which happens to be the
petabyte of flash project over there.
Why? Because this one is freaking water cooled.
It's got six GPUs in it.
It comes in two flipping and wooden crates.
That's really all I know about it
because it's been stuck at the border for over a month
due to some kind of stupid COVID delay.
So, we're gonna open this puppy up
and show ya exactly what's going on inside.
Whoa.
- [Alex] Also it's only four GPUs.
- [Linus] Oh, I thought it was six.
- [Alex] Yeah, sorry.
- That's still pretty good. - [Alex] Yeah.
- Pretty good, just like our sponsor,
Signal RGB with signal RGB you can control and sync
your favorite RGB devices all from one app.
Best of all, it's free.
So, download signal RGB at the link
in the video description.
(upbeat music)
These are wooden crates sealed up with porks screws.
Alex has his way but I think I'm gonna go try my way.
- [Alex] No, don't use that.
- [Linus] I can't find a crowbar.
- [Alex] There's no way he's getting the crowbar
before I'm done.
- Oh, well that was no fun.
(Alex laughing)
- [Alex] What is in here?
Why do we have two boxes?
- I don't even under...
Yeah, I was gonna say, I don't even understand
why there are two boxes?
- [Alex] Are these are GPUs?
- This ain't no ordinary GPU ladies and gentlemen.
- [Alex] No.
- Now there's a big difference
between the water cooling that we as gaming enthusiasts use
and what you'll see in workstation or server
or data center applications.
- [Alex] I think this is all custom by Comino look,
look at this back memory cooler.
- [Linus] What boards are these?
They have NVLink fingers, so, they must be 3090 class.
- I think they're all A6000s.
Yeah, I've been trying to figure out the pricing on this.
I don't know yet, it's all just like, you know,
if you have to ask you can't afford it kinda stuff.
- I didn't think gigabyte was a Quadro board partner though.
Check out these quick connects, love it.
So your cold comes in here.
Presumably does some stuff heats up and comes out here.
- [Alex] It does look like they're five grand a pop US.
That's like MSRP,
they're currently going for like nine grand.
- If we screw up the packaging enough
we might not be able to ship it back.
Oh wow, it really did not make it through there.
Thanks, Jerry rig everything box cutter.
You can tell it's serious when it just
does not look performance at all.
512 gigabytes of DDR4 3200,
six A6000 GPUs,
two Xeon platinum 8368Qs, two SFX-L power supplies.
What a reopening?
- [Alex] What the heck is this?
- [Linus] It looks like meat.
- [Alex] Oh.
- [Linus] Oh my God, it's just power cables.
Well, that was...
- [Alex] Wait, but how many power cables do we need?
- I'm a little confused
because there are multiple power supplies listed
on the bomb.
Wait, what? What is it?
- Oh, that's not power cables.
- Looks like some kinda handle.
I think I gave you the light side.
Oh wow, that's heavy, yep.
- It's designed to be rack mounted, it comes with rails
but I can feel little nubbins on the bottom,
I think they're rubber feet.
I guess you can kinda use it however.
I mean that would be a big advantage
to going liquid cooling like this with your compute node
is that you wouldn't have to put it in a server room.
You could actually, ow,
have it on your desk and without being super disruptive.
I mean, yeah, that machine out there might be more powerful
but it's obnoxious
to be in the same building as let alone room.
- [Alex] Look at this.
- Wait, this side's just rad.
- I am really confused and you're about to be confused too.
Let the camera see your confusion.
Whoop, everything's fine, it's just
a very durable LTT store.com water bottle.
This particular one belonging to Alex.
- [Alex] Well, but yeah, what?
- [Linus] It's got GPUs in it already.
- [Alex] So what are those
- [Linus] They're GPUs smart guy.
- [Alex] Oh, well, obviously they're GPUs.
- [Linus] It has three power supplies.
Shout out Comino for their custom IO sticker
that's actually a really smart way of branding your machine.
- [Alex] What would we use this for?
- [Linus] Almost doesn't matter.
Okay, well Kyle has that thing upcoming.
He's got that raccoon that's like pooing in his yard
and he wants to train like a machine learning model
to recognize it
and like shoot it with a airsoft gun or something.
Apparently it takes like four days
to do the training on his like consumer GPU.
This you can fit so much machine learning in here.
What's the front Alex?
- [Alex] I think this is the front.
Wait, but is it?
- No. - Yeah,
yeah, this must be the front.
- This has to be the front
'cause it's got a thermal sensor in front of the radiator.
So, you'd wanna know your intake temps, right?
Oh wow, are these ever tight?
That stripped one could be a problem,
we could end up drilling this out Alex.
- Oh, I really don't want to do that.
- No, it's dead.
And it mangled your torque spit.
- Oh, great.
- One power supply, ah, ah, ah.
- Wait, why are you doing that?
- I'm powering it up.
- But I'm about to drill it.
- I mean, that's fine.
- What about we don't do that until we know that I did it.
- Whoa, here it goes.
- I guess it's on now.
What?
- It just had a little panic moment and shut off.
- [Alex] Cooling system standby.
- [Linus] Wait, is it even full of water?
- I can see right here there's a reservoir.
The water's right up at the top. I think it's fine.
Should I just drill this and get the top off?
- Yeah.
Look at that now we're both being productive.
Do you wanna get footage of him?
Wait, stop Alex.
- [Alex] Too late.
- Brandon footage. Don't...
Just pretend, pretend to have the breakthrough moment again.
- [Alex] All right.
- Hold both buttons to turn on/off.
Cooling system start, fan RPM monitoring on here.
It feels like it's intended to be up like this a little bit.
- [Alex] Yeah, kind of.
- Yeah,
whoa,
lots of bubbles.
I mean surely if they picture it like this
and the text is this way, it's intended to be run this way.
(Linus screaming)
- [Alex] There we go.
- Good thing I didn't stab that,
that would've been pretty bad.
Wow.
- [Alex] Holy crap.
Look at that back plate.
- What are the GPUs that are in here?
What was the point of those other ones then?
These have HDMI ports on them.
So, I do not think that these are A6000s.
- [Alex] Are those 3090TIs?
- They could be 3090TIs
but I don't understand why they have these
crazy eight pin connector adapter doodads here then.
Wait,
I don't think I've actually powered on the system, Alex.
- [Alex] Like I don't have any lights on the keyboard.
- This controller system up here is trippy.
Here's our temp sensor for the front, hanging off of it.
And then you can see the 24 pin connector goes into here.
Oh crap, is that USB?
Yeah, I don't wanna wrench that off too hard.
And then you can see fan leads, power,
presumably there's a cooling flow rate monitor.
All that appears to be completely custom.
Should we take a closer look here before we fire it back up?
- Yeah.
- [Linus] These guys have come a long way
since the last time we checked out one of their systems.
I recognize this triple power supply setup from last time
that's super cool.
We've got three 140 millimeter fans.
So, these are taking advantage of the
full four U height of the chassis
and then pushing that air through
what looks like a triple 120 millimeter radiator
which I honestly gotta say feels a little optimistic
in terms of cooling.
We've got dual Xeons and then quad GPUs
that are all supposed to be cooled
by this single thick radiator.
I love this distribution block here
for every heat generating device,
you can see we've got the cold and hot sides
coming into this Delrin distribution plate
and then we've got cold coming out to the components.
These are all completely custom water blocks
and then hot coming back to the distribution plate
all joining together and going back to the radiator
which then goes to our pump reservoir combo unit up here.
What we don't know is what the heck these things are.
There's two pumps in here by the way.
I'm not sure where they are, but that's pretty.
Oh, there they are, I think they're DDC variance.
On/Off commands not available.
Just rip out the power just there.
Boom, they're gone.
- Another news I checked in,
we actually have a terabyte of ram.
- 64 gigs each.
This single dim has more ram than your workstation.
- [Brandon] I wonder how many people are buying this thing?
- I think it's less about
how many people are buying this thing
and more about how many of them some people are buying.
Is this on or what?
Select the Input.
- Like I don't even have keyboard lights.
- [Linus] Interesting.
- Well, first of all let's talk about
what went wrong yesterday?
- Oh yeah, let's do that.
Did I screw it up?
- Yes, you did.
- Oh, was it when I pulled this thing off?
- [Alex] It was, yeah.
- [Linus] Cool.
- [Alex] So it turns out that the power switch
is like in here.
So, this starts up the cooling checks everything.
Make sure it's good.
- [Linus] Yeah.
- [Alex] And then... - Powers on the board.
- [Alex] Yeah, powers on the board.
- So it just didn't.
- Yeah, because when you did this,
the power switch connector,
you can see a little jumper right in there
where you like really reefed on it.
- Cool.
- Yeah, that was unplugged.
- Okay.
- Took a call with the people in Austria.
Jake, you spent what like two, three hours on this?
- [Jake] 20 minutes.
- [Alex] Oh, only 20 minutes, not too bad.
- [Jake] What did you just do?
It literally just blue screened.
- 20 minutes...
It what? It blue screened?
- [Jake] Yeah, yeah. - [Linus] Shut up.
I didn't do anything.
(Alex laughing)
- [Jake] You just have an aura.
- No, I don't have an aura.
- [Jake] K, It's working now.
- Yeah.
- [Jake] Don't, just don't touch anything.
- I'm not touching it.
- [Jake] Stop, stop.
- I'm screwing in a VGA ca...
- [Jake] Look users use keyboards,
just go to the keyboards, okay?
- [Linus] Do I get to know what GPUs they are now?
They have the A6000s in there
which means those must be 3090TIs then.
- [Alex] They're actually 3090s,
I talked to them and they sent them
just so we could show them getting swapped
and show how easy it is.
- [Linus] Okay, cool.
What are we gonna do now?
Because my understanding is you've spent some time
working on a benchmark that can actually take advantage
of all of this GPU compute.
How much memory do these fricking things have?
48 gigs each? We have 200 gigabytes of video memory.
- Like just the CPU screen even like.
- 76 cores, a 152 threads,
I don't know Intel's lineup very well anymore.
So what? These are each 38 cores.
- Yeah. - Are, is that good?
Are they fast?
- I don't know, should we test it?
- Okay, so we're going up against
what, what do we have here in the reference ones?
What ways it compared to a 32 core 2990WX?
- I think favorably.
- Thank you for that.
I wanna feel the heat though.
- Oh, just watch this for a second.
- Okay, all right, I'll watch.
Okay, it was only necessary to watch for a sec.
No meaningful heat buildup was going to occur.
All right then.
- [Alex] Wow, that's fast.
- [Linus] 71,000,
what the hell is in 8368?
They turboed a 3.7,
base 2.6, 270 watt TDP
and a maximum memory size of six terabytes each.
- During cinebench
they reached a maximum temperature of 55 degrees.
- [Linus] They also weren't running very long.
Hey, what happened to our power meters?
- [Alex] They're right there.
- [Linus] We only have two.
- [Alex] We also have the sketchy one.
- [Linus] Let's use it one more time before we throw it out.
Cinebench, here we go.
I can pretty much guarantee you
that the eight pin CPU power connectors
of each of these two power supplies
are going to the two eight pin inputs on the motherboard.
So, we're sitting at about 850 watts right now.
Shall we do something to the GPUs?
- Might as well.
- Okay, so V-ray cuda.
- [Alex] Sure.
- Or RTX?
- [Alex] Now, let's do cuda,
I think that will hit it harder.
- [Linus] Here it comes.
- [Alex] Oh my God, holy, how is it not crashing?
- It's pulling over 2000 watts.
We have to be on two separate circuits.
- We, we are, yeah.
- [Linus] I wanna feel the heat.
- [Alex] Yeah, we're just 2000 watts from...
I'm gonna get a sun pan boys, this is crazy.
We're already at the point where the incoming air
is 22 degrees and the outgoing air is 44
(Alex laughing)
which is pretty nuts.
And what's crazier is that if you check this out
this V-ray benchmark only runs for one minute at a time.
So, we're barely even stressing these GPUs,
I have to manually restart it every single time
which means I think it's time
for us to switch over to your solid works benchmark.
- Okay, I'm concerned that it's not really going to work.
So, I've been having a problem in the past.
It just like renders a bunch of passes.
On this they take such little time
that I've been having trouble
getting them all to get fully loaded.
- To its credit, when I'm not unplugging power supplies
it appears to be rock freaking solid stable.
And in terms of balancing the load
here let's hit it with V-ray again,
no single one of our power supplies
is doing an outsized amount of the work
when you're hitting the entire system.
I think the most is around 850 watts.
And given that these are 1000 watt units from Silverstone,
that's not bad.
I do have concerns about those poor power supplies
drawing in like, no, it wouldn't even be 40 degree air
because they're on this side.
- Yeah.
- Of the radiator.
So actually they're getting pretty decently cool air.
- This is weird 'cause if you
put your hand behind the radiator, it's super hot.
But if you put it down where the power supplies is,
it's still nice and cool.
- Solidwork's crashed,
maybe some kind of driver issue, we're not sure
but blenders opening.
- [Alex] I'm curious a lot of my laptop benchmarks
take about one and a half minutes on really fast ones
to like three minutes
on like something that's a little bit slower.
I'm kinda curious what this will do.
- [Linus] I'm thinking a lot less than that.
Holy (beep) is it done?
No, okay, wait is it...
(Alex laughing)
Did we just witness a sub eight second blender BMW render?
Samples 3000,
sure.
- Maybe just increase the number of light paths as well.
- Yeah, total lots.
- Okay, (Alex laughing)
it still did it in 16 seconds.
This is ridiculous.
- That was 3000 samples and 10 times as many light paths.
- Okay, can we do like 10,000 samples though
- [Alex] Is it even hitting all of our GPUs?
- It's not even hitting all the GPUs.
(Alex laughing)
Is it possible that's misreporting?
- Oh, they are all hot, it probably is just misreporting.
- I think it's misreporting
because it just did that in 31.8 seconds.
(Alex laughing)
- It's gonna be a problem trying to stress this.
- Classroom,
well, you saw it here first ladies and gentlemen
that is a 15 second classroom benchmark, 14.66 seconds.
- I normally expect that to be like 10 to 15 minutes.
Who would've thought that the render machine
is incredible at rendering.
- From 300 to 10,000 samples.
It's still chewing off 20 seconds at a time of the CTA.
It's definitely using our cores,
as for our GPUs we are at each one of these
is pulling about 280 watts.
We've been added a minute.
We still have about two minutes and 15 seconds left
can confirm we've managed to fully load down the system.
860 watts on that power supply.
625 there, so, that's just shy of 1500,
about 2000 watts total.
- Yeah, 2100, maybe a peak.
- Oh wow, it's really hot now
that the coolant seated up.
Oh, that radiator is hot to the touch.
I'm amazed at how reasonably quiet it is though.
- Like you could have this in your room.
- Yeah.
- And aside from your room being
a billion degrees in like 20 minutes,
you could live with it.
And like GPU's been hitting this for a little while
but they're all at like 65 degrees max
while drawing 300 watts.
And that's on top of the CPU.
Each of which also are doing 300 watts.
- Oh, it's done,
it finished in three minutes and eight seconds.
It looks better though.
- [Alex] Well, I'd imagine there's 10,000 samples.
- Well, look, I've never looked at it rendered like that.
Wait, I think you're right.
- Yeah, I am right.
- They're NVLink, I think they'll run in four way SLI.
- [Alex] They will.
- Is that even supported?
That's not even supported on the 3090TI.
- I don't know.
- I don't think so.
Now, rendering device, wait, did you already click run?
- [Alex] I haven't clicked run.
- [Linus] Okay, so there's only one
which would seem to suggest,
it's gonna hit all four of them.
Here we go.
Here's where we find out, if it's running in four way SLI.
Oh, I'm fidgeting.
- For some context, I spent a day
and a chiller, and a almost killing a 3090
to get a score of 12,000 in this.
- 500, 450, 6... holy (beep) it's hitting them all.
So, we're sitting at about 1700 watts
that's a little lower than what we saw in blender
but
we know,
okay?
That the CPU is not gonna be hit too hard
during the GPU por...
Is this running at 120 frames a second?
- Yep.
- It's up to a 160 now.
What we know
is that this will not stress our CPU nearly as much
as that blender render was.
So, that 300 wat discrepancy
is just one of our CPUs sitting ideal.
These GPU's are going full grunt.
- I guess we should probably calculate
about how much it would cost.
- 7,500 times 2, there's your CPUs.
Okay, use a calculator I give up.
- [Jake] About 60 grand.
- 60 grand?
Don't forget that they just ship you random RTX 3090s.
- [Jake] Did it just crash?
- Oh, I think it did crash.
- Oh.
- It crashed.
Oh, she's done bud.
In fairness to Comino four way SLI has not been like
properly supported by Nvidia in a very long time
and it's probably a driver issue.
Since we're happy with the A6000s in there
and we're not gonna swap in the RTX 3090s,
I might as well just explain why they included these.
It was so that we could show you guys just how easy it is
to swap out the cards from your workstation.
Comino will provide these pre-done up cards.
So, basically it's just a one screw,
undoing these quick disconnects, slotting in the new GPU,
I mean, I guess you should also remove your power connectors
at some point,
and then these actually come pre-filled with coolant.
So, theoretically there's almost no liquid loss
when you go from the old card to the new card
and you should almost never have to top up the coolant
which I thought was pretty cool.
Attempt number two is running a lot slower,
power usage is lower.
It's around 1300 watts.
- [Alex] Oh, it's crashing
- Oh, and it's crashed.
- I don't know if we're getting 3D mark
on this system.
- I don't know if we're getting away
without telling the viewers about our sponsor.
- Ting mobile.
Do you like saving money?
Ting mobile is a low cost carrier
with rates to help you do just that.
Almost every phone on the market will work with Ting mobile
and they have the perfect plan for everybody,
no matter what your needs are.
Start with unlimited talk and text for 10 bucks a month
or data plans from $15 a month.
Their set 12 plan with 12 gigs of data
is only 35 bucks a month.
And if you need it, unlimited data plans are $45 a month.
You can even share your data on a family plan
and save even more.
Ting mobile also offers, pay per use plans
with their flex plans charging just five bucks per gigabyte.
Even with those savings, you still get nationwide coverage
and award-winning support.
Consumer reports named Ting mobile
their number one carrier in America
which is a very large place.
So check them out at linus.ting.com
and receive a $25 credit.
- And if you guys are looking for another video to watch,
how about a system that's even more expensive
you can check out this thing over here.