- 11.420 gigabytes per second.
Wow! We just copied a hundred gigabytes of data.
And what, it was like 15 seconds?
- [Jake] Looks like 15, 16 grand depending on what way
you want the air to flow. (narrator laughing)
- I'm really glad we bought Honey Badger.
Okay then, it was back in 2018,
that we took our first dive into the insanity.
That is 100 gigabit ethernet.
With a direct connection
using a direct attach cable like this one,
between two systems, thanks to some network interface cards
that the folks over at Mellanox sent.
That yielded results that were fairly unimpressive actually,
due to limitations in both windows file copy
being single-threaded
and our lack of support for RDMA.
A technology that can greatly accelerate file transfer speed
over the network and efficiency.
Well today that changes,
because not only
do we have our shiny new 100 gigabit switch,
we've also got some shiny new fiber optic cabling,
a bunch of both 100 gig and 25 gigabit network cards,
and, we've got both RDMA
and multi-threaded file transfer working.
Things are about to get very interesting for our videos.
Ridge wallet wants to redefine the wallet
with its compact frame and RFID blocking plates.
Check out how they can keep your wallet bulge down,
and use offer code Linus to save 10%
and get free worldwide shipping.
(upbeat music)
Performance issues aside.
Any networking setup where you're relying
on having the systems connected to each other,
is a pain in the butt to set up and manage.
So, any kind of practical high-speed deployment
starts with the switch.
This, is the Dell S5048F-ON.
It's actually pretty similar to the S4048T-ON switches
that we already have in our server room.
Except that instead of 48 ports of standard RJ45,
that runs at 10 gig,
with six 40 gigabit QSFP pluses over at the end,
this one, has 48 ports of SFP28
which are capable of 25 gigabit each,
and then six QSFP28 that can run at 100 gigabit.
We just ran the fiber cabling
that we're gonna need for all of our editors
to plug into these ports,
giving them 25 gigabit per second connectivity.
As for the six 100 gigabit ports,
we can use these to link to our NVMe storage server.
Which means theoretically,
we should be able to do 100 gigabit per second
off of that server, all freaking day.
In fact, we could probably even do 200 gigabit.
We've got so many of these.
We could like aggregate them
and use a dual port card or something like that.
Let us know if you wanna see that.
Around the other side,
we've got dual redundant 750 watt,
80 plus platinum power supplies,
in a switch, that doesn't even have PoE?
Man, this thing's a chunky boy,
but realistically, the reason we need it,
is to run the fiber optic transceivers,
that physically light up the laser,
that shoots outta here,
and goes into the nick on the other end.
And also to handle processing.
3.6 terabit per second, of full duplex processing,
in a single 1U rack?
Well, yeah, like okay.
750 watt power supply i get it.
Conveniently, this came outta the box
with the correct air flow orientation.
You can actually have it so that the air goes in the back
and out the front, or in the front and out the back.
And it's just like a part
that you can buy on Delta, switch it around.
Should we open it?
We gotta open it, right?
Let's open it.
- [Jake] I think we technically have a warranty
but that's never stopped you before.
- Here we go.
- [Jake] Uuh!
- Ooh! whew, holly bananas!
This thing's got upgradable, sodium slots, boys.
16 gig and MC drive up here.
Got a couple...
What is this?
Eight gigs of DDR three memory?
So we throw another eight gigs in it Jake?
- [Jake] I don't think so. (laughs)
- I don't think so.
But I mean, it must have expandable memory for a reason.
Is this heat sink not beautiful?
- [Jake] Yeah, I think it's pretty cute.
I wanna pull up our video on the other switch
because I feel like this one has less cooling in it 'cause--
- It does, it has way less cooling.
- [Jake] Because RJ45 are what get hot.
- Hmm actually, it's funny you mentioned
the RJ45 one using more power,
so my understanding
is that there's actually more processing involved
on the switch side of things for RJ45
and that's one of the reasons that RJ45 switches
are more expensive than optical ones,
where a lot of the cost ends up being,
there you go,
in the transceivers that you have to plug into them.
You guys ready for lift off?
(device whirring)
Yeah, we're gonna unplug that for now.
Let's talk about the machines we're gonna be running first.
One of them is running a 3990X,
this one right here.
The other, is a 32 core Threadripper Pro.
We finally got our hands on one.
So make sure you're subscribed,
if you wanna see the video on that.
For now, let's set up this machine.
Our 3990X is gonna be acting as our file server.
It's running windows server 2019 and for its storage
to make sure that it can actually keep up with 100 gigabit,
we will be using a Honey Badger.
If you're not familiar with this puppy, it's from liqid.
And it's an eight slot NVMe card,
full of four terabyte M.2 drives from Sabrent,
that's capable of nearly 30 gigabytes per second
of sequential read speeds, on its own.
This way, we can avoid any storage bottlenecks in our test,
without resorting to a Ram disk.
Which as we discovered,
because the software is so junky,
actually will not alleviate bottlenecks
at all at these speeds.
Also, since this is the server,
we're gonna go with a,
100 gigabit connection to it.
That's something you don't see every day,
we've got our storage and networking plugged
into the 16 X slots.
And then our graphics card is limping along in a,
I think this is eight X slot, isn't it?
Yeah, it is. (laughs)
Oh, well, because we're going over such a short distance,
it's a little more cost-effective
to use direct attach copper cables.
So we're gonna use that
to go from our 100 gigabit network card,
to our 100 gig port on the switch.
Now, we're gonna change gears.
This is, oh, this is two, one moment please.
(Narrator laughing)
Fiber optic cabling.
If you get the really baller stuff
with like, corning indestructible glass inside them.
They're not that fragile,
but otherwise they can actually be very fragile.
We got our cables from Infinite cables,
and our transceivers from fs.com.
And you can actually see they're slightly
different part numbers, IS, DE.
So this, is gonna handle 25 gigabit connectivity
from our switch to our workstation.
We're using this Threadripper Pro workstation from Lenovo
to kind of stand in place of what we normally use.
Which are a Threadripper 3960X's
on just like regular AC gaming motherboards.
Start with the Honey Badger down here in the bottom.
This is why it's so all important for workstation use
to go HEDT, rather than with a regular desktop processor,
even if you don't need more than 12 or 16 cores.
It's because it's all about that connectivity.
High speed baby.
Now we're starting with 25 gigabit
because this is the use case
that we're actually testing for and validating.
But before this video is over,
we will put a second 100 gigabit card,
in our workstation here.
Now that the switches fired up for real,
we're running at 25 gigabit per second on our client.
And yup, both that 100 gigabit.
But there's another step, check this out.
If I go into the system info here
you can see we're not running regular windows 10,
or even windows 10 pro.
This is windows 10 pro for workstation.
Now that doesn't mean
that you're gonna get better performance
in Adobe Premiere or Cinnabuns
'cause it's for a workstation.
And in fact, it doesn't even require a re-install.
It's just a software key unlock.
What it does have though,
is some workstation specific features
that we need most importantly, SMB Direct
which is Microsoft's implementation of RDMA over ethernet.
RDMA or remote direct memory access,
is a technology that allows you to read and write
to the system memory of a remote machine
without interrupting the processing of the CPU
on that system.
So it essentially cuts
the entire network processing stack kernel
and CPU out of the data transfer equation.
Meaning that you can do faster transfers at lower latency
and at least theoretically
with barely any CPU usage on either system.
Now, this thing used to be a total maze to set up
like a flipping nightmare.
But with the addition of SMB Direct to windows server,
and now to windows 10 pro for workstation,
it is theoretically, plug and play.
No configuration required,
I guess we will see about that,
I'll believe it when I see it.
On our server machine,
we've pre-configured a windows storage space
and its own special RAID zero type mode here.
Is it fast?
The answer is, yes.
We're gonna change that to gigabytes per second.
(laughs) Yeah, so it's really nice compared
to the stack windows dynamic disk striping,
which was actually deprecated ages ago,
because it's actually multi-threaded.
Meaning that it can usually handle much higher speeds.
We've got a network share set up on this puppy.
So, let's go ahead and head over here.
Whonnock three, users.
There's a big fat 20 gigabyte file.
Let's transfer it.
1.47 gigabytes per second.
That's well shy of 25 gigabit per second.
But part of the problem
is that windows file transfer is single-threaded.
Now there is a multi-threaded version called robocopy
but we're gonna wait 'til we move to a hundred gigabit
to try that for you guys.
This is reading from the server.
We're seeing about 1.45 gigabytes a second.
But let's go the other way.
2.7 gigabytes a second.
Wooh! Flipping nuts.
Oh no, we dipped, we dipped.
We were at 2.3 gigabytes a second.
Oh, this has plugged.
This is plugged here now, unfreaking real.
And get this,
if we run CrystalDiskMark, over the network,
it's multi-threaded, let's have a look.
Let's do this bundle.
Okay, here we go.
Oh, actually, oh, this is really cool.
So I'm gonna show you this.
Now, normally, when you're doing this kind of
like crazy amount of data movement,
your CPU usage should light up.
But here we are doing 1.1 gigabytes a second,
and it's barely freaking touching it.
That's right to the server now here.
Give it a sec.
2.8 gigabytes a second.
(stamps feet) I couldn't even blink.
Even more mysterious though,
is the fact that we're running CrystalDiskMark here,
and our networking usage shows basically nothing.
That is because this transfer actually bypasses
the traditional TCP and UDP network stacks,
which windows task manager monitors.
So the only way for us to see if RDMA is even working,
is in a different application called, Resource Monitor.
2.2 gigabytes per second, there it is.
Oh, these are probably gigabit, actually.
- [Jake] Bytes per second.
- Are they gigabytes?
- [Jake] Yeah.
- Okay, and then out down,
there it is, three gigabytes per second.
Where's my other a hundred gig card,
is this it?
- [Jake] Yep.
- We don't have a PCI back plate on this one,
but that's okay.
Should still work fine. (chuckles)
- [Jake] Just be gentle.
- Yeah (chuckles).
A hundred gigabit time.
CrystalDiscMark, (clears throat)
here we go.
(slow upbeat music)
- [Both] Oh!
- Excuse me (coughs).
11.420, nice, gigabytes per second.
Wow, okay, less speeds. - [Jake] Come on,
Not wow, you're supposed to say, holly! (beep)
- Yeah, holly *
What the (beep) (beep)
Bald, bald, Wow!
(indistinct)
Now let's try robocopy, which is command line only,
in spite of the fact that it's been around forever.
But that's okay,
you can get gooeys for it.
This one's called, ChoEazyCopy (laughs).
Eazy with the z (laughs).
I love you, developers.
You guys are hilarious.
For comparison though,
naturally we should run a regular windows file copy first.
So, just shy of two gigabytes per second
copying from our local machine to the server.
That's well shy of a hundred gigabit,
but like, dang! It's not like it's slow either.
Now let's go the other way from our local machine to...
Oh, from the server - [Jake] No.
to our local...
Oh God damn, whatever, (Jake laughing)
it doesn't matter.
Going the other way,
we get, around two gigabytes per second.
And now for that,
let's bring out the big guns, ChoEazyCopy.
Making sure it's running with multi-threading on.
Yes, and go.
I am expecting, some big boy number six.
Wow! Did that just peak at six gigabytes a second,
just four now.
holly crap, we have already transferred 51 gigabytes.
What is even happening here?
What is even going on?
We have transferred a hundred gigabytes.
A hundred gigabytes, it's done.
- [Jake] It's not done.
- It's not done?
Well we've got a hundred gigabytes.
- [Jake] It's still like pre-allocated.
It's almost done.
You can see it in the...
there it is. - Oh yeah,
80%, 90%, boom!
We just copied a hundred gigabytes of data.
And what? Was it like 15 seconds?
- [Jake] 20 seconds, something like that.
- 20 tops? Can we try 16 threads?
'Cause that would allow us to copy all of the files.
- [Jake] I tried it,
and it wasn't better, but short, right?
- Not better? Unless I don't know.
- [Jake] Just send it.
- 4.2 billion bytes per second, 4.2 gig.
Okay, so yeah.
Not appreciably better.
- [Jake] But better than windows file transfer.
- Definitely a lot better than windows file transfer.
And if we were on Linux,
we probably could see full 100 gigabit.
So about load that work out to about 12,
12 and a half gigabytes a second.
Yeah, something that.
Man! These stations are gonna be like,
(snaps fingers thrice) real snappy.
Now we're gonna go the other way copying from this machine,
to the server.
And see what we're looking at here.
Holly! (beep) 14 gigabytes a second?
- [Jake] What?
- No, no, it's only wait.
Oh yeah, I was looking ...
Okay, yeah, yeah.
The 14 million was just... - [Jake] I was like...
That was the inbound, okay.
- I was like, what?
- Yeah, about 2.7 gigabytes a second.
So, we're definitely limited somewhat.
At least our CPU usage is basically nothing
which is great news for me,
'cause it means we don't have
to go buy a bunch of 32 or 64 core threadrippers
for the workstations.
24 is gonna be fine, just fine.
Should we fire up a Premiere timeline?
Do we have it on here?
- [Jake] Yeah, we can if you want.
- Inbound bytes per second,
actually only about 250 megabytes per second
running at full resolution using about 66% CPU.
This is red 8K footage, running at full res
in Adobe Premiere.
Which is ridiculous, you would never do that.
Wow! The timeline is super responsive with BRAW.
Wow! That's not too shabby, is it?
Wow! What we're looking at now,
is a mix of 8K and 12K BRAW footage.
Okay, so that's well in excess of one gigabyte per second.
Of course, it doesn't show up here about poor CPU.
Even at 1/4 res, this thing is chagging.
So let's go ahead and scrub around on that timeline.
Holly crap! Okay, we peaked at three gigabytes a second.
Yeah, it's not bad.
Realistically, what we're after with this,
is not necessarily more throughput to this workstations.
It's more about latency,
because what happens
is if you have any kind of latency spike
on the server side, not just one,
but all of the workstations are liable
to crash at the same time.
And One-X server was good for a long time
with our 10 gigabit, RJ45 connection.
The problem is that we've added so many more editors,
that we're starting to run into trouble again.
So this is hopefully, the last solution
that we'll need to that problem.
But realistically, probably not.
What's the, in your online security toolkit?
Adding a VPN lets you mask your IP
and encrypt traffic to and from your devices,
and private internet access has reliable service
with thousands of servers in dozens of countries.
They offer, no bandwidth caps, configurable encryption
with a kill switch
to keep you in control of your connection.
And when combined with private browsing,
Pia can make websites think you are in a different country.
Try it out, risk-free
with their 30-day money back guarantee,
at the link below.
You can connect up to 10 devices at once,
with clients from windows, mac OS, Android, iOS, and Linux.
What are you waiting for?
If you guys enjoyed this video,
maybe check out..
- [Jake] The first time we tried to do a hundred gig.
- Actually, you know what?
Go learn more about the Honey Badger.
This thing is freaking cool.
(Jake laughing)
It's sick, check it out.
(upbeat music)