10 Gigabits on a Pi?! Jeff, what are you doing?
Last year, I got this ASUS 10 gigabit NIC working on the Raspberry Pi Compute Module 4,
and I was gonna make a video about it.
But it was the first 10 gig device I had in my house, so to test it,
I popped an older Mellanox card into my PC, and I bought this Mikrotik 10 gigabit router so I could
connect both to my home network. Then I found out the router sometimes doesn't switch at a full 10
gigabit speed in certain situations, even after I switched it to the lightweight "Switch OS".
So I bought this hybrid 10 and 2.5 gig QNAP switch, and it seemed to work a little better.
But then I found out my HP desktop computer also can be flaky sometimes with the Mellanox
card I put in it, so testing between this computer and the Pi couldn't be relied on.
So then I bought this 10 gig Thunderbolt 3 NIC for my Mac
and connected it to the 10 gig network. And it worked great, but then I found some quirks in
the bandwidth testing tool iperf3 when you use it with multiple network interfaces.
At this point, I started getting annoyed by the fan in the QNAP switch,
since it was just under my desk, and I thought it would be nice to move that switch
and my NAS with its four loud hard drives out of my office.
And I got a free UPS from a closed-down radio station,
so I had to find a rack to put everything in. And when I got a rack that was 19" deep,
I realized the giant UPS rails wouldn't fit without major modifications.
So I made those modifications, and got the UPS and some of my network gear into the rack. But then
I realized my cable runs to my old patch panel weren't long enough to make it to the rack without
splicing. Plus many of the runs were Cat5e, which isn't really rated for 10 gig networking.
So then I rewired most of the runs with this monoprice Cat6A riser cable,
and replaced all the old Panduit RJ45 jacks in my house with Cat 6-rated keystone jacks.
It was also around this time I got accepted into Starlink's public beta,
so I set up Starlink for a redundant Internet connection.
And while I was on a roll, I decided I also hated the fan noise of my old Intel MacBook Pro
so much that I replaced it with an M1 Mac mini
with built-in 10 gig Ethernet after I saw Linus and Anthony give it such a glowing review on LTT.
And yeah, so here we are today: I can finally test this 10 gig
card in a Raspberry Pi and be one hundred percent sure none of my network cabling,
gear, or other computers on the network are bottlenecking the Pi.
10 Gig networking can really be annoying when you're on a budget and running secondhand gear.
And in the end, none of the problems I ran into make a huge difference,
because the Pi itself has a single PCI Express Gen 2 lane available,
which means no matter what we do, the theoretical maximum is less than 4 Gigabits.
And I was able to get through 3.26 gigabits,
which is pretty solid compared to all the other devices I've tested on the Compute Module.
To get the card working on the Pi, I had to recompile the Pi OS kernel,
and use menuconfig to build in support for Aquantia NICs, along with the aQuantia
AQtion Ethernet driver.
There's a guide for cross-compiling the kernel and copying it over to the Pi linked below.
With the new kernel, the NIC showed up like any other network interface when I ran ip a,
and I could connect to my network using whatever speed it supported, up to 10 Gigabits.
But the Pi's bus only supports up to 4 Gigabits theoretically,
so in reality, when I ran iperf3 tests, I could only pump through a little more than 3.2 Gigabits.
And the nice thing is this NIC seems to be able to handle the packets much better than
others I've tested, since the Pi doesn't max out a CPU core with IRQ interrupts while it's
copying data. On other NICs I've tested that happens and leads to a lot less throughput.
Because this card is so efficient, I didn't have to overclock the Pi
or use Jumbo Frames to sneak through more bandwidth.
Now, in reality, are you gonna see many people putting 10 gig network cards on their Raspberry
Pis? Probably not. But it can be done, and that's kinda cool. Maybe the next generation of
Pi will be able to take full advantage of 5 or 10 gig networking, we'll see.
Now before I go, I should mention that this video, even as short as it is,
probably cost more than a few thousand bucks to make.
And yeah, that's my own fault for basically rewiring my whole house for 10 gig networking.
But still, if you don't want my wife to tell me to stop making these fun videos,
maybe consider supporting the channel on Patreon or GitHub Sponsors!
Or at least go in the description and use one of my affiliate links
to buy some new networking gear from Amazon for your own 10 gig adventures!
I have some other 10 Gigabit cards on my test bench too,
like Dell's Intel X520 with two 10 gig SFP+ ports, and a Mellanox Connect-X3,
and I'm still testing them. Subscribe so you don't miss my future networking videos!
Until next time, I'm Jeff Geerling.