This is absolutely ridiculous in the best possible way.
This is a 48 terabyte Raspberry Pi NAS, and it cost _more than five thousand bucks_!
These SSDs are overkill for a Raspberry Pi, not to mention the insane 8 terabyte NVMe
drive hiding out on the bottom!
To be clear, _having 48 Terabytes of SSD storage_ isn't ridiculous, but strapping that onto
a Raspberry Pi most definitely _is_, and I'll show you why. But you might be wondering at
this point, "Jeff, how can you afford all those SSDs?" Well, let me tell you about this
video's sponsor, Lambda!
Remember that video where I used a Raspberry Pi to detect Red Shirt Jeff's face, and it
barely managed 3 frames per second?
Well, Lambda's like strapping rocket boosters to your AI and Machine Learning workflows.
They have specialized hardware and cloud GPU compute resources built for machine learning.
Their servers make things like PyTorch and TensorFlow work way faster than anything else
on the market, and even if you don't wanna buy one of their insane servers, you can use
their GPU cloud! Check 'em out at lambdalabs.com.
Stupid can be fun, and that's what this channel's all about.
I figure since everyone's gonna point it out in the comments anyways, I'll give a quick
rundown on why this whole build is insane, then give the reasons I did it anyways.
First, here are some reasons not to build this NAS:
These drives are _expensive_, at $88 bucks per terabyte for the EVO SSDs, compared to
about $30 bucks per terabyte with 3.5" drives. And I could use cheaper drives with extension
cables, but then it gets messy and loud.
Anyways, these SSDs are way overkill, since the Raspberry Pi can only put through a little
over 400 megabytes per second, TOTAL, on it's entire PCI Express bus. So no matter how many
drives we slap in here, they're gonna bottleneck! Especially that fast NVMe drive on the bottom.
The Pi also doesn't have ECC RAM, or, well, technically it does, "from a certain point
of view". And I won't get into what it is or why it's important for enterprise-grade
storage servers... but this Pi definitely doesn't have what it takes to be a top-tier
storage server.
But it _is_ fun. And in the course of making this video, I learned a lot.
But is all that learning worth _five thousand bucks_? Probably not. But for this video,
it's go big or go home. So first, let's look closer at the Radxa Taco.
It's this main board that connects a Compute Module 4 to a varity of IO through this PCI
express switch chip.
The Compute Module 4 is my favorite Pi because it gives me access to a PCI Express gen 2
lane. I can plug in almost any PCI Express device, and, well, a lot of them work with
a little coaxing... And maybe a kernel recompile—I have a shirt for that on red shirt jeff.com.
The official Compute Module IO Board only has a single slot. But PCI Express, similar
to Ethernet, can have 'switches', which let you plug in multiple devices.
And the Taco's switch lets the Compute Module 4 interface with 2.5 Gigabit Ethernet, an
M.2 NVMe slot, AND 5 SATA III ports.
As a final bonus, the production board'll have an extra M.2 slot for things like WiFi
6 or machine learning accelerators!
The board also has a few headers for a fan and IO, a 12v barrel plug for power, a power
button, HDMI, USB 2.0, and an extra gigabit port, so it's no slouch when it comes to IO.
Before I put it together, I put on these metal standoffs just to hold the board up off my
desktop. Radxa'll have a case for this soon, but I couldn't get one in time since they're
still working on it.
I popped a 4 GB Lite Compute Module on, then I screwed in this 8 TB Sabrent NVMe SSD. Why
8 terabytes? Well, thank Lambda for that! And this SSD is actually dual-sided—it has
memory chips on both sides of the board, and it still fits.
Next I flipped over the board and installed five 8 TB Samsung 870 QVO SSDs.
Again, why 8 terabytes? Well, two reasons. One, sponsor. Two, I wanted a quiet, low-latency
NAS, and SSDs are the way to go. These 8 TB SSDs are actually slightly cheaper per terabyte
than a similar spec 2 TB SSD, and there aren't any 2.5" mechanical disks even _close_ to
8 terabytes!
And yes, there are even bigger enterprise SSDs, but a channel my size? I'd have to win
the lottery to afford 'em. If you work for Linus Tech Tips and wanna ship me a few of
the old drives from Whonnock, comment below.
I put in a microSD card with 64-bit Pi OS, and was ready to plug in the Taco. To be honest,
I was a bit nervous. There were five thousand dollars of hard drives attached, and I'm about
to plug in a 96 Watt power adapter!
That's right; this isn't your father's Pi, this thing needs a lot of power! I have a
link to the power supply I used in the description.
The Taco booted, but I also wanted to make sure it stayed cool, so I plugged this 5 volt
Noctua fan into the Taco's USB port.
Even without active cooling, in open air the controller chips only reached around 45-55°C,
so I'm not too worried.
I did run a stress test on the Pi and it reached 79.8°C, but it never actually throttled in
my tests. If you want the Taco to stay cool, I'd recommend
a heat sink and a fan.
I logged in and checked what I could see on the PCI Express bus. `lspci` tells you everything
about the PCI devices on your system, and it'll even tell you if there's a 'kernel module'
loaded for 'em. Raspberry Pi OS doesn't include a lot of the drivers you'd find in a typical
Linux distribution, so it'll probably need some drivers.
And looking at the devices the Pi sees, I can see there are five entries from the asmedia
switch chip, and then the JMicron SATA controller. And look at that, it says the 'ahci' kernel
module is in use, so it looks like Pi OS is already set for SATA support!
Next up is the Realtek Ethernet controller. If I look at the bottom I don't see any Kernel
module loaded for it, so that means the Pi doesn't have the driver for it yet. I'll put
a pin in that for now, and move on to the last entry, for the NVMe drive. It's using
the `nvme` kernel driver, so it should be good to go too!
And to verify all the drives are visible, I'll run `lsblk`, which lists all 'block'
storage devices on the Pi. And look, there are all of the Samsung SSDs, sba to sbe, and
there's the NVMe drive, good old NVMe0n1!
Now let's get back to that Ethernet controller. I just spent a few months upgrading my home
network so I have at least 2.5 gig Ethernet everywhere, and the Taco's a perfect way to
test that out! File copies over 2.5 gig networks can get up to about 250 MB/sec in ideal conditions,
which beats the 120 MB/sec you get with 1 gig.
And unlike 10 gig networking, 2.5 gig works fine in my home's older Cat5e cable runs,
which means I don't have to pull new cables for faster speeds.
Now, to get the NIC working I downloaded the driver from Realtek's website and after installing
the Raspberry Pi kernel headers package, I ran it's setup script.
I checked if the NIC was working by glancing at the `dmesg` logs. And there it is! The
`r8125` 2.5 Gigabit Ethernet driver says it's loaded, and it looks like if I plug in a network
cable, it shows here the 'link is up'.
I wanted to make sure it was connecting at 2.5 gigabits, so I also ran `ethtool eth1`,
and that shows me it's running at 2500 megabits, full duplex, meaning I should be getting the
fastest possible speeds through this Taco to my network.
To make sure it's actually _working_, though, I turned to the trusty network benchmarking
utility `iperf`.
And in a funny turn of events, it was super fast at 2.35 gigabits from the Taco to my
Mac, but it was really slow and jittery in the other direction. As it turns out, the
10 gigabit transceiver I had was flaky at 2.5 gigabit speeds. This is one of the many
'fun' problems you run into with 10 gigabit networking!
And that's why I think 2.5 gigs is the next logical step for most people. Red Shirt Jeff
told me he'd rather go to a hundred gigs, but I don't have the budget for that. At least
not yet. Let me know below if you think I should accept a fat, juicy RAID Shadow Legends
sponsorship!
Speaking of RAID, it's time to check on drive performance.
I wanted to test a number of setups, to see what options you might have with a Raspberry
Pi-based NAS. I'm gonna test drives individually, then in
RAID 0, RAID 5, and finally in RAIDZ1, which uses ZFS.
Oh, you didn't realize the Pi can run ZFS? Well, it can.
ZFS 2 is easy to install on 64-bit Pi OS. You install the Pi kernel headers and run
this apt command.
It took a little while, but I didn't have any problems.
Now, I'm not gonna get into the weeds on how I set up my RAID arrays, how I set up my zpool,
or how I ran my benchmarks. There are very detailed links below.
But I will say, ZFS offers a ton of interesting features, and I might do an entire video on
it at some point, once I have a better understanding of all its moving parts. It's kinda like systemd:
it does about a hundred things, from RAID, to being a filesystem, to managing mounts,
to snapshotting and replicating data, so I need to spend a lot more time using it.
Speaking of spending a lotta time, I spent the next couple days benchmarking, and the
results were a little surprising! The NVMe drive tested just as I expected; the sequential
operations basically max out the Pi's bus, so you get a little over 400 megabytes per
second.
And large block size random reads and writes are also pretty fast.
But when you get down to tiny file sizes, the NVMe drive pulls away a bit. Neither drive
is slow for a Pi—both are ten times faster than the microSD card. But both drives would
be even faster in a modern desktop—_Especially_ the NVMe drive, which is hampered by the Pi's
single gen 2 lane.
RAID performance was also interesting. I knew what to expect from building previous Pi storage
devices, and all three RAID types I tried were great for random reads.
RAID 5, because it's using the Pi's CPU to do parity calculations, slows down considerably
on random writes. And even though ZFS RAIDZ1 is conceptually the same, it's caching mechanisms
seemed to boost the speed there pretty well.
Small random reads is where ZFS seems to really shine; but everything falls apart on 4K random
writes. None of the RAID setups that require the Pi's CPU to crunch data are all that fast.
But these synthetic tests highlight the curse of the reviewer: no benchmark is perfect.
There's no substitute for real-world tests.
So I also tested Samba and NFS file sharing. I'm only gonna share the Samba results here
for simplicity's sake, but they're pretty representative.
All three RAID setups perform admirably, and at least for read speeds. I got more than
a gigabit all around.
But I found out that CPU interrupts seemed to be causing some bottlenecks, especially
with NFS. While I was copying 8 gigabytes back and forth, I monitored the Pi with atop,
irqtop, and other utilities, and found that one major bottleneck seems to be the fact
that all the data has to be passed through the Pi's CPU—which is no powerhouse—before
it can be written to the disk, or sent out over the network.
I should note all my testing is done with the default clock speed, which is 1.5 gigahertz.
Overclocking helps a little, but what surprised me most was I thought ZFS would perform much
worse than normal software RAID, but it's actually not that bad.
And I found something else, too. If I used smaller files, that could be cached through
the Pi's RAM, ZFS especially was _many times faster_.
So I could actually measure file copies in the hundreds of megabytes per second _under
certain conditions_. And using a Pi with more RAM would definitely help.
But if you want a 'serious' storage server, a Raspberry Pi ain't that. But it is adequate,
and the Taco was rock solid the whole time.
The last thing I want to cover is energy efficiency.
At idle, the board used 11 watts. And at full-tilt, 18 watts. With full-size 3.5" hard drives,
it'd probably be a bit more, but what surprised me most was that it's about on par with the
big NAS I currently use, this ASUSTOR in my rack.
The ASUSTOR peaks a bit higher, but on average, it's close to the same energy use, and it
gives me about 20 percent faster file copies.
Anyways, I've had a blast building this thing. Months ago, when I built my own hacked-together
NAS, I said what I really wanted was one board that had everything on it.
And Radxa delivered. As far as Raspberry Pi NASes go, this is the new gold standard. You're
not going to find a more feature-filled yet compact solution, and I love how everything
is integrated. No USB to SATA conversion, no extra power adapters, and getting faster
Ethernet is a nice cherry on top.
Radxa told me they should be shipping the board by itself for less than a hundred bucks
by the end of the year. In Q1 2022, they plan on releasing a two hundred dollar kit with
the case, the board, and a compute module.
Until next time, I'm Jeff Geerling.