This is 1.2 petabytes of hard drives... all connected to a single Raspberry Pi.
In my last video on the PetaPi—that's what I'm calling it now—I got the whole thing
working... but just barely. I spent the past few weeks making it better,
and in this video, I'm gonna answer all your burning questions, like
"what does it sound like when 60 hard drives spin down at the same time?"
Or how much power does this thing suck down?
That's, uh... yeah, I'll get back to that and a bunch of other questions later in this video,
but first, I mentioned in the last video the HBAs—these cards that connect the Pi
to all 60 drives—were running really old firmware.
With that firmware, when I tried running the drives in RAID 0,
one of the cards would reset any time I wrote more than a few gigabytes of data. Worse,
when the card came back up, one or two drives would just be missing.
I had to reboot the Pi every time I tested the Petabyte, and that got old, fast.
The Broadcom engineers told me I should try upgrading the firmware. But
that led me down quite a rabbit hole!
First I tried it on the Raspberry Pi, but it kept giving me signature validation errors.
So then I was like "I have this Windows PC. I can pop the cards in it and then upgrade them there!"
Oh, but it's never that easy.
Quick sidebar, you know how sometimes on LTT,
the writer for the video appears on set with Linus to explain something?
Let me tell you. There are little parts in these videos that might be 5 or 10 seconds long, or
they might not even make it to the final cut,
but it took days or weeks to figure it out. Well, this was one of those times.
I grabbed my HP desktop and popped one of the HBAs into it. In Windows,
I downloaded the latest version of StorCLI, Broadcom's utility for managing their HBAs.
Since the card was running on super-old version 5 firmware,
I tried first upgrading to version 12. And that worked! Version 14 worked too!
But then I tried upgrading to versions 15, 16, 17, and 18, but none of those worked—I
got that same "firmware flash image is invalid" warning that I got on the Pi!
Well that's weird.
So the next thing I tried was downloading an old version of StorCLI, from April 2020.
The Broadcom engineers mentioned that they often updated StorCLI alongside new firmware versions...
and maybe using an older version would help.
And what do you know, it did! I was able to upgrade to version 16,
then 18, rebooting between each update. Then I tried using the latest version of StorCLI
again for the version 20 upgrade, but I got that same "image is invalid" warning again!
So dropping back to the old StorCLI version,
I was able to complete the cycle and upgrade all the way to the latest version, 23!
I pulled the second card out, plugged it in, and decided to just try the
really old version of StorCLI to update to version 23 right away, and that worked!
So I popped the third and fourth cards in,
flashed them, and plugged all the cards back into the Pi. Heck, I might've just
been able to do all that with the Pi if I'd known to use an older version of StorCLI.
But after I booted the PetaPi up, I created a Btrfs RAID0 array again,
and this time, it actually ran a little better.
Most of the performance numbers were slightly better on firmware 23, and what's better,
I still got random cards that would reset under load, but they would recover all the hard drives,
so the array could still work without having to reboot every few gigabytes.
But that's not good enough for me. You all gave me a lot of suggestions.
One of them in particular would help me rule out power issues, at least,
and that was forcing the Raspberry Pi back to PCI Express gen 1 speeds.
Using this script, I forced the Pi's bus to run at half speed, which is still plenty for
my gigabit network connection, since half speed is 2500 megatransfers per second.
After I confirmed all the cards were running at that link speed, I ran my benchmarks again, and
what do you know, they were all slower! But that's just when running locally, not over the network.
Before I tried a sixty-drive network copy, though, I wanted
to see what the breaking point was at PCI Express gen 2 speeds.
So I tested 15 drives, then 30, then 45. And I found out that RAID works
great with one or two HBAs, but when I tried three HBAs with 45 Drives,
that's when one of the cards would start to reset under load.
I also swapped around cards during that testing,
to make sure it wasn't just a one bad card... and it wasn't.
So now I can confidently say that you can do up to
30 direct-attached drives on a Raspberry Pi at full PCI Express Gen 2 speed...
but after that, it gets a little dicey. At least with the adapter I used.
The USB 3 cables these things use can be flakey, and they're not even rated to route
PCI Express traffic... but heck, a Pi isn't really rated for sixty hard drives, either.
Anyways, the next question was, can I get all 60 drives working stable at PCIe gen one speeds.
And the answer is yes!
I copied about 70 terabytes of data over the network, and got about 70 Megabytes
per second write, and 100 megabytes per second read. No cards reset at that speed.
Just to be complete, I also ran all the tests with a 2.2 gigahertz overclock on the Pi,
since the giant fans in the Storinator kept it down to like 35 degrees celcius,
and that did make things a lot faster. Network copies were up to 80 to 100
megabytes per second write, and maxed out the Gigabit connection for reads.
So heck... if you just wanted a massive, low-power storage server, and didn't need to write more than
70 or 80 megabytes per second to it... then this might not be the most insane idea ever.
Speaking of power, that was one of the questions people asked the most: how much power does it use?
Well, full-out, when I was running a benchmark in RAID 0?
About 512 watts. At idle, with 60 drives, 502 watts.
But during boot, while it's spinning up 15 drives at a time,
the maximum power draw I measured with my Kill-A-Watt was 632 watts.
And that's not bad, all things considered! But
that is about 350 killowatt-hours of electricity per month, and here in the midwest,
running this 24x7 would cost about $45 a month in the summer, and $30 in the winter.
If I just run 15 drives, the PetaPi only uses 200 watts. So if I set up the array in multiple vdevs,
and let some of them spin down, I could definitely save some money!
There were a bunch of other questions, too.
Like how much does it cost? Well, assuming you can get a Pi, this whole system,
including all the drives and HBA cards, would run you around fifty thousand dollars.
And someone asked me why I didn't do a petabyte of flash storage, like Linus did recently.
Well... that would cost at least half a million dollars. So yeah... not gonna happen.
How much does it weigh?
About 300 pounds. In fact, before I move it, I have to take out all the hard drives.
Could I use hardware RAID cards instead of HBAs and have them set up four separate RAID arrays,
then have the Pi join them together? Yeah, maybe,
but that would still probably run into the weird bus problems with my PCI Express switch.
Could I use one HBA and a SAS Expander, which would basically multiply how many
SATA drives I could use on one card? Yes, in fact the Broadcom engineers even suggested I
try out this card if I wanted to do that. But I didn't. Maybe something to test in the future.
Could I run multiple Raspberry Pis, and split up the hard drives so each Pi just has fifteen?
Yeah, I could definitely do that. In fact, I'm working on an Ceph cluster
video using this new Pi cluster board soon, so subscribe if you wanna see it!
But clustered storage is a whole different ball of wax. And getting all that running
in one Storinator chassis wouldn't be as fun.
Could I upgrade the networking on the Pi to 2.5 or 10 gigs?
Yes, actually... but I did a lot of testing with that in the past. The problem
is because the Pi just has that one PCI Express interface, with most NICs,
interrupts cause the real-world throughput to stay below 100 megabytes per second with Samba or NFS.
So you could get a ten gig link, but file copies are still gonna be about the same.
Almost every test I've done, I hit the edge of the Pi's performance envelope. But hey,
I do that so you don't have to! Not everyone needs a 10 gig network, or even a petabyte!
What happens in Linux when you have more than 26 drives?
It just starts over and adds another letter, so you get like "sdaa" and "sdab".
The final question is what name did I pick. And obviously, if you're this far, I stuck
with PetaPi. It just feels nice and simple, and it's a good shortening of 'Petabyte Pi'.
Some other names people suggested were "Pi-bibyte", "Pi-tabyte", "P3", "the PP"
or "the big PP", and "Petafile"... eh-uh... yeah, definitely not gonna do that last one.
All right, so if you have any other questions, comment below. For the next few weeks,
I'm gonna replace the guts of the server, pull all the drives, move it into my rack,
and get it set up for the long term. And my next video on it'll have a special guest appearance,
so make sure you're subscribed for that!
This project surprised me. I knew there was a good chance I could get a Petabyte
of storage on a single Pi. And I did! And I even found a way to make it stable in RAID!
But would I recommend it? Well, if you don't need more than 70 megabytes per second, go for it. But
most people who want a giant storage server would probably want to stick with a better server CPU.
But heck, who knows what the future holds? New ARM SBCs with the RockChip RK3588 can have up to 7
lanes of PCI Express. That's a lotta bandwidth. If the next Pi generation can do anywhere near that,
and still run stable? Running a PetaPi in production might not be such a wild idea.
Until next time, I'm Jeff Geerling.