Remember this storage backplane, from my Raspberry Pi Hardware RAID video? Broadcom sent it with
a MegaRAID storage card, and it worked great with the SATA and SAS drives I tested. But
this backplane is special. It's not just a stock Serial Cables backplane, it's a customized
box, code-named "Elrond", and it demonstrates a new storage standard, SFF-TA-1005.
Along with the MegaRAID card's 'Tri-mode' support, I can in theory hot-swap SATA, SAS,
and PCI Express NVMe drives all using the same slot.
In my last hardware RAID video, I could only demonstrate SATA and SAS drives, because apparently
some other YouTube channel got first dibs on Kioxia's latest drives, which I was trying
to borrow for my own testing.
I had asked Kioxia if I could borrow a couple of their enterprise NVMe drives, and they were like,
"Yeah, sounds good! So are you gonna be testing this on a beefy new server with PCI Express
Gen 4?"
I was like... "Well, actually, I have a Raspberry Pi. With PCI Express Gen 2!"
And they were like, "Wait, what? That's... not even possible. And who uses high-end enterprise
drives on a $35 single board computer?"
So I told them that it is possible with the CM4 now that the Broadcom driver's sorted
out, and as to who uses high-end enterprise gear on a Pi? Well... I do!
So here they are, the Kioxia PM6 and CM6.
And you heard it here first: these bleeding-edge SAS 24G and NVMe drives work brilliantly on
the Raspberry Pi.
They might be a little overkill, but that's how I like it.
I should note this video is NOT sponsored, and I'm NOT keeping these drives, either.
I just want to see what's possible with a humble Raspberry Pi.
And if you want to skip to the part where I start hot-plugging drives and see if the
Pi explodes, you can jump ahead to this time.
Kioxia, if you didn't already know, is descended from Toshiba Memory Corporation, so they're
not new to the flash memory game.
The drives they sent me use the same internal flash chips, so what's the difference between
these drives? And why are companies building NVMe SSDs in this form factor? I thought all
NVMe drives were M.2?!
Well, server vendors try to build their servers for a broad range of applications. Having
one standard form factor—in this case, the SFF, or "Small Form Factor"—for all mass
storage means they don't have to build and sell dozens of different types of servers
depending on someone's storage needs.
And besides, this drive, tiny as it is, is a LOT bigger in volume than the board space
on a tiny M.2 drive. With that volume comes more room for up to 30 terabytes of storage,
not to mention better thermals. Which is important since these drives can suck down up to 25
watts per drive!
These cooling holes aren't just for good looks!
And there are similar-size U.2 SSDs, but they required different cabling than SATA and SAS
bays.
And while SATA speeds have gotten stagnant, SAS, or Serial Attached SCSI, has continued
to get better every generation. The PM6 is SAS 24G, which, if you do the math, is four
times faster than the fastest SATA drives.
The PM6 is capable of up to 4.3 Gigabytes per second read speeds, and the CM6 can pump
through 6.9 Gigabytes per second on a PCI Express Gen 4 bus.
The major difference between the two is their communication interface.
I won't get too technical here, but the interface used for the NVMe drive follows the "SFF-TA-1001"
spec, which is sometimes called U.3, but that's not the official name (due to a kerfuffle between
different storage vendors). It allows one port to serve three types of drive: SATA, SAS,
and PCI Express NVMe.
And while some people say SAS's days are numbered, rumors of SAS's death in enterprise storage
have been greatly exaggerated.
Some features of SAS like effortless hot-swap, backwards compatibility, a mature management
protocol, and existing implementations still keep it relevant.
But there are advantages to PCI Express-based storage, and the SFF-TA-1001 spec allows NVMe
drives to work with the exact same slots and cabling where you traditionally only had SATA and
SAS drives.
And if you have a faster computer and the latest MegaRAID card, you can get PCI Express
gen 4 speeds for all the drives, giving more than one hundred gigabits per second of total
data throughput.
Enough about the standards, though. Let's throw these drives in Elrond and see how the
Raspberry Pi handles them!
All right, so in my older video I was using some cheap Kingston consumer SATA SSDs, and
some cheap used HP 10K SAS drives.
I have the Elrond enclosure connected to the Pi through this MegaRAID card, and I have
the patched driver running so I can see the card. Using storCLI, I can see two SATA SSDs
and two SAS hard drives in the first four slots.
I'm going to hot-unplug one of the Kingston SSDs with the Pi running.
After a few seconds, storCLI stops showing it in the list of drives. And if I plug it
back in...
It reappears. So SATA hot-plug works fine.
And now I'm going to plug Kioxia's PM6, which is a SAS 24G drive, into the bottom bay, and
we'll see if it comes up.
It takes a few seconds, but it also shows up just fine!
And now is the part that's really fun, since I've tried hot-plugging NVMe M.2 drives on
the Pi before, and it never goes well.
I'm going to plug the CM6, which is an NVMe PCI Express drive, into the 5th slot, and we'll see if
it's automatically recognized too.
And look! It took a few seconds, but now it's showing up too!
Now I'll plug in the other CM6 drives, and again, after a few seconds, they also appear.
So hot-plugging all three types of drives works great!
To see how they perform, I wanted to test individual drives first. I switched the RAID
card into JBOD mode, so the Pi would see "just a bunch of disks".
Then I used "lsblk" to identify all the drives by serial number and size.
Once I identified which drives were the PM6 and CM6, I partitioned, formatted, and mounted
them.
After the initialization finished, I ran my disk benchmark script on both drives, and
got these results.
Interestingly, the CM6 NVMe drive performed better than the PM6 SAS drive for random writes,
but was neck-in-neck for read speeds.
And if you're one of Kioxia's marketing representatives watching this part of the video, you might
be screaming at the screen that these numbers are wildly incorrect.
And you'd be right! These numbers are being measured on a Raspberry Pi. This tiny little
thing has a 5-year old mobile CPU and one lane of PCI Gen 2 bandwidth. It's pretty much
maxed out, at 400 Mebibytes per second.
If you can afford a real server like ServeTheHome can, you can get speeds up to 6.9 Gibibytes per
second! That's 17x faster than the Pi can ever dream of getting.
So with that in mind, I wanted to see if the Pi could get a little bit faster if I put
the CM6 NVMe drives into a RAID 0 array, and it did. Not a huge difference, but now large
block size random IO was about as fast as it could possibly be on the Pi, and 4K random
reads were a little bit faster.
To give a little more data, I also compared the speeds of the single CM6 against a single
Kingston SSD and a single old HP SAS Hard Drive.
To the surprise of nobody, the super old used hard drive is absolutely decimated in random
performance, even by a cheap consumer SSD.
But throughput isn't the only consideration when it comes to storage. Enterprise drives
usually have better warranties, better ability to handle massive read or write-optimized
workloads, and price tags to match.
Speaking of price, these are some of the fastest enterprise SSDs on the market, and judging
by the prices people are paying on eBay, they ain't cheap!
And with that price tag, do I think anyone else will be plugging Kioxia CM6 drives into
their Raspberry Pi any time soon? Maybe, maybe not. But as someone who lives on cheaper second-hand
hardware, the fact that it can be done today means that when these drives are recycled
in the next generation of servers, it's good to know that this and future Pi generations
should be able to work with them.
Of course, with Chia miners eating up all the hard drives on the market today, maybe
it'll be cheaper to buy enterprise drives like these than SATA hard drives! Who knows?
Subscribe so you don't miss out on the next insane test, and until next time, I'm Jeff
Geerling.