This Mini ITX motherboard has SIX computers on it,
and if I flip it over? There are six SSDs on the back. What time is it? It's CLUSTERIN' Time!.
This is the Super6c, a Raspberry Pi clusterboard. Like the Turing Pi 2 supercomputer I built last
year, it fits in normal PC cases, but unlike the Turing Pi 2, it holds two more Pis,
it fits inside a 1U enclosure, and you can buy it right now, for $200.
A lotta people test out things like Kubernetes or K3s on a little cluster like this.
But today I'm gonna explore something a little different. I'm gonna test this thing by turning it
into a Ceph storage cluster so I can pool together ALL these NVMe drives. And we'll
see if it can even play in the same league as a twelve thousand dollar Ceph storage appliance.
Spoiler: It can't. But we'll get to that.
I bought my Super6c from DWMZone also sells it on their website.
It comes with a 100 Watt power adapter, some standoffs so you can run the board
without a case, and a set of screws to mount up to six M.2 SSDs on the bottom.
There are spaces for up to six Raspberry Pi Compute Module 4's—that's this little guy,
which is an entire computer that's smaller than a playing card—and
the Super6c has a built in gigabit switch.
Now, that doesn't take all the Pis and put 'em together and make one giant Pi. You still
have to manage the Pis yourself, it's just this board makes it a lot simpler, since you wouldn't
need to buy six Pi 4s, six power adapters, six Ethernet cables, and a separate network switch.
That's how I built my Raspberry Pi Dramble cluster back in the day,
but after using this board, I don't think I'll ever go back.
This truly is a cluster-in-a-box... And it's super thin.
I'm gonna install it in this tiny PC case, and I also designed an IO shield. Actually,
I had to design four versions before I could get one that fit,
because the tolerances for some of the ports are a little off.
While we're looking at the ports, I guess I'll talk about IO.
There's the power connector that accepts 19-24 volts, and I'll talk more about power consumption
later. There are two Gigabit Ethernet ports connected directly to this Realtek Ethernet switch
chip, which connects through to each Pi at a full Gigabit. Then there are two HDMI ports,
a micro USB port, and two USB 2.0 ports, all connected straight through to CM4 #1, meaning
you can manage this entire cluster standalone, if you plug in a keyboard, mouse, and monitor.
Finally, on the back there are these six green LEDs, and they show you the
activity for each Pi in the cluster. But if you like LEDs, that's not all you get;
each Pi gets its own set of four LEDs: two for power and activity, and two for Ethernet.
Each Pi also gets its own little microSD card slot on the back,
which is useful if you have Lite Compute Modules.
And each Pi gets its own micro USB port on the topside, for flashing modules with eMMC storage.
The board has it's own little PMU, or power management unit,
and it comes with a power and reset button over here. You can also plug in front panel
connections, but I noticed the power LED, and any fans you have plugged in,
are always on. That means it can be hard to tell whether the cluster's running if it's in a case.
But you can turn on the cluster by pressing the power button, and you can force shut it down by
holding down the power button for five seconds. I asked DeskPi if their board's firmware is open,
or if they have any remote access features planned, but they said no.
Also, the onboard Ethernet switch is not managed, so you can't do things like set
up VLANs or do any other advanced routing. And no, just plugging in two connections
to a gigabit switch won't double the bandwidth to this board. I tested that.
Both of those features are coming on the Turing Pi 2, so with all these kinds of boards,
there are always tradeoffs. I mean, looking at the board space, I can understand—trying to cram
every feature possible for six computers on a Mini ITX motherboard is pretty much impossible!
So I plugged all the Pis in on the top, but I couldn't use the heatsinks I normally use,
so for now, I popped them off and mounted a big fan to the case. I actually started with a Noctua,
but even with a quiet adapter it was a little loud. Since this board doesn't have PWM, I swapped
in this Arctic F12 Silent. It was almost silent, and everything stayed cool even without heatsinks.
On the bottom, I installed six of these. These are Kioxia XG6 NVMe SSDs, and they're overkill for a
Raspberry Pi. But since I had a couple on hand already, KIOXIA reached out and sent four more!
So if you see the 'includes paid sponsorship' thing on this video,
that's why. They didn't pay me, but they did send me four SSDs to fill up the rest of the slots.
After I put the drives in, I flashed 64-bit Pi OS to six microSD cards, and put one in
each slot. When I flashed the OS, I made sure to add hostnames, like deskpi1, deskpi2, and so on.
That way, when it comes time to connect to them, I don't have to figure out their IP addresses.
I think if I had one major complaint about the hardware,
it's how hard it is to access the microSD card slots. They're all at different angles,
and since they're flat against the board, I can't even fit my fingers
in to remove a card. A couple times I even had to use my spudger to get a card out.
But now with everything put together, it's time to mount this thing in my case!
I found this cheap Mini ITX case on Amazon and I'll put a link to it below. I slipped
in my 3D printed IO shield, put in the motherboard, and screwed it in.
I plugged in the front panel connectors, but then when I went to plug in the front panel USB
2.0 plug, I noticed the header on the motherboard had all the pins on it. It's supposed to be
a keyed connector, with one pin missing. So I couldn't plug in the USB connector.
I asked DeskPi about this and they said that was a problem with the first production run,
but it's been fixed now. So on to the first boot!
When I plugged it in, the front panel power LED lit up,
and the fan started spinning right away. It looks like those are just, always turned on.
When I pressed the power button, all the Pis lit up almost at the same time,
and it was fun and kinda mesmerizing watching all the blinking activity LEDs
on the back. It reminded me a little of WOPR from the movie War Games.
I also measured power consumption and found that the board uses less than a watt powered down,
with just the fan running, about 17 watts with six Pis running, 24 watts max,
and 11 watts if you shut down the Pis but didn't power off the cluster using the power button.
One thing I don't like about the design is how the Ethernet ports on the back don't have
status lights. I had to check the other end of the cable on my switch to make sure it was connected.
But everything seemed to work great, except for one of the CM4s had a bootloader issue,
which meant its activity LED would just light up green, and it wouldn't boot.
After a quick trip over to my PiTray to re-flash the Pi's firmware, I got it booting too.
A few quick notes about board power: it doesn't look like there's a way to just shut off power
to one Pi at a time, so hot-swapping Raspberry Pis doesn't seem like a very safe thing to do on this
board. On the Turing Pi 2, it has more advanced firmware so you can control power to each slot.
That's not a huge issue, but it is something to be aware of.
And you know how I mentioned this board doesn't just take six Raspberry Pis and
slap 'em together into one big Raspberry Pi? Well, the natural question then is,
'how am I going to manage six Raspberry Pis?' Well, you could log into each one with SSH,
but in my case, I'm gonna use a tool that's perfect for managing a Pi cluster: Ansible!
I have a whole book and even a YouTube series on Ansible, and I'll link them below,
but to get started, all I need to do is create an inventory file to tell Ansible where to find
the servers—that's this file here—and then a playbook to tell Ansible what to do with 'em.
I won't get too deep into it here, but I've posted the entire playbook
and a more detailed guide on my GitHub, so check the link below.
The first thing I wanted to do was make sure all the Pis were up to date running
the latest version of Raspberry Pi OS. So I wrote this playbook,
and ran it on the Pis. It updates them all and automatically reboots them if they need to.
Then I wanted to try out some software I've never used before,
Ceph. Ceph is a distributed storage system. You might know RAID, where you can plug multiple hard
drives into one computer and mash them together for redundancy or performance.
Well Ceph is like that, but instead of multiple drives on one computer,
you can put multiple drives from multiple computers together in a storage array,
and let Ceph deal with redundancy and networking.
I followed the instructions from this blog post,
and used a tool called cephadm to set everything up.
First I just tried installing cephadm,
but apt said it couldn't find it. I actually had to turn on the unstable Debian repository
since cephadm is so new it's not in the repos that ship with the Raspberry Pi yet.
But I did that, updated apt's cache, and installed cephadm without any problems.
Then I ran this bootstrap command, and to get the cluster set up, I had to know the Pi's IP address,
so I hopped over to another terminal and grabbed that.
It took about five minutes, but at the end,
it spits out some information for accessing the Ceph Dashboard in a browser.
Before doing that, I had to copy the Ceph public key from the main Pi to all the other Pis.
But I didn't have that Pi set up to log into all the other Pis yet,
so I decided to just do all that in Ansible, since it would make the process easier.
So I wrote this task that saves the public ceph key from the main server,
and then this task that tells Ansible to add it to the authorized keys for the root user
on all the other Pis. This way Ceph will be able to manage all the other Pis when I connect 'em.
And it was a good thing I was using Ansible, because I realized after I logged into the
Dashboard that you had to have some other stuff installed too, before Ceph can work on those Pis!
It was saying it needed a container engine, and also something called
lvcreate, which is part of the LVM2 package. So over in Ansible,
I added this task that makes sure podman and lvm2 are both installed on all the Pis.
All right. With that done, I could add all the hosts. It took a few minutes
for Ceph to get them all healthy, but after maybe five or so minutes,
they all showed up, and Ceph could tell me how much raw storage was on each Pi.
Looking at the main dashboard, the cluster status was OK,
and it showed the total raw capacity available to Ceph is 3.3 Tibibytes. Not bad!
I wanted to get NFS working, so I test everything from my Mac on the network,
so I tried that, but I kept getting this error. I tried installing some
missing packages, but even with that, it wouldn't create the NFS service.
So instead I set up a storage pool and
ran benchmarks using Ceph's built-in benchmarking tool.
Write speeds were around 70 MB/sec, and read speeds were around 110 MB/sec.
That's... better than I thought I'd get! In fact, that's about how much speed I could get through a
gigabit network, so I'm not disappointed. Encryption or other fancy features will
slow it down, but getting 110 Megabytes per second on a Pi is pretty decent.
And you might be tempted to just plug two Ethernet cables into the
board, thinking that'll double the network bandwidth, but it just doesn't work that way.
Since the switch on the board isn't managed, you can't configure anything like
link aggregation, so at best, it'll just provide a redundant network path.
I actually tested this out, and the total speed when running iperf between my Mac and the board,
no matter how many Pis I connected to, always maxed out at 1 gig.
And to prove it's not just a weird networking issue on my Mac, I even ran
a benchmark across two switches to two computers simultaneously, and total throughput was always
under a gigabit. In fact, sometimes the network performance was a little worse,
and I wonder if the switch chip was overheating a little. It only got up
to about 50°C in my thermal testing, but maybe it doesn't like that much switching. Not sure.
So after all that, could this thing replace a twelve-thousand-dollar enterprise storage
appliance like the Mars 400? In a word: no. And there are two main reasons for that:
networking, and the IO limitations of the Raspberry Pi.
The Mars 400 has less memory, and fewer CPU cores. But it has faster internal Ethernet connections,
plus four 10 gigabit uplinks, so this thing could pump through gigabytes per second. The Pi can't
really touch that, especially since the Super6c can only put through 1 gigabit over the network.
So this board is an interesting one. The Turing Pi 2 is a very different product.
It has managed Ethernet, so you could assign VLANs or set up link aggregation.
This board has a dumb switch, so having two ports on the back isn't actually all that useful.
The Turing Pi 2 can work with other form factors, like the Jetson Nano. The Super6c only works with
CM4-compatible boards. And the Turing Pi 2 also has other neat tricks, like Mini PCIe slots,
which I used to build a remote 4G Kubernetes cluster that I ran out on my cousins' farm.
So both of these boards have their virtues, but I think the biggest thing going for this
board is how thin it is, and how it crams six CM4s on one board. It can
be very useful for experimentation, or maybe even some types of edge computing. Oh, also,
you can buy it right now and you don't have to wait for a Kickstarter.
Assuming Raspberry Pis ever become available again, I think this board would be ideal for
learning Ceph, Kubernetes, and other clustering tools. It's certainly a lot less hassle than
putting together separate Raspberry Pis, power supplies, Ethernet cables, and a switch.
If you wanna to pick one up, check out the link in the description. I've also linked to
all the other parts I used in my build, and all the guides and code used in this video.
Until next time, I'm Jeff Geerling.