After I posted my Raspberry Pi Blade server video last week, lots of people asked what
you'd do with a Pi cluster. Many asked out of curiosity, while others seemed to shudder
at the very idea of a Pi cluster, because obviously a cheap PC would perform better...
right?
Before we go any further, I'd say probably 90 percent of you watching shouldn't build
a Pi cluster.
But some of you should. Why?
Well, the first thing I have to clear up is what a Pi cluster isn't.
Some people think when you put together two computers in a cluster, let's say both of
them having 4 CPU cores and 8 gigs of RAM, you end up with the ability to use 8 CPU cores
and 16 gigs of RAM.
Well, that's not really the case. You still have two separate 4-core CPUs, and two separately-addressable
8 gig portions of RAM.
Storage can sometimes be aggregated in a cluster, to a degree, but even there, you suffer a
performance penalty and the complexity is much higher over just having one server with
a lot more hard drives.
So that's not what a cluster is.
Instead, a cluster is a group of similar computers—or even in some cases, wildly different computers—that
can be coordinated through some sort of cluster management to perform certain tasks.
The key here is that tasks must be split up to work on members of the cluster. Some software
will work well in parallel. But there's other software, like games, that can only address
one GPU and one CPU at a time. Throwing Flight Simulator at a giant cluster of computers
isn't gonna make it any faster. Software like that simply won't run on any Pi cluster, no
matter how big.
Luckily, there is a LOT of software that does run well in smaller chunks, in parallel.
For example, right now in my little home cluster—which I'm still building out—I'm running Prometheus
and Grafana on this Raspberry Pi, and monitoring my Internet connection, indoor air quality,
and household power consumption. This Pi's also running Pi-hole for custom DNS and to
prevent ad tracking on my home network.
This next Pi is running Docker and serving up the website Pi Dramble.com.
And the one after that is managing backups for my entire digital life, backing all my
data offsite on Amazon Glacier.
I also have another set of Pis that typically runs Kubernetes, but I'm rebuilding that cluster
right now.
But there's tons of other software that runs great on Pis. Pretty much any application
that can be compiled for ARM processors will run on the Pi. And that includes most things
you'd run on servers these days, thanks to Apple adopting ARM with the new M1 Macs, and
Amazon using Graviton instances in their cloud.
I'm considering hosting NextCloud and Bitwarden soon, to help reduce my dependence on cloud
services and for better password management.
A lotta people run things like Home Assistant on Pis, and there are thousands of different
Pi-based automation solutions for home and industry.
But before we get to specifically why some people build Pi clusters, let's first talk
about clusters in general.
Why would anyone wanna build a cluster of any type of computer? I already mentioned
you don't just get to lump together all the resources. A cluster with ten AMD CPUs and
ten RTX 3080s can't magically play Crysis at 8K at 500 fps.
Well, there are actually a number of reasons, but the two I'm usually concerned with are
uptime and scalability.
For software other than games, you can usually design it so it scales up and down by splitting
up tasks into one or more application instances. Take a web server for instance. If you have
one web server, you can scale it up until you can't fit more RAM in the computer or
a faster CPU.
But if you can run multiple instances, you could have one, ten, or a hundred 'workers'
running that handle requests, and each worker could take as much or as little resources
as it needs. So you could, in fact, get the performance of ten AMD CPUs split up across
ten computers, but in aggregate.
Not everything scales that easily, but even so, another common reason for clustering is
uptime, or reliability. Computers die. There are two types of people in the world: people
who have had a computer die on them, and people who will have a computer die on them.
And not just complete failure, computers sometimes do weird things, like the disk access gets
slow. Or it starts erroring out a couple times a day. Or the network goes from a gigabit
to 100 megabits for seemingly no reason.
If you have just one computer, you're putting all your eggs in one basket. In the clustering
world, we call these servers "snowflakes". They're precious to you, unique and irreplacable.
You might even name them. But the problem is, all computers need to be replaced someday.
And life is a lot less stressful if you can lose one, two, or even ten servers, while
your applications still run happy as can be—because you're running them on a cluster.
Now I mentioned that I'm running one instance of each of these applications on this cluster
right now. I'm planning on splitting up a few of them, though, and probably using Kubernetes
on the entire stack so I can have even better redundancy. But having multiple Pis—and
having good backups and automation to manage them—means when a microSD card fails, or
a Pi blows up, I toss it out can have a spare running in minutes.
Okay, so those aren't all the reasons for clustering, but two of the main reasons most
people would consider a cluster over one computer.
But that doesn't answer the question why someone would run Raspberry Pis in their cluster.
A lot of people questioned whether a 64-core ARM cluster built with Raspberry Pis could
compete with a single 64-core AMD CPU. And, well, that's not a simple question.
First I have to ask: what are you comparing?
If we're talking about price, are we talking about 64-core AMD CPUs that alone cost six
thousand dollars? Because that's certainly more expensive than buying 16 Raspberry Pis
with all the associated hardware for around three thousand, all-in.
If we're talking about power efficiency, that's even more tricky.
Are we talking about idle power consumption? Assuming the worst case, with PoE+ power to
each Pi, 16 Pis would total about 100 Watts of power consumption, all-in.
According to [Serve The Home's testing](https://www.servethehome.com/amd-epyc-7742-benchmarks-and-review-simply-peerless/), the AMD EPYC 7742 uses a minimum of 120 Watts—and
that's just the CPU.
Now, if you're talking about running something like crypto mining, 3D rendering, or some
other task that's going to try to use as much CPU and GPU power as possible, constantly,
that's an entirely different game.
The Pi's performance per watt is okay, but it's no match for a 64-core AMD EPYC running
full blast.
Total energy consumption would be higher, 400-plus Watts compared to 200 Watts for the
entire Pi cluster full-tilt, but you'll get a lot more work out of that EPYC chip on a
per-Watt basis, meaning you could compute more things, faster.
But there are a LOT of applications in the world that don't need full-throttle 24-7.
And for those applications, unless you need frequent bursty performance, it could be more
cost-effective to run on lower power CPUs like the ones in the Pi.
But a lot of people get hung up on performance. It's not the be-all and end-all of computing.
I've built at least five versions of my Pi cluster. I've learned a LOT. I've learned
about Linux networking. I've learned about Power over Ethernet. I've learned about the
physical layer of the network. I've learned how to compile software. I've learned how
to use Ansible for bare metal configuration and network management.
These are things that I may have learned to some degree from other activities, or by building
virtual machines one one bigger computer. But I wouldn't know them intimately. And I
wouldn't have had as much fun, since building physical clusters is so hands-on.
So for many people, myself included, I do it mostly for the educational value.
Even still, some people say it's more economical to build a cluster of old laptops or PCs you
may have laying around. Well, I don't have any laying around, and even if I did, unless
you have pretty new PCs, the performance per watt from a Pi 4 is actually pretty competitive
with a 5 to 10 year old PC, and they take up a LOT less space too.
And besides, the Pis typically run silent, or nearly so, and don't act like a space heater
all day, like a pile of older Intel laptops.
But there's one other class of users that might surprise you: enterprise!
Some people need ARM servers to integrate into their Continuous Integration or testing
systems, so they can build and test software on ARM processors. And it's a lot cheaper
to do it on a Pi than on a Mac mini or an expensive Ampere computer, if you don't need
the raw performance.
And some enterprises need an on-premise ARM cluster to run things like they would on AWS
Graviton, or to test things out for industrial automation, where there are tons of Pis and
other ARM processors in use.
Finally, some companies integrate Pis into larger clusters, as small low-power ARM nodes
to run software that doesn't need bleeding-edge performance.
Another sentiment I see a lot is that it's "too bad the Pi doesn't have ECC RAM."
Well, be ready to be shocked, the Pi technically does have ECC RAM! Check the product brief!
The Micron LPDDR4 RAM the Pi uses technically has on-die ECC.
Now, when people say ECC they mean a lotta different things. And I'd say half the people
who complain about a lack of ECC couldn't explain specifically how it would help their
application run better.
But it is good in a server setting for many types of software. And the Pi has it... or
does it? Well, not in the sense that expensive high-end servers do. The on-die ECC can prevent
memory access errors in the RAM itself, but it doesn't seem to be integrated with the
Pi's System on a Chip, so the error correction is minimal compared to what you'd get if you
spent tons of money on a beefy server with ECC integated through the whole system.
So anyways, those are my thoughts on what you could do with a cluster of Raspberry Pis.
What are some other things you've seen people do with them? And have you built your own
cluster of computers before, Raspberry Pi or anything else? I'd love to see your examples
in the comments.
Until next time, I'm Jeff Geerling.