A homelab is a special thing for nerds like me.
From a little router and a Raspberry Pi in a rat's nest of cables to a giant rack full
of servers with screaming fans and petabytes of storage, everyone's homelab has a story.
This is my story, at least so far, and I told it at AnsibleFest 2022 in Chicago.
After you watch, I'd love to hear about YOUR homelab in the comments below.
And if you wanna to learn Ansible, check out my book, Ansible for DevOps.
For a short time, the eBook version is free if you use the link down below.
This is my home office.
It's also my little YouTube studio.
And it's in my basement, so it's nice to finally be back at AnsibleFest after three years.
In fact, AnsibleFest 2019 was the last in-person conference I went to, and if you look behind
me, that lanyard is still the last one on my little lanyard hook!
So it'll be nice to finally have a new one.
And there are some big companies at AnsibleFest, like Microsoft, Amazon, and Google like you
see here.
They all have giant data centers and employ thousands of sysadmins.
But me?
I represent Midwestern Mac, LLC, a tiny little company headquartered in St. Louis.
And I don't have a data center.
I have a rack.
Specifically, I have this rack.
This is in my basement, and yes, it's sitting right next to my HVAC system, with a condensate
pump and humidifier right next to it.
That means, water lines.
And on the other side is the basement floor drain, which has backed up a few times.
That's why I bought casters for my rack.
If you want more where that comes from, check out my rack setup video on the Geerling Engineering
YouTube channel.
My Dad and I have some fun working on projects together.
Maybe a little too much sometimes.
But I put my rack on casters because if I get a sewage backup, I'd rather sacrifice
the casters than thousands of dollars of rack equipment.
And wouldn't you know?
They already came in handy once.
Earlier this year, my AC condensate pump failed, and the water alarm I set up also failed,
and I looked on in horror the day I came downstairs to see this.
Luckily, the water was all just going under the rack, and wasn't coming out of the rack.
None of my servers wet themselves.
So anyway, my homelab today has three main purposes:
First, it supports the video production needs of this YouTube channel.
Second, it ensures my kids have five-nines of uptime for Doritos—that's what they call
the media app they use to watch all the episodes of Berenstain Bears and all the other kids
shows and movies I ripped onto our NAS.
And third, it lets me have fun cosplaying as a sysadmin.
Hey, I even have a shirt for that!
Check it out along with other favorites like my 'It was DNS' shirt over on Red Shirt Jeff.com.
But let's get to the meat of this presentation.
The homelab.
Every homelab starts with humble beginnings.
You move into a new place.
What's the first thing you do?
Plug in a router.
Get Internet.
Water?
Food, a bed?
That stuff can wait.
You need to get Internet going ASAP.
Most of us start things out with a router.
Sometimes it's built into the modem your ISP sends you.
Sometimes you have a separate modem and plug that into a router, like I do here with my
ASUS WiFi router.
But you wanna do more.
You wanna store stuff on the network and maybe run some apps like Pi-Hole to block ads, or
a home VPN.
So you add a NAS!
But then you realize running a VPN, Plex, Pi-hole, backup software, and home automation
on a single consumer NAS is a bit fragile.
And now you're looking at Craigslist for used server hardware, UPSes, and racks, and your
significant other starts wondering why the electric bill shot up so quickly last month.
So that brings me to my homelab.
The first version you see here was just a piece of plywood with a modem and a network
switch.
I plugged my WiFi router in upstairs, so you don't see it in the picture here.
But at least I had a little UPS.
And an empty punchblock, not sure why.
I never even had an phone line at my house...
I guess I just thought it looked cool.
I moved everything to another part of my basement when we remodeled, and added on a 3G cell
and a Raspberry Pi to run some local services and the website pidramble.com.
But then... things started getting serious when I bought my first rack.
I didn't think I'd ever fill it up.
But I did.
I had the space, so I grabbed a free UPS from a local business that was moving, but that's
when I found out the rails for it only worked in server racks, not a smaller network rack
like the one I bought.
So I modded it.
The grinder, circular saw, and a tap and die set were involved, but dangit, I got that
UPS installed.
And then I filled up the rest of the thing.
I upgraded to a 2.5 gig switch, added some more Raspberry Pis, and started actually caring
about cable management.
I added a second NAS for two local copies of all my data, and I was quickly running
out of rack space, to the point I didn't have room for that new server laying on the ground.
So I invited my Dad over, and he helped me upgrade to this bigger rack, the one I'm using
today.
And before I talk about automation, I'll walk you through the entire rack.
Starting from the top, I have a patch panel with cable drops from around the house, and
a couple from other parts of the rack.
Then I have a shelf for random things I can't rackmount, like my Philips Hue hub, my cable
modem, and my Home Assistant Yellow.
I put 'em near the top since they have radios in 'em, and the open rack with a wood top
lets them connect to IoT devices without any issues.
Then there's my 10 gig switch, from MikroTik.
The core of my network goes through there, so all my storage and my wired computers can
transfer data around 1 gigabyte per second.
In the middle, I have a PoE+ switch from QNAP that also has a couple 10 gigabit ports, and
I use it to power some Raspberry Pis and my IP cameras.
Under that is my Pi rack, where I run most of my networking applications.
I'll get to those in a bit.
Then there's a drawer, and I won't show you the inside, just like I won't open up the
neat patch and show you the rat's nest of cables in there, either.
Under that is my edit NAS, running TrueNAS with 40 terabytes of raw SSD storage.
And if you think that's overkill...
well, just wait.
Below there are my two storage NASes, one with six bays and the other with four.
The one on the right is just a replica of the one on the left, so I have at least two
complete copies of all my important data in my house.
Between those two NASes I have 200 terabytes of raw capacity.
And then there's the Storinator.
This thing is from 45Drives, and I actually built what I called the PetaPi, a single Raspberry
Pi addressing 60 hard drives through four Broadcom HBAs.
I did a whole video on that called the Petabyte Pi Project, so I won't get into it here.
But if you look closely, you can see the fans aren't spinning.
I'm actually not running that server yet, since I'm gonna revamp my local storage later
this year.
So it's not wrapped up in my homelab automation—yet.
But I'll be dealing with that in a video later this year, so make sure you're subscribed!
Under that, at the bottom, is the UPS I got free from a local business that closed.
I bought a used network card for it off eBay, so I can hook it up to my network.
I'm integrating it into Home Assistant, and using Network UPS Tools to integrate it into
my other servers' power management.
I don't have that all working quite how I like it yet, so if you subscribe, you'll probably
hear more about that setup next year.
In my office-slash-studio, where I'm recording this, I have a second rack, where I have another
10 gig switch, my main workstation, which is a rack-mounted Mac studio, and my audio
processing hardware.
And just to be complete, here's the back of the main rack, where I have a PDU mounted
to distribute power to all my devices.
And here's the back of my office rack.
Not gonna win any awards for cable management, but it's functional.
So as far as a datacenter goes, this is nothing.
But for a homelab...
there's a decent amount of gear.
And trying to manage every device by hand in my spare time would be impossible.
I mean, you could just throw equipment in the rack and let it sit until it's hacked
or dies, but if you want to have a secure, useful homelab, you have to do stuff:
[click]
You have to configure everything.
You have to have backups, and test the backups.
You have to have a patching workflow.
You have to be able to replace and upgrade hardware.
And I choose to do as much as possible with Ansible.
One of the most important things, and something that you can set up in your own Homelab pretty
quickly, is to have some sort of maintenance and upgrade playbook.
I have a Homelab project that has this Ansible inventory file.
It has all the connection details for all the Linux, FreeBSD, Raspberry Pi, and IoT
devices on my network.
With that inventory, I can run things like this upgrade playbook to quickly upgrade and
reboot all the servers for maintenance, but also if there's ever an urgent patch like
for Heartbleed.
It's still a work in progress, and I'm also interested in getting my two Windows machines
managed, but I'll be publishing it on GitHub soon.
The homelab project also has a playbook that sets up the displays and hardware on all the
Pis in my Pi rack.
Since I maintain a lot of open source code for each of the main parts of my homelab,
I have individual device configurations split out into their own playbooks and GitHub repos.
If I were just doing this for myself, I would probably store all the playbooks in one repo.
The first Pi, the one you see here, runs Pi-hole.
Pi-hole provides local DNS and tracker blocking on my network.
It also runs Prometheus and Grafana, and I use it to monitor things in my house, like
Internet performance and some environmental sensors.
All that is running in a set of Docker containers set up by Ansible.
The configurations are all Ansible templates, and this makes it so I can maintain this open
source internet-pi project.
The project has hundreds of users, and it's really cool to see how other people have expanded
it and monitor even more things in their homelabs.
Just before AnsibleFest, I added on monitoring for my actual router so I can measure performance
both at the edge and inside my network.
It's a little tricky with a consumer router, but I was able to get it working and I have
a blog post where I go into detail how I did it.
I've been burned by a lack of visibility into my production infrastructure so many times
before, and it's really helpful having a homelab environment where I can explore different
tools and visualize things in different ways, without risking production data.
I mean, some people would look at that and just say you're taking your work home with
you... but well.
Yeah, I guess maybe you could look at it that way.
But that doesn't make it any less fun and rewarding!
I'm also running a few other Pis in the rack, like a Pi running my backup scripts, another
one running Pi-VPN, and a couple more running a Drupal website and an NVR, for Network Video
Recording for my security cameras.
All of those Pis are managed by Ansible playbooks, and the links for those are here, but I'll
also be posting this to my website soon, with even more detail.
I also have a few scrappy playbooks for some of the other devices.
Like the Home Assistant Yellow here.
There's actually a Raspberry Pi inside, but since it's managed by Home Assistant OS, I
haven't found a good way to fully integrate it into my automation.
Which is ironic, because that Pi's controlling all my home automation.
But since it's just Linux under the hood, I'll find a way.
I also have two Macs I use for my main workstations: there's this Mac Studio in my office at home,
and a MacBook Air I take on the road.
Some people just YOLO, and use cloud sync and cloud apps to make multiple computers
work for them.
But me?
I use Ansible.
So I have this playbook that I use to fully automate the configuration of both Macs, even
including things like the position of all my dock items!
I won't dig deep into it because I already did that at last year's AnsibleFest, and there's
also a whole video about it, linked below.
But there's a lot left.
You might've noticed there are a lot more devices in my two racks, especially all the
network switches, the NASes, and my router.
And speaking of the router.
Earlier I showed you this picture of my Internet connection, and you're probably like "oh,
Jeff has Gigabit Internet, nice!"
Well, no.
I didn't show you the rest of that graph.
I mean, 40 megabits of upload isn't nothing, but it can be painful when I'm trying to ship
an offsite backup of a 200 gig video project every week.
And you can also see a bunch of dips on there; one of those was when I was running a backup,
but those other three are outages.
I get at least one or two outages a month, so I've been working on a new router setup
with a 4G fallback.
Right now I'm waffling over whether to build it with plain Linux or use OpenWRT.
We'll see.
But either way, my pi-router project is also up on GitHub, and right now there isn't any
automation, but hopefully soon that'll be wrapped up into my Ansible homelab too.
Unfortunately, there are some devices, like my managed switches, that don't have any Ansible
modules, so I'm still trying to figure out how to manage them better.
Right now I have to go into the UI in my browser, and that's just not fun.
There is a little support for RouterOS on my main switches, but nothing from QNAP for
my PoE switch.
If you get network equipment from one of the better-supported vendors, you can automate
a lot easier.
But wrapping things up a bit: people who manage homelabs do it for a lotta different reasons.
But I think for most of us, whether the homelab looks like a shrine to computing, or something
that would be at home in a trash dump, we just like the thrill of learning.
And cosplaying as a sysadmin.
You get full control over an environment where mistakes can be made and should be made on
a small scale, so you can learn better ways of dealing with them on larger scales.
And just like with the hardware, the same goes for automation.
In some cases, you can manage every single bit of your homelab using Ansible automation.
From your network router to IoT devices, most things have an Ansible module or collection
ready to use.
If you can do it on the command line, chances are you can do it with Ansible.
My homelab is growing every day, and if you wanna use any of my automation in your own
homelab, check out my projects on GitHub, there are links below.
Until next time, I'm Jeff Geerling.