People will often tell you that the reason they prefer Linux to Windows is that they don't
need to constantly reboot it. They'll brag about months or even years of uptime, and then someone
else will chime in to say what they're actually doing is cultivating a massive security threat.
So, what's the truth? Are they right? Do you need to reboot Linux? Yes, you need to reboot; but that's
the simple answer. The more complicated answer is that they might actually be right... kind of. It's
nuanced, and their victory may be a shallow one, but there is an option where everybody wins!
Welcome back to The Pro Tech Show. There are a few reasons someone might say that you don't
need to reboot Linux. Some of these reasons are better than others. The biggest misunderstanding
occurs when people approach Linux like it's Windows. Windows, as I'm sure you know,
has zero reservations about nagging you to reboot after installing updates. Anyone
familiar with Windows is used to installing updates and getting either an "all done" message
or being told you need to reboot now. This applies to servers as well as desktops. Linux on the other
hand... a lot of people worrying about reboots on Linux are dealing with servers. Nobody cares if you
need to reboot your laptop; but reboot a server and now you need to arrange downtime with your business.
Linux servers tend to be command-line and when you update Linux using the command line
it tells you what it's going to do and then it tells you when it's done. That's it. There
is usually nothing shouting at you to reboot. It varies between distros, of course. Some will tell
you that a reboot is needed when you log in, for example. Others won't, but most I've used
don't actually tell you at the point you install the updates. It may sound silly to some of you
but on several occasions I've seen an admin who is more familiar with Windows declaring that their
Linux server didn't need a reboot simply because the package manager didn't tell them to reboot.
They were so used to Windows they didn't realise it was never going to tell them!
Most desktop distributions I've seen will quite clearly tell you that you need to reboot when you
update using the graphical tools. They nag you almost as much as Windows does, so desktop Linux users are
probably more used to rebooting; but this notion that you don't need to reboot doesn't simply come
from ignorance. There really is a valid argument for it. Linux is much more granular than Windows.
Windows is Windows. Linux is a kernel packaged up with hundreds, if not thousands of other pieces
of software - most of which have been developed independently and can be updated independently.
It's also better replacing files that are in use due to differences in the way the filesystem
works compared to Windows. This means you can swap out files that belong to running programs without
taking the system down. It works because Linux is using a reference to the file and when you
replace the file the new version of that file has a new reference which doesn't affect the old one.
But that means the running program isn't updated. You may have replaced the file as it appears
on your hard drive, but if the program keeps the old version open in memory it doesn't get updated.
Because of this there's a common need to stop and restart running software when you update it,
whether you're using Windows or Linux. Windows is built as more of a cohesive unit than most Linux
distributions. Windows hotfixes are also packaged up into one big cumulative update that affects
many different components. You can't really update individual components of Windows. It's more of an
all-or-nothing affair, and restarting core parts of Windows inevitably means restarting the whole
thing because it's built as one single experience. Linux is different. You don't need the vast
majority of software on a Linux install to make a basic functioning computer. A kernel plus some core
packages is enough for a basic command-line system, and then we add more to make it useful to us.
Because the Linux operating system is made up of lots of what are essentially optional packages
we can update them and restart them without affecting everything else. You can even restart the
entire graphical environment in Linux. You can't do that in Windows because Windows is the graphical
environment. In Linux it's optional. You can stop, restart, and even swap out your desktop environment
without rebooting the computer because it's all just optional packages layered on top of a
command-line operating system, layered on top of a kernel. In fact until you get to the kernel itself
there is very little that you can't restart on the fly. The kernel, though... that's very much a red line.
If you stop the kernel you stop everything so pretty much everyone will agree that that needs
a reboot. Except there is a way around it. You can live-patch a running kernel in-memory. I made a
video about this a while back that I'll link in the description, using a tool called KernelCare.
TuxCare is the umbrella brand that provides KernelCare and they have sponsored this video; but
despite what you may be anticipating I'm not here to tell you about live kernel patching. I mean it
is really cool and all but you can watch the other video for that. I'm here to talk about a different
problem, and that is libraries. Very, very few pieces of software are truly written from scratch.
It would be massively wasteful to keep writing the same common functions over and over again so these
common functions are packaged up into libraries, then anytime a developer needs one they just
use that pre-written library. Those libraries need patching too or they can introduce vulnerabilities
into your applications. I'm going to show you an example using a very common library called glibc.
I have an aging Ubuntu server here that's missing some updates for glibc. These missing updates leave
it vulnerable to a local privilege escalation attack. You can see my username here is "Andrew".
If I try to install an update for glibc it says "no" because my permissions aren't elevated. Now I'll
run my exploit. I say "my exploit"... I wrote nothing. It's halfdog's exploit for CVE-2018-1000001.
When I run it you'll see my username has changed to "root". No password entered, no sudo, I'm just root.
I can now run that update and it'll work just fine. Let's undo that by reverting to a snapshot I took
before I started. This time we'll use sudo like a good admin and install the update to glibc, first.
Notice that I actually have to type a password when I'm not using an exploit.
Now we've patched it, let's try our exploit again. You can see multiple attempts but in the end it
gives up. I'm not root, I'm still Andrew. This is why we patch things. Now, that didn't require a reboot,
did it? Are you sure? Remember what I said about other applications using libraries like glibc?
And remember what I said about replacing files on disk not updating all the references that were
already in memory? The exploit calls glibc directly to take advantage of its vulnerability, but there
will be other software running on the server that also uses glibc and if it was loaded before we
patched it it's still vulnerable. To give you an idea of the potential scope here let's try and
remove glibc. Look at all these packages it would force us to get rid of because they rely on it!
"You're about to do something potentially harmful. To continue typing the phrase 'Yes, do as I say !'"
No. Do not type that! We can get a better idea of our exposure by running a free Python script called
uchecker from our friends at TuxCare. It will look for unpatched libraries in memory and report
back to us. I'll link it in the description. There's a lot going on here so let's grep for glibc.
We can see a lot of services are running outdated versions of glibc. That's a problem. I spot Dovecot,
Apache, and Exim in that list. That's IMAP, HTTP, and SMTP; which are often exposed to the internet.
This could be a publicly accessible email server, which means we're at a bigger risk here than a
local privilege escalation. If those services are still using a vulnerable version of glibc
they could be vulnerable to remote code execution - meaning we could be hacked by someone on the other
side of the planet! There are a few ways of dealing with this. One is to restart the affected services
so they dump their vulnerable version of glibc and load the patched version.
If I restart Apache and rerun that script, you'll see it has disappeared from the list;
but I did just incur a brief service outage. The problem with doing it this way is that it won't
always be a static list of processes. The results here could change from time to time so you'll
have to go through them each time, figure out the services to which the affected processes belong,
arrange downtime for them, and restart them. Spoiler: you're going to get stuck on D-Bus. It'll
need a reboot, anyway. To be honest: I would just reboot. It's easier. It will sort out things like
D-Bus and your kernel. You don't have to worry about missing something. In my mind having an extra 10,
20 seconds or so of downtime to complete a reboot instead of messing around with tracking down and
restarting individual services just makes sense; but I know not everyone can afford to have that
downtime, so let's revert to snapshot again and bring in TuxCare. In addition to the KernelCare
product that we looked at before that can live patch a running kernel, TuxCare have another
trick up their sleeve called LibraryCare. LibraryCare is an add-on for KernelCare that can live-patch
glibc and OpenSSL - two very commonly used libraries. The first thing we need to do
is install KernelCare, which is just a one-liner that works on any of the supported distributions.
Then we need to register our licence and we're good to go. We can set this run every four hours or
just run it right now. I'm assuming you don't want to wait four hours! First, let's run the
uchecker script to get the "before" state. To really check this works as advertised I'm going to open
a manual IMAP connection to Dovecot and see if it manages to stay connected throughout. IMAP is a
protocol used by your email client but it's based on plaintext so I can literally just open a
command prompt and connect to it. Now "kcarectl --lib-update" and it's found and patched 42 processes.
No reboots, no service restarts, no downtime. My IMAP connection hasn't even dropped. Let's run
the uchecker script again and that's looking a lot better. All those scary services that could
be exposed to the internet have gone. In fact, most of what's left was me running uchecker!
Which brings me to an important point... our exploit still works.
What!? Wasn't this supposed to protect us against the exploit? LibraryCare protects processes
in-memory. It does not update the binaries on disk. You still need to patch your server or every new
process is going to load the unpatched version. If I restart Dovecot now it's going to load the
vulnerable version of glibc from disk. It'll also kill my IMAP connection. LibraryCare will find
it and patch it again in-memory the next time it runs, but that's not the way you're supposed to use
it. If you can restart services then you don't need LibraryCare. Just patch them and restart.
LibraryCare is for long-running processes that are too critical to be restarted. You patch the
installation as normal. LibraryCare complements your patching by going and protecting all of the
processes in-memory that have outdated copies of glibc or OpenSSL that your patching wasn't
able to touch; and it does it without the need to restart any of them, so you don't need downtime.
So if we can restart non-critical processes, we can't restart the kernel but we can live-patch it
with KernelCare, and we can't practically restart every memory-resident copy of glibc or OpenSSL
but we can patch it with LibraryCare... surely by now we can say Linux doesn't need to be rebooted?
I mean you could say that but I'm going to disagree. Just because it doesn't need a reboot
doesn't mean you shouldn't give it one. Live-patching isn't intended to be indefinite, and
there are other reasons to reboot beyond patching. Let me tell you about one of the first jobs I was
given when I started in IT. A company needed help restoring a backup of their system so I
was sent to give them a hand. I was familiar with the software so it should be an easy job, right?
Yeah, no. What had happened was that their server was having some relatively minor issues
and as a troubleshooting step they rebooted it. Unfortunately once they rebooted it, it did not come
back up. The system files were a mess and it wasn't bootable so they restored from last night's backup.
That didn't work either. It turns out they kept six months of backups and this server hadn't been
rebooted in over a year. I don't know exactly when the problem developed but I can tell you that it
was more than six months and less than over a year. None of their backups could be used to recover.
We went back the very earliest one they had but it was still broken. They incurred quite a bit
of disruption while I put this thing back together manually. It wasn't a lot of fun for them. Sometimes
it's a good idea to reboot if for no other reason than to prove the system will come back up.
Obviously, there were other problems here besides a lack of reboots. Like a lack of backup testing!
But if you're going to get a surprise you want to find it during a planned maintenance window
when you have a rollback option. Not in the middle of production because you left it too long.
But this is the catch, isn't it? Getting that maintenance window. We'd all like to be able to
just have one whenever we want but if the system is business-critical you could be waiting a while.
If it's safety-critical you could be waiting a long time, indeed! That's a problem because
the bad guys aren't waiting. We're starting to see widespread exploitation within hours of
vulnerabilities being made public. Not months, not weeks, not days. Hours. Security standards like Cyber
Essentials require you to install security updates within 14 days of release. Even having a monthly
patching window doesn't cut it anymore! For those who aren't British, Cyber Essentials is basically
required to work the public sector here so it's a big deal. Let's say a security vulnerability is
published today and your next maintenance window for a critical system is three months away. What
do you do? You can't just reboot it. The fact that Linux lets you apply updates on the fly with just
a few service restarts is cold comfort if you can't afford the outage to restart those services.
Nor is it of much use if it affects one of those few rare things that you can't manage to restart
like D-Bus, that happens to have loaded a vulnerable library. This is why I said LibraryCare lets
everybody win. I still think the occasional reboot is a good idea, but LibraryCare lets you choose
when that happens. You can install the updates to disk. You can patch the vulnerable libraries
in-memory right now. You don't need to reboot... not yet, anyway. Reboot in three months' time when you
have that maintenance window; but for now you can tell the security auditor that LibraryCare's
dealt with vulnerable processes. That's the use case where I think this makes most sense.
Not to eternally avoid reboots but to mitigate the security risk while you wait for the chance to
reboot. But I'm interested to hear your thoughts. Do you agree with my take that you can almost
get away without ever rebooting Linux, but even so you're better off just rebooting it anyway? Are you
a hardcore uptime junkie who will do anything to avoid a reboot? Do you just reboot it after every
patch? That's the simplest option, after all. Let me know in the comments, and if you find the idea of
live-patching shared libraries interesting then check out TuxCare. Also... they currently support
glibc and OpenSSL but they did tell me that they could support other libraries as well if
there's sufficient demand; so if you have something you'd like to see, drop it in the comments and get it on
their radar. If you want to go further down the live-patching rabbit hole this video will show you how
to live-patch your kernel using KernelCare, and this video has been live-patched in by YouTube.
I don't know what it is but I told them to choose something that you'd like, so it should be good!