Rebooting servers is annoying. If you don't do it you put your business at risk. If you do it
you upset people. You often need to beg for permission in the first place and then deal
with people repeatedly asking you "Is it back yet?" Well, what if I told you you don't need to reboot?
Welcome back to The Pro Tech Show - the place for tech, tips, and advice for IT pros and decision-makers.
Just so you know where I stand on this: I'm not one of those uptime fetishists who likes to
post pictures of the server showing 10 years of uninterrupted uptime. Most of those pictures
aren't impressive, they're negligent. There are exceptions if you know what you're doing and
if you're one of those people you've probably guessed where this video is going; but if you're
happily showing off your Windows 2000 server with 20 years of uptime then please go hand in your
IT card and call a forensics team to investigate the breaches you've been ignoring. Some updates can
be installed without the need to reboot your server. Some, but not all. Linux tends have an
easier time than Windows and can often get away with service restarts rather than full reboots;
but eventually you'll come across something like a kernel update that needs a reboot to take effect.
Sometimes people don't realise this because while Windows will continually nag you to restart,
Linux tends to assume you know what you're doing and will generally just leave it up to you
(on servers at least - they tend to be command-line ). The problem with reboots is downtime. Nobody
likes downtime and nobody likes users complaining about downtime. So is there another way? Yes. It
turns out the technology to patch a running kernel without downtime has existed for some time.
It's just not that well adopted because it doesn't come working out of the box and is
generally an add-on from enterprise providers. Microsoft are currently trialling their
version of this technology in Azure, but in the enterprise Linux world this already exists and
has been production-ready for years. Red Hat have kpatch, SUSE have kGraft, Canonical have Livepatch,
and CloudLinux have KernelCare. KernelCare is the one I'm going to focus on. It has the widest range of
supported distributions (often outperforming the other vendors on their own distros)
as well as most competitive pricing for datacentre use. That last statement may not hold
true if you've already bought one of the others as part of an enterprise support subscription,
but that comes with a hefty dose of vendor lock-in and an even heftier price tag.
CloudLinux, the makers of KernelCare have sponsored today's video.
That means they've paid to be featured on the channel. It does not mean that they get to change
my opinion or my recommendations. YouTube doesn't pay my mortgage so I only work with people if I
think they've got something worth talking about. KernelCare is definitely worth talking about,
but first let's talk about why this even matters. What's wrong with a reboot? Nothing.
You should reboot your servers from time to time. You should also manage that process. For me
this isn't about never rebooting your servers and winning some uptime competition on Reddit.
This is about managing your risk. In the UK for example it's becoming a common requirement in
contracts to require suppliers to hold Cyber Essentials certification. Part of that certification
requires that security updates are applied within 14 days of release. There are plenty of
other certifications with similar requirements. Not everyone can have downtime windows every 14 days.
I can think of one customer off the top my head for which an outage on their main system typically
takes several months of advance notice and a whole host of signatures because of the impact it causes.
So on one hand you have this requirement to protect the business from risk and meet your
contractual obligations, and on the other hand you have this requirement to protect the business
operations so they can stay in business. KernelCare removes that conflict so you can patch the
system now and protect the business; and you don't need to reboot so that can wait until the next
planned maintenance window, which might be months away. At the risk of broad-brushing here I would say
most of my customers over the years have largely fitted into one of the following three categories.
You've got the enterprise customer with a critical system that they can't afford to go down,
so they invest in making that system highly available. At the end of the day if you can't
afford to have a system go down for a reboot then you can't afford to have it go down because
of a hardware failure either; so they scale it out. They'll at least lead an extra server for failover,
maybe a third to arbitrate failovers between the first two, then a load balancer, actually two load
balancers so that doesn't become a single point of failure itself, and there you have yourself a
nice little cluster that you can patch one node on while the other keeps running... usually. Whilst this
is unquestionably a good idea, as you can imagine it's also a fair bit more expensive than your
single server. So a lot of smaller organisations will say "No thanks. We'll just plan in the reboots
and suck it up". Which is fine, but then you have the other ones that say "We'll plan in the reboots"
and then when it comes time to reboot they say "No, no, no, no. We can't have it". Don't be that one!
So what does KernelCare do for these guys? Let's start with the breach magnet over here.
If you're unfortunate to work in that IT department then KernelCare is basically
your "Get Out of Jail Free" card. You can patch kernel vulnerabilities without the argument about
downtime, and at a much lower cost than having to scale out to a highly available solution just
for the sake of patching. How about this guy who accepted the need for occasional planned downtime?
For them it's all about reducing the risk window. If a patch comes out and your next maintenance
window is a few weeks away, that's a few weeks while you're still vulnerable. I have recently
had first-hand experience of a rather extreme case of this. You've probably seen the news
about the Microsoft Exchange Server breaches. It's a very widely used piece of software and there
are several zero-day exploits that were discovered. I'm not kidding. Within hours of that announcement
going out we were seeing multiple attacks aimed at our customers. Fortunately, our team were quick
off the mark and contacted customers immediately to tell them they needed to approve an outage now,
or they were going to get breached. Not everyone was quite so quick off the mark.
Hundreds of thousands of servers were breached, globally. I'm telling you right now that if there
was such a thing as KernelCare for Microsoft Exchange it would be flying off the virtual
shelves. OK, let's move on to Mr. Enterprise, here. He's got a cluster - what does he care?
Clusters take a lot of the pain out of rebooting, but that doesn't necessarily mean they get updated
any more frequently than the last guy, or that they're completely non-disruptive. So the same use
case exists. Typically, you'll update part of the cluster and run it through some testing for a bit,
and of course the live half is still vulnerable. KernelCare checks for and applies patches
every four hours. It avoids the need for a reboot by patching the running kernel in-memory.
Normally, you'd install the new kernel alongside the old one, the old one keeps running, and then
you reboot into the new kernel; but KernelCare modifies the live kernel directly in-memory,
as it's running. Because it's done in-memory it's also easy to undo if you need to. You don't even
need to reboot to revert the change, which gives it an edge over other options on the market.
If you can honestly tell me that you fully patch and reboot all of your clusters within four hours
of an update being released then... seriously, tell me in the comments because I'd love to hear about it.
It's not impossible but it does need a high level of automation and I find that stuff interesting. As
one of my viewers kindly suggested in the comments of another video, though; "Don't tell me. Show me"...
or he might have said the other way around. In any case, "THB Admin"... fine. Let's stop talking
and let's put this to the test. I'm going to fire up a virtual machine running CentOS 7.2.
I know that's old. That's the point. I want one that hasn't been patched so I can exploit it.
If I test KernelCare against a fully-patched machine that's not a very good test, is it?
I know CentOS 7.2 is vulnerable to Dirty COW so that's what I'm going to throw at it.
I'm not throwing a literal cow, it's a copy-on-write bug. Copy-on-write: C-O-W. It was a big deal
a few years ago because basically the entire Linux world was vulnerable to it. Dirty DOW
is a privilege escalation vulnerability. I can use it to take an account with limited permissions and
gain root access over the entire server. It goes without saying that you should never
do this without the explicit permission of the person who owns the computer you're running it on.
Don't bother asking for a link to this one. You're free to Google it but I'm not linking
to actual exploit code. I'm logged on with a standard, non-privileged user account, creatively
named "Andrew". It has no special permissions, so if I try to do something silly like delete my DNS
resolvers file it will tell me to go away. If I try using sudo there's no joy either. I just get
told that i'll be reported to myself for trying. Let's see what happens if I exploit Dirty COW,
running as Andrew. My username is no longer "Andrew", it's now "root"; and I can delete my resolvers file.
I now have full control over the entire server and I can do what I want. Dirty COW wins. Now let's
revert this virtual machine back to a snapshot I took before I ran the exploit. That will undo any
changes we've made to it, and reset us back to a clean slate. I'm going to switch to the root
user and I'm going to update the kernel. The current kernel is 3.10.0-327 and we know this
kernel version is vulnerable. "Yum update kernel", and I'm going to fast-forward the video a bit.
OK, you can see I've installed kernel 3.10.0-1160 which is not vulnerable to Dirty COW. I'm going to
drop back to my non-privileged user account and try exploiting it again.
And... that's worked again. Why? Because I've installed the kernel update but I haven't rebooted yet;
so I'm actually running the old, unpatched kernel. Let's revert to snapshot and I'll do it all over again
but I'll fast-forward the video to the end and this time I'll reboot. OK, rebooting now.
We're back. We're now running our updated kernel and if I try to exploit Dirty COW now
it doesn't work. It just gets stuck here. This is the normal way to patch a kernel vulnerability, but it
didn't work without that reboot, which means I need to arrange an outage; and that's the problem that
KernelCare is going to solve. Now, you might ask why I'd suggest using KernelCare on this CentOS
machine rather than Red Kat's kpatch service, given Red Hat were the ones to release CentOS.
Well, Red Hat doesn't support kpatch on CentOS. I'd have to switch over to Red Hat Enterprise Linux.
There is another option I'll mention later. I don't recommend it, but I will cover it for completeness.
For now, let's walk the supported path. Compared to the price of KernelCare,
Red Hat's subscriptions are... expensive. These support packages include much more than kpatch, of course.
Quite notably, most of them include full operating system support; although not the
self-support option, obviously. Don't misunderstand me - this isn't an apples-to-apples comparison;
but maybe you only want to buy an apple instead of the entire fruit bowl. If you're
happy on CentOS then a Red Hat subscription is an expensive way to get live kernel patching.
You should really be thinking about moving off CentOS for other reasons, but that's another video.
KernelCare by comparison starts at $25 per server, per year at its cheapest; and is
$45 at it's most expensive if you literally buy a single license; and it doesn't have
limitations like only supporting the most current miner release. In fact, KernelCare provides updates
for more versions of Red Hat than Red Hat do - even with their Premium support plan. It's a similar
story for the other distro's. KernelCare comes in cheaper than native options from SUSE, Canonical,
and Oracle too. Of course it's only actually worth the money if it works, so let's give it a go.
We're back at the same server, at our snapshot before any changes were made.
Our kernel is the version that ends "-327", which is the vulnerable one.
The first thing we'll obviously need to do is to install KernelCare. I'm going to
switch to the root account for that and then paste in a single command to download and install it.
That's it installed, and now we need to register the server with CloudLinux. One more command for
that and it's live. KernelCare is now actively protecting the server. KernelCare will check
for new patches and automatically apply them in-memory every four hours; but nobody's going to
watch this video for four hours, so I'm going to jump the queue and tell it to update immediately.
There we go - it says the patch has been applied, and we haven't so much as dropped a ping. If I run
"unamr -r" to see the current kernel it still shows the old one ending with "-327". That's because
KernelCare has done its magic live in-memory. The installed kernel on disk hasn't been modified.
Because the effectively running kernel and the installed kernel are now different, KernelCare
includes a command "kcare-uname" to show your effective kernel version. You can see that command
says the kernel is the "-1160" version, which is the updated version that should protect us against
Dirty COW. That means that according to KernelCare we have injected the security fixes from 1160
into our live 327 kernel; but we've not rebooted so clearly we have to test this and see what happens.
I'm dropping back to my standard user account now and here's the moment of truth. Go Dirty COW!
Hopefully you can see the little blinking cursor so you know the video is still running here. That
hasn't worked. The exploit has been blocked because even though the installed kernel is vulnerable, the
kernel actually running in memory has been updated on the fly by KernelCare to a newer version;
and there was no downtime required. If I do reboot now, CentOS will boot using the installed kernel
(which is the old one), but KernelCare will reapply the patch in-memory again as soon as we reboot
to make sure we stay protected until we get around to updating the CentOS installation.
So don't worry - you can't break rebootless patching with a reboot. I tried. And that's KernelCare in a
nutshell. It runs in the background and makes sure the running kernel in memory is kept up-to-date
with security fixes, whether you reboot or not, and even if you haven't installed any kernel
updates via your package manager yet. In effect, your kernel is always protected. You can install
your normal updates whenever you want and bring your applications up-to-date, but you don't need to
reboot the server because the thing that needs the reboot to take effect (the kernel) has already been
patched in-memory anyway. So I like it, but I'm sure some of you are wondering... if this is Linux, has
anyone released the source code? And if they have, has anyone packaged it up with a free distribution?
Yes. No. Maybe... but be careful! There are some limited-use free versions and trials kicking
about to entice you over the various enterprise providers; but all of them have limitations and if
your use case is the datacentre (which let's face it is where the stuff lives) then you'll end up paying.
By all means, use them to learn what's out there, but when I talk about "free" I'm not considering those.
I mean really free. That does exist because Red Hat's kpatch is open source. You can go
download to the source code and compile it yourself - even if you're not using Red Hat. Do I recommend
doing that in production? No. Applying a live kernel patch is a fair bit more complex than
running "yum" or "apt upgrade". You're building and applying a custom patch every time you use it.
That comes with two major issues. The first is spelled out quite clearly in bold on the
kpatch GitHub. It says: "WARNING: Use with caution! Kernel crashes, spontaneous reboots, and data
loss may occur!" Kernels are complicated. Modifying running kernels live in-memory
is complicated. There's a reason why this isn't a standard part of free distros, despite
being open source; and why the enterprise Linux providers charge top dollar for it.
The business case for doing live kernel patching hinges around the benefit of risk reduction.
By all means have a play in your own time. I always encourage people to learn about new technologies.
But if you try to YOLO this in production can you still claim that you're reducing business risk?
I would argue not, which leads me to the second reason I don't recommend the DIY approach in
production: it's not cost-effective. OK, how can I say it's not cost-effective when it's free?
Well let's take another look at GitHub, and another sentence written in bold. "Do not
Do not blindly apply patches. There is no substitute for human analysis and reasoning on a per-patch basis.
All patches must be thoroughly analyzed by a human kernel expert who completely understands the patch
and the affected code and how they relate to the live patching environment". OK, we've covered the
point about not YOLOing it, but let's talk about this "human kernel expert". If you're going to use
this in production in a sensible and safe way, then you're going to have to do all of
this analysis and testing yourself; or rather, you'll need a kernel expert to do it for you.
Experts don't come cheap. Their time has value. Add up the cost of the kernel expert's time to do the
analysis and the testing every time an update is released, then compared to the cost of KernelCare.
According to Google, the average salary for a Linux kernel developer is $162,623 per year...
or you can get KernelCare for $25 per server, per year;
and that comes with an entire team of kernel developers. By the way, is anyone else now thinking
they're in the wrong job? That was average, not top-end! To be clear, I'm not saying you shouldn't play
with the open source code. You should. Learning is good. I'm saying that you shouldn't bet
your business on your ability to DIY your own kernel patches. KernelCare's value proposition
isn't about getting the software itself. It's about getting CloudLinux's expertise to build,
test, and support the kernel patches for you in a timely manner, so you can be protected
as quickly as possible. While I'm talking about pricing I should point out the KernelCare also
have a "plus" version that's a bit more expensive. The differences are more responsive support options,
integration with vulnerability scanners, an option for an on-premises patch repository, support for a
few additional distributions... These actually seem to be the same distros but an additional version
back; so with Red Hat you get 5 onwards with plus whereas it's 6 onwards without plus. The
main difference I wanted to flag though is support for live patching of OpenSSL and glibc as well as
the kernel. The thing with these shared libraries is that because they're used by so many things;
although you can theoretically update them without a reboot you might have to find and stop
so many services to do it that it practically becomes a reboot anyway, with a similar outage.
At that point you may as well just reboot and get it over with.
So expanding the live patching technology beyond the kernel is an interesting idea.
The proposition here is basically: pay a bit more, avoid a bit more downtime.
I don't have a lot more to say about it, really. I think the technology is really interesting
and it addresses a real need for companies looking to reduce their risk exposure without sacrificing
uptime. If you have any questions then drop them in the comments. I will check back in a bit and
I'll pop them over the CloudLinux team if they're not already stalking the video. My goal with these
collaboration videos is to provide as much value as I can for you guys, so make the most of it.
Also, don't forget that for only $0 per month you can like and subscribe to this channel!
If you found this video interesting then have a look at one of these ones over here.
As for me? I'm off to retrain as a kernel developer!
Thanks for watching, guys. See you next time!