SANDEEP DINESH: When it comes to distributed systems,
handling failure is key.
Kubernetes helps with this by utilizing controllers
that can watch the state of your system
and then restart services that stop performing.
But on the other hand, Kubernetes
can actually forcibly terminate your applications
to ensure the overall health of the system.
In this episode of Kubernetes Best Practices,
let's take a look at how you can help Kubernetes
do its job more efficiently and reduce the downtime
your apps experience.
[MUSIC PLAYING]
In the pre-container world, most applications
were running on VMs or physical machines.
If an application crashed, it could take quite a while
to boot up a replacement to take its place.
Or even worse, someone would need to log into the machine
and fix the issue by hand--
even in the middle of the night.
If you only had one or two machines
to run the application, this time to recovery
was unacceptable.
So instead, it became common to use process-level monitoring
to restart applications when they would crash.
If the application crashed, the monitoring process
could capture the exit code and then
restart the server instantly.
With the advent of systems like Kubernetes, this kind of system
was just built into the infrastructure itself.
Kubernetes uses an invent loop to make sure
that resources are healthy from containers
to the nodes themselves.
This means you no longer need to manually run these monitoring
processes.
If a resource has failed a health check,
Kubernetes will just automatically spin up
a replacement.
But Kubernetes does a lot more than monitor your application
for crashes.
It can create more copies of your application
to run on multiple machines.
It can update your application.
It can even run multiple versions of your application
at the same time.
So this means that there are many reasons why
Kubernetes might terminate a perfectly healthy container.
Like if you update your deployment
with a rolling update, Kubernetes
will slowly terminate old pods, while spinning up new ones.
If you drain a node, Kubernetes will terminate all pods
in that node.
And if a node runs out of resources,
Kubernetes will terminate pods to free those resources.
So it's important that your application
can handle termination gracefully so
that if there is a minimal impact on the end user
and there's a minimal impact on the time to recovery.
So this means that it should save all the data that
needs to be saved, close down any network connections,
finish any work that's left--
other similar tasks.
So in practice, this means your application needs
to handle the sigterm message.
When it receives this message, it should begin to shut down.
Once Kubernetes decided to terminate your pod,
a series of events takes place.
Let's look at each one of these steps in the Kubernetes
termination lifecycle.
So let's say we want to terminate one of these pods.
At this point, the pod will stop getting new traffic.
Containers running in the pod will not be affected,
but all new traffic will be prevented from coming in.
So now let's look at the prestop hook.
The prestop hook is a special command or HTTP
request that is sent to the containers in the pod.
If your application doesn't gracefully
shut down when receiving a sigterm,
you can use this hook to trigger a graceful shutdown.
So most programs gracefully shut down when receiving a sigterm.
But if you're using a third-party code or some system
that you don't really have control over,
the prestop hook is a great way to trigger a graceful shutdown,
without modifying the application.
So now, at this point, Kubernetes
will send the sigterm signal to the containers in the pod.
This signal lets the containers know that they're
going to be shut down soon.
So your code should listen for this event
and then start shutting down cleanly at this point.
So this may include stopping any long-lived connections,
like a database connection or a WebSocket stream,
saving the current state, and doing things like that.
So even if you're using the prestop hook,
it's important that you test what
happens to your application when you send it the sigterm signal
so you're not surprised when it happens in production.
At this point, Kubernetes will wait
for a specified time called the termination grace period.
By default, this is 30 seconds.
It's important to note that this happens
in parallel to the prestop hook and the sigterm signal.
Kubernetes will not wait for the prestop hook or the sigterm
signal to finish.
If your app finishes shutting down before the termination
grace period is done, Kubernetes will just move to the next step
immediately.
Now if your pod takes longer than 30 seconds to shut down,
make sure you increase the grace period.
You can do that by setting the termination grace period
seconds option in the pod YAML.
For example, this example has it set at 60 seconds.
And now, the last step--
if the containers are still running after the grace period,
they are sent the sigkill signal and then forcibly removed.
At this point, Kubernetes cleans up all the objects as well.
Kubernetes terminates pods for many reasons
and making sure that your application handles
these terminations gracefully is core
to creating a stable surface.
I'll see you on the next episode of Kubernetes Best Practices.
[MUSIC PLAYING]