[MUSIC PLAYING]
ANTHONY BUSHONG: When upgrading node pools in a GKE cluster,
you probably care about a few important things.
Do workloads remain as available as possible during the upgrade?
Can we mitigate risk with granular control
of upgrades and rollbacks?
How do we manage cost, speed, and the duration of an upgrade?
In this episode, we're going to work
through how to navigate these important questions when
using the node pool upgrade strategies built into GKE.
[MUSIC PLAYING]
In GKE, there are two node pool upgrade strategies.
GKE has surge upgrades in which nodes in a node pool
are upgraded in a rolling fashion.
With surge upgrades, you can control
how many nodes are upgraded at once, adjusting this
to the disruption that your workloads can tolerate.
GKE also has blue-green upgrades in which
all old, or blue, nodes are kept in the node pool
during the upgrade as you validate your workloads
on the new, or green, nodes.
Let's start by reviewing the default node pool upgrade
strategy, a pre-configured surge upgrade.
And we'll use an example node pool that has three nodes.
The default node pool upgrade path
will begin by creating an additional surge
node in the node pool, one with the new version of the node.
Once that node has completed creating and is ready,
we'll briefly have four nodes in our node pool.
Then the upgrade will begin the process of turning down
one of the old nodes.
This involves draining the old node so existing pods can
terminate, coordinating the nodes
so new pods won't be scheduled there,
and then deleting the node itself.
Here we can see the benefit of creating
that new node, the extra surge node that temporarily brought
our node count above the baseline.
Its extra resources can now be allocated
to pods that have to reschedule from the old node.
This reduces the time it takes to get
pods restarted and working.
Of course, there are considerations
when pods are moving from node to node,
like how can we get pods to shut down gracefully?
That is, how do we let them know they
need to perform tasks and give them time to do so,
like closing connections or flushing things in memory
to disk.
In Kubernetes, you can have workloads
specify values for preStopHooks and a termination grace
period in the pod spec.
These allow you to define what tasks your pod
should carry out before they shut down
and give them a defined time to perform them.
Because upgrades perform a drain,
the containers in your pods receive a sig term message,
which allows for them to begin carrying out these tasks.
Of course, workload availability is more than just
how a single pod shuts down.
It's about making sure you have enough pods running
to carry out whatever your workload does.
This means working through details like making sure
you have enough pod replicas and that number of replicas
can scale up and down, making sure replicas don't bunch up
on a couple of nodes, using things like nodeAntiAffinity,
and defining rules around keeping
a minimum number of replicas around at any given time using
PodDisruptionBudgets.
GKE does interact with PodDisruptionBudgets
specifically, allowing to block an upgrade for 60 minutes
should one be violated.
But overall, these recommendations
are not specific to GKE.
In fact, you should consider these
for disruption sensitive workloads
in any Kubernetes environment.
With that said, once the pods have
been drained from the old node, that node is deleted
and we repeat the cycle until the whole node
pool is complete.
The default upgrade strategy shows us
how surge upgrades work in GKE.
And while we just went through the default configuration,
you can actually tune a couple of settings
to best suit your needs.
The first is called max-surge-upgrade.
By default, this parameter is set to 1.
We saw this earlier in the default upgrade strategy
when the upgrade started by adding one surge node
above the baseline node count.
If we were to set this value ourselves, for example to 2.
During the upgrade, the node pool
could surge above the baseline node count
by two additional nodes.
There are a couple of reasons we want to tune this.
We may have larger node pools that we
want to finish upgrading faster, so we
want to increase the speed of new nodes being added
to the node pool, or we may want to ensure
that it's worth the slightly extra cost to have
spare resources to allocate to pods moving around
during the upgrade.
Now on the other side of the coin,
we have max-unavailable-upgrade as a tunable field.
This is the number of nodes below the baseline node
count that can be unavailable at any given time.
In the default upgrade strategy, this value
is set to 0, meaning that the node pool never
drops below ready and available baseline node count.
If we set this to 1, the upgrade can turn down nodes
at a rate that allows the upgrade
to fall one node below the baseline count.
By doing so, it helps us speed up
upgrades by cordoning and draining nodes faster.
So in this example, we left off with five nodes,
two that surged above the baseline count.
This means that our upgrade can now
cordon and drain up to three nodes at the same time.
The faster nodes are cordoned and drained,
the more pods are moving around and being
rescheduled at any given time.
So again, this can be slightly disruptive for pods,
but it does speed up the upgrade process.
There's a lot of power in between these two settings
with a lot of configurable outcomes
depending on what's important to you.
You could take a slower balanced approach,
similar to the default configuration.
You could turn down nodes faster than you create new ones, which
optimizes for speed and lowering the cost of an upgrade.
But this can be disruptive if workloads
don't have available resources to reschedule to.
Finally, you could create nodes faster
than you turn down old ones.
This optimizes for speed and spare capacity
as pods move around.
And there are also some protections
built into these strategies.
For example, when you set max-surge-upgrade to 0,
you're protected against any temporary stock
outs should the upgrade not be able to secure capacity
as it adds nodes.
There are a lot of different ways
you can use these two settings.
Surge upgrades give you a lot of control
over the speed of an upgrade.
And this is helpful, especially in scenarios
in which you have large node pools that
need to go through an upgrade.
However, no matter what, surge upgrades
are performed in a rolling fashion.
The tradeoff for controlling the speed of an upgrade
is less granular control over workload disruption and slower
rollbacks should you ever need to.
So if your workloads are disruption sensitive
and your priority is to have more control to mitigate risks
during an upgrade, you might consider
the other strategy built into GKE, blue-green upgrades.
This upgrade strategy provides more control
around promotion of new node versions and safer rollbacks.
We can step through an example blue-green
upgrade via its phases.
We start in the first phase by creating
an entirely new green set of nodes
with the new versions of the nodes we want.
More importantly, this set of nodes
will be equal to the old blue node count right at phase one.
In this example, we add three new green nodes
to the node pool.
Once those are ready and available,
we step into phase two, coordinating old blue nodes.
After all, we don't want newly created pods
to be scheduled to old nodes only to be moved again,
especially since we can utilize the free and allocatable
capacity of the new nodes.
Once we've cordoned off the old blue nodes,
we can begin the process of draining them.
This will flush any existing pods on those nodes,
allowing them to reschedule to the new green nodes.
And this is where we begin to get more granular control.
We can define either an explicit number of old blue nodes
to drain at a time or an explicit percentage
of old blue nodes to be drained at a specific time.
This example shows what would happen
if we drain nodes in batches of two
until all of the old blue nodes are drained.
We can also define a window of time
to sit and wait before draining the next batch of nodes.
This helps us move workloads to the new green nodes as slow
as we want, all without having to worry about whether or not
the capacity is available.
It's also important to call out that we are not
releasing the old blue nodes during this time.
Workloads such as stateful workloads
that need more time to shut down to do things
like flushing memory to disk can now take that time
on the old blue node.
And if we wanted to roll back for any given reason,
the old nodes will be uncordoned and quickly
made available for workloads to schedule back to.
Once that's done we step into phase four, in which we can
define a soak time duration.
This let's us observe how all the workloads are
behaving on the new nodes.
And if we need to, we can quickly
roll back workloads to the old blue nodes during this time.
And we can even automate this process based on SLOs
that we set in Cloud Monitoring.
Now the default soak time is one hour,
but we can bump this all the way up to seven days,
giving us control to observe our workloads,
especially ones with diurnal or spiky usage patterns.
Once we validated our workloads, we can promote the upgrade
and release the old blue notes that we no longer need, that is
if there is soak time left.
The upgrade will also continue if the soak time expires.
It is then that we finally release the old blue nodes
by deleting them.
With the upgrade complete, we are finally back
at our stable baseline node count of three.
As you can see, blue-green upgrades
optimize for stability, availability, and risk
mitigation.
Resources are still there to roll back
to as you validate your workloads
on the new set of nodes.
The tradeoff in this case is the additional cost
of doubling your node pool resources
during the upgrade time.
With that said, many will evaluate
the cost of a slower rollback process for critical workloads
and take that into consideration as well.
Surge upgrades and blue-green upgrades
optimize for different things.
Surge upgrades are better for workloads
where speed is more important to you
at the cost of a little bit of disruption.
Blue-green upgrades are better for workloads
focused on stability, availability, and risk
mitigation at the cost of running
more nodes for the duration of the upgrade.
The best part is these are on a per node pool basis.
In GKE, we understand that your cluster
may have multiple profiles of workloads
across multiple different node pools,
so you can choose your upgrade strategies accordingly
per node pool.
Of course, there's more to follow up
on to fully shape your upgrade strategy in GKE, like release
channels, testing new versions of GKE and preprod clusters,
and maintenance windows and exclusions.
To learn more, stay tuned and check out the links
in the description below.
Thanks for spending this time with me.
And we'll see you next time on GKE Essentials.
[MUSIC PLAYING]