ANTHONY BUSHONG: Welcome to "Bit by Bit,"
a series where we highlight one thing you can do to optimize
and potentially run a bit better in Google Cloud.
Today's topic-- making sure we're
more efficient with resources in our GKE standard clusters.
One way to do that is by configuring the cluster
autoscaler in GKE with the optimized utilization
autoscaling profile.
Let's get to it.
[MUSIC PLAYING]
The Kubernetes Cluster Autoscaler
is the gold standard for making sure your cluster can
add or remove nodes based on demand.
And my favorite thing about the cluster autoscaler in GKE
is that Google fully manages the software.
It's one less cluster component lifecycle
that we have to worry about.
So in this demo, we're going to use these two clusters,
each with cluster autoscaler enabled,
but with one main difference.
In GKE, the way you influence this fully managed cluster
autoscaler is by configuring autoscaling profiles.
These inform the cluster autoscaler's behavior.
So the default autoscaling profile
is set here in our California GKE cluster.
It's called Balanced.
This means it prioritizes keeping more resources readily
available for income pods.
Our Oregon cluster, on the other hand,
has a different autoscaling profile
called Optimize utilization.
This makes the cluster autoscaler scale down
nodes more aggressively to be more resource efficient.
So we're going to run an experiment
to see the difference between these two
clusters and their respective autoscaling profiles.
So we can see here that they have the same node count,
and they have the same amount of CPU and memory.
And we can use the KUTTL command line tool to check in on them
and make sure that they are also running the same workload.
So we can see California is running a deployment called
alpha with 45 pods.
And we're now going to use KUTTX to switch over to the Oregon
cluster, can see that it's running
the same deployment, same number of pods.
And then what we're going to do now,
is we're going to delete the workload or this deployment
in each of them and watch how the Oregon GKE cluster scales
down more aggressively than the California GKE cluster
with the default autoscaling profile.
So we're going to delete the deployment in the Oregon
cluster, switch contacts to our California cluster,
and then delete the deployment there as well.
[MUSIC PLAYING]
And so that will take a moment.
So in the meantime, let's take a look
at previous scaled-down experiments
I ran before this demo in our cluster autoscaler logs.
So even though the cluster autoscaler in GKE
is a fully managed component, it is
nice to have its logs automatically sent
to Cloud Logging for us.
And so in our Logs Explorer here, I have two same queries--
one for each of the GKE clusters in this demo
to give us insights and to scale down events
that happen in each cluster.
So let's jump into the query for the Oregon cluster
with the optimized utilization profile.
And so I won't get bogged down in query syntax.
But you'll see here that we're looking
for logs from the autoscaler for our Oregon cluster.
And its only node pool.
And we're specifically looking for scaled-down events
in this cluster that happened when the ratio of requests
from pods running on a node that was scaled down
to its total allocatable resources was over 50%.
This signals a more aggressive scale
down of nodes that might have less spare allocated
resources but spare resources, nonetheless.
And so we see here in many cases,
cluster autoscaler did make a decision
to scale down these nodes.
And if we dig down into one of the individual scaled-down
events, you can see this information.
So if you click down in here, and we
take a look at the individual node,
we see that the memory request to total allocatable
ratio for this node that was scaled down was at 60%.
Now, keep in mind, cluster autoscaler
will only scale down a node if it
knows that it can schedule the pods running on that node
to another node in the cluster.
But, of course, there's still the trade-off
that if you turn down more nodes,
the workloads on those nodes are evicted
and have to restart elsewhere.
So in this case, we had three that
were on this node that was scaled down.
So this is when choosing an autoscaling profile
becomes a trade-off of efficient resource utilization versus how
tolerant your workloads actually are
of the potential disruption.
And so now, we're going to run the exact same query
in the California GKE cluster with the balanced autoscaling
profile.
We're actually not going to get any scaled-down events
over the same time window, right?
And then the reason that this is occurring
is that more often than with the optimized utilization profile,
the balance profile is preferring
to keep resources readily available for income pods
as opposed to aggressive scale down.
And so now, we'll jump back to our original demo.
And we can see that the expected behavior has occurred.
We can see that the organ GKE cluster
with the optimized utilization cluster autoscaling profile
has scaled down more aggressively
than the California GKE cluster with the balanced cluster
autoscaler autoscaling profile.
And so when it comes to resource efficiency,
scaled-down efficiency is a big part of it.
But there's also optimizations that we
can have at scheduling time to make sure
that of the spare resources on our nodes that we use them--
that we use them all, right?
So when you take a look at the Oregon cluster
with the optimized utilization profile,
you can see that we have two nodes.
And we'll look at the node with the suffix h02a.
And we can see that CPU, it has more CPU
requests than the other node and about, yeah,
it's about a wash for memory requests.
And so when it comes to utilizing
the optimized utilization profile,
let's take a look at how this pod will get scheduled here.
So we have a pod requesting 100 ml core of CPU.
And so what we would expect with the optimized utilization
profile is that this would actually
land on the h02a nodes.
We're going to-- we just applied it.
We created it.
And we should see that it was scheduled to that node.
And so the reason they get scheduled to this node
is that when you use this optimize utilization profile,
that's influencing the kube-scheduler
to place these nodes on more heavily utilized nodes
to make sure, again, that all those spare capacity that
sometimes can be left idle that we're actually taking advantage
of it and using it when we can.
And so it prefers to schedule node pods
to these nodes that, again, are more heavily utilized.
And so the way that they've actually
influences the kube-scheduler is that when you enable
this feature for this autoscaling
profile on a cluster, we will then inject this scheduler
named GKE optimized utilization scheduler
to basically influence a kube-scheduler
to perform this behavior at scheduling time as well.
Scaling down nodes more aggressively
and scheduling workloads so that we get the most from our node
can make things a bit better with resource
efficiency in GKE.
So check out the optimized utilization autoscaling
profile.
As always, you can learn more in the links in the description
below, and we'll see you in a bit.
[MUSIC PLAYING]