ANTHONY BUSHONG: Hi, I'm Anthony Bushong,
developer advocate at Google.
And welcome to GKE Essentials.
[MUSIC PLAYING]
Did you know that GKE clusters can scale up to 15,000 nodes?
Owning a cluster with that sheer scale is hard for me
to comprehend at times.
But let's dive into it together.
This video is the first part of a two-part introduction
to building large-scale GKE clusters.
Over the course of these videos, we'll
talk at a high level around two key areas--
one, why you might consider building clusters
with thousands of nodes, and two,
what considerations you should account
for when designing clusters to be able to scale.
Kubernetes especially shines for those
of you looking to build a platform to run your company's
workloads.
It gives you a means for abstracting away
infrastructure.
No need to directly deploy workloads
to a specific machine.
Rather, you deploy it to the cluster,
and Kubernetes will make sure it runs somewhere.
It also gives you declarative APIs
that codify a workload's characteristics
and enforces that behavior.
You can even extend these APIs with your own definitions
of how workloads should run.
This is why one of the founders of Kubernetes
went as far as calling the open-source technology
a platform for building platforms.
Now, these platforms that you build
could be home to a set of large, resource-intensive workloads,
such as batch processing or machine learning.
Or they could also be the generalized way in which you
enable developers to access infrastructure
for their various and potentially differing
workloads.
It should be noted that when you're
building multi-tenant clusters, the harder
isolation or stronger boundaries that you require
between these workloads, the more likely
it is you should consider using multiple smaller clusters.
Nevertheless, using fewer large-scale clusters
in some of these cases can help in a handful of ways.
One, they can help you minimize the number of clusters
you have to actively manage.
This helps with problems like configuration drift
across clusters or juggling multiple concurrent upgrade
cycles.
Two, you can more easily handle resource spikes.
If you're building a platform, you
may have many different teams as users,
each with their own plans for rolling out
their own workloads to the platform
or scaling up their workloads.
Building a cluster that can scale to larger sizes
can help absorb these varying demands concurrently.
And finally, for those of you building specialized platforms
for batch processing or machine learning,
building a large-scale cluster brings the simplicity
of the Kubernetes API to these complex,
performance-intensive workloads.
Even if you're not looking to build a cluster that's
15,000 nodes, you can still realize these benefits
by building a cluster that can scale to house
large amounts of resources.
So let's talk about considerations
for designing and building a large-scale cluster like this.
When it comes down to it, as a platform builder,
you are providing a contract to your end
users and their workloads.
You ask them to package their workload
in a container with assurances that your platform will
run that workload reliably.
So at the heart of building large-scale clusters
is not just the concept of adding more nodes or more pods,
but rather, figuring out how to maintain your cluster
reliability when doing so.
Of course, we can't cover every single design consideration
for large-scale clusters in this episode.
But let's dive into where you should start thinking.
It's important to think of cluster design
across two categories.
There are the considerations at the greater Kubernetes
level, which revolves around the Kubernetes API
and their objects, such as pods, services, namespaces.
And these are not specific or limited
to any one infrastructure provider.
Then there are considerations at the GKE level, which
include Google Cloud resources, such as VPCs,
underlying compute, storage, and GKE specific features
that help tune your cluster to reach these larger sizes.
We'll actually cover the Google Cloud and GKE parts
in part two of this series, so let's cover the Kubernetes
resources first.
Kubernetes scalability is defined
by the special-interest group focused on scalability
as a multidimensional envelope.
Within this envelope, your cluster
will work and do so reliably.
This envelope has configuration limitations
at various dimensions.
It could be the number of pods, the number of namespaces,
the number of secrets.
So when designing a cluster for large scale,
it helps to understand and start thinking
about how your cluster will fit within this envelope.
Now, I definitely don't work with multidimensional envelopes
on a day-to-day basis, so let's actually
talk a little bit about how an envelope like this works.
In the envelope of Kubernetes scalability,
dimensions are not always independent of one another.
An example.
In Kubernetes, we have the ability
to run 110 pods per node.
We're also able to run thousands of nodes in a given cluster.
But to do both at the same time?
This likely would result in an unreliable
cluster, as you are stretching multiple dimensions too far--
pods per node, nodes, and total pods in the cluster.
In a scenario like this, knowing how these dimensions relate
to one another can help you make adjustments,
like changing the maximum number of pods per node
in your cluster.
In GKE, for these large-scale clusters,
we generally recommend considering fewer pods per node
than the maximum.
A good starting benchmark we've seen with successful users
is roughly 30 pods per node.
The next thing to consider is, scaling
to the maximum in any one direction
can taper your ability to scale across the other dimension.
So it's important to think about what resources your platform
could push to the limit.
A highly specialized cluster to serve
as a batch-processing platform, for example,
might have numerous pods being created and deleted
at the same time, depending on the status of jobs
that are running.
And this will affect pod churn.
A multi-tenant enterprise platform
might have hundreds of workloads across hundreds of teams,
each isolated to their own set of namespaces.
It's important to plan around what
is most critical to make your platform on Kubernetes
work for your users and to test those dimensions one at a time
and independently.
Finally, all the resources your platform uses in Kubernetes
share an upper bound, which is set by the underlying state
system for Kubernetes, etcd.
This not only includes the Kubernetes objects
you're probably familiar with-- deployments, secrets,
stateful sets, et cetera--
but also the resources that are created using custom resource
definitions.
In fact, many open-source technologies
built atop Kubernetes utilize this
to persist their own objects through the Kubernetes control
plane that their own set of controllers interact with.
All of these objects are subject to etcd's underlying
limitations, such as 800 mebibytes per object type
and the overall total storage of etcd.
Given this, we recommend understanding
the implications of what is actually getting
created in your cluster.
For example, the built-in Kubernetes job resource
requires certain configurations to ensure that resources
are cleaned up upon completion.
Custom resources can also expand one resource
into several others, such as pipeline runs Tekton.
These are not bad things.
If you need to run Kubernetes jobs or custom resources,
like what Tekton provides, great.
But it's important to know their behavior when building
a platform for your end users.
Now that we've taken a high-level walkthrough
of Kubernetes scalability, we'll need
to cover how to design around GKE features and Google Cloud
infrastructure to support these large-scale clusters, which
we'll cover in part two.
In the meantime, take a look at these resources
to better understand the Kubernetes
scalability dimensions that we discussed today.
Thank you for your time, and we will see you
on the next episode, part two in GKE Essentials.