So, the other advisor recommendation that I'd thought I'd
drill in on is around unsupported Kubernetes versions.
So, I'm going to talk a little bit
about the cadence of the community project itself,
how that's being reflected in AKS,
how we consume new releases into
the service and why it's important for you to stay up-to-date.
Then I'll talk a little bit about the upgrade process as
well as an alternative best practice that we've seen
a lot of customers follow around using node pools for consuming
new Kubernetes versions and some
of the work that we're doing and capabilities that you can
take advantage of, if you're looking to follow that best practice.
So, the Kubernetes project is fast-moving.
It is shipping new minor releases
about every three months on a fairly consistent cadence.
The most recent release was 1.18 at the end of March.
1.19 is scheduled to be released at the end of June.
So those are minor releases.
Patch releases are then shipped as required,
for bug fixes and security patches.
This is 1.18.1.2 and so on.
In the upstream project, only the latest release and
the last two minor releases are supported.
So, at any given time you've got
roughly up to the last nine months of releases being supported.
What this means is that,
if you're running a version that is older than that,
you may be running one that has critical security vulnerabilities,
certainly...potentially performance issues and other bugs,
but security vulnerability is obviously the most important here.
And what we've seen is with the growth of Kubernetes,
it has become a greater target.
There's been a notable increase in the number of CVEs that
have been highlighted over the last year or so.
And so, it is that much more critical that you're staying
up to speed with Kubernetes releases as they come out.
So, in AKS, we try to stay up to speed
with the upstream project as much as possible.
So, we have mapped our support matrix to that upstream process.
So, we also support three minor releases at any given time.
The way this works in terms of process is you'll
see a new minor release or new minor version released upstream.
Typically, that'll be incorporated into AKS within
a matter of weeks as a preview, and at that point,
we will announce the pending retirement of
the oldest minor release after 30 days.
You have 30 days notice if you're on
that minor release to move forward before you fall out of support.
After that 30-day period,
we graduate the new minor release to a G8 state,
and at that point drop the oldest one off the back.
Now, it's important to point out that
clusters that are running those older versions don't stop working.
The applications that are still running
within them remain available.
However, you do have
potential vulnerabilities that are running in there.
You're not going to be able to consume
new improvements that are shipped,
and if you do create a support issue,
that's something where customer support will typically
ask you to upgrade to get onto
a supported release as there's obviously a lot of
changes that go into these releases every three months.
So, how do I actually consume those upgrades?
Well, as a managed service,
AKS provides a first-class upgrade capability as you
have the ability to upgrade the cluster either as a whole.
So both the control plane and all of the associated node pools,
or you can do that individually.
So you can invoke a control plane
only upgrade and then work your way through the node pools,
so you have full control over those versions.
With the only caveat being that
the control plane always needs to be
at the greatest version within the cluster.
So the node pools can be behind,
but they cannot be ahead.
Now, of course, it's the node pool upgrades
that are the most interesting
and that's where we get into the value
of something like a Pod Disruption Budget.
So, the way that those work is we initially will
add a new node into the node pool with the target version.
So if you're moving from say 1.16 to 1.17,
the new node will be based on 1.17.
Then we will pick one of the older nodes
and perform a cordoned drain operation
similar to the one that we just saw,
where we'll try to drain all the existing workloads off of
that node and move them elsewhere
within the node pool until that node is completely empty,
at which point it will be removed.
A new node will be added and then
that process continues until all of the nodes have
been moved forward and
replaced with ones that are running on the new version.
Now obviously, that fully automated upgrade process
is very convenient.
For a lot of scenarios it works beautifully,
but there are certain use cases where it has drawbacks. So,
first of all, for very large clusters it can be slow.
So, as we're following a node by
node upgrade process and
requesting that each of those pods gracefully terminate,
that is something that can take a certain amount of time,
and then at the very least is out of your control.
So you don't have the ability to manage
the creation of the new nodes
or the ordering of draining of the old ones,
and so if that's something where you want to have a little bit of
more oversight as part of the upgrade process,
the built-in upgrade experience doesn't allow you to do that.
So what we've seen is a lot of customers end up
following an alternative approach using node pools
where rather than running that node by node upgrade,
they will actually create an entirely new node pool
targeting the new Kubernetes version.
So again, maybe a new node pool targeting version 1.17,
and then they will drain
the nodes themselves on the old node pool,
which will force the rescheduling onto the new nodes,
and then once all the old nodes
have been drained; delete the old node pool.
So, at a high level, it is effectively following
the same process that built-in experience does,
but you're able to create all of
the new nodes as a single operation,
and then control the ordering and
the timing of draining of the old ones.
So you can drain a handful of nodes,
get the pods rescheduled onto the new nodes,
make sure that everything's running correctly,
and then continue with the process as opposed to having
completely outsourced that to the service.
So with that in mind,
we've been making a bunch of improvements
to our node pool support.
So the first one is...we've introduced the notion of node pool types.
So the background here is,
you have a set of system services that need to run on your node.
So this is things like CoreDNS and it's Autoscaler Metric Server,
the Tunnel, and others.
And those are services that are critical to the health
of your cluster and so they need to be kept running,
they need to have dedicated resources,
and so it's typically a best practice
to isolate those components into
their own nodes so that you can then
run your applications completely separately from them.
And so we've now elevated that idea into a first-class concept,
where you can signal that
a particular node pool is
a system node pool and that's
the default one that gets created with the cluster,
and then subsequent node pools can be
created as user node pools to run your application.
So, it creates a nice way of
segmenting the different workloads
that are running within the cluster.
Within user node pools,
you can use those to run either Linux or Windows pods.
They can be made up of either Spot instances or on-demand.
We'll talk about Spot here in a second and then critically,
they can actually scale down to zero nodes.
So, while the cluster to remain available,
those system services that I mentioned need to
continue to run your application if
it doesn't have any work
to do and can actually scale down to nothing,
you can likewise scale the associated infrastructure down to zero.
And so, you can really manage that set of
resources associated with your user node pools.
We've also added the notion of Spot node pools and so this
is built on top of the Azure Spot capability
that became generally available recently.
This is an evolution of
the low-priority capability that we had previously within VMSS.
And so, the idea here is to allow you to get
unused Azure capacity at a pretty steep discount,
so up to 80 percent in some cases
while being mindful of the fact that
that capacity is not guaranteed.
So, if it needs to be reclaimed
for other customers who are willing to pay full price,
then your nodes will get evicted.
But for certain workloads,
namely things that are
maybe lower priority and certainly things that
are very tolerant of interruption, so batch jobs,
Machine Learning type workloads,
this can be a very nice way to get a significant discount
on your compute costs
while still running in an AKS environment.
We recommend that you associate the Cluster Autoscaler
with a Spot node pool such that if a node does get
evicted based on capacity constraints that the Azure fabric
can attempt to redeploy it elsewhere within the environment,
which will show up as just a new node within your node pool.
The Autoscaler can manage that process for you.
So, just a couple of caveats here.
One is that Spot node pools cannot be upgraded,
so because of the way that we run
that upgrade process and the need to provision
new nodes because we can't guarantee that those new nodes will
be provisioned in a timely fashion,
we won't allow you to start an upgrade process.
So you should think of these Spot node pools as
being effectively tied to the version that they are created with,
which then ties back into that best practice around
performing upgrades via a new node pool creation,
and then they cannot be used as system node pools.
So obviously, we want to have dedicated resources to
run those system services and
so Spot node pools should be really
reserved for user applications.
With that, I will say thank you
for attending this virtual session.