>> Hello and welcome to this on-demand session.
My name is Sean McKenna and I'm
the Lead PM for the Azure Kubernetes Service.
There's always going to be cases where
a recommendation is missed or something goes wrong,
and now you're in a state
where there's a problem with your cluster,
there's a problem with the applications
that are running inside of it.
And so we're trying to take a lot of
those recommendations and best practices and apply them to
doing diagnosis on your cluster as
well through an experience that we call AKS Diagnostics.
And so the idea here is to provide a set of
self-served troubleshooting tools built into the Azure Portal.
Again, much like advisor built on top
of telemetry that we have about your cluster,
combined with the experience that we have from
working with literally thousands and thousands of
customers and the associated support cases
that those customers have created.
So, what we found is,
many of the support cases that customers
create are ultimately root cause to something
where they didn't configure their cluster
correctly or they didn't configure their applications correctly,
and so we can actually
identify those issues in advance and allow you
to take the appropriate action to avoid
having to create a support case in the first place and to get
your cluster back into a good working state.
So a few of the examples that you'll see in
that diagnostics experience today are around pod evictions,
very restrictive firewall rules,
those are a common source
of issues in terms of how the cluster operates,
issues with DNS resolution, subnet IP exhaustion,
there is a whole series of additional checks
that we perform against the cluster.
But I want to dig into one specifically because this
has been a very common source of issues for customers,
and something that, again,
much like pod disruption budgets is not something that is
obvious as you are learning Kubernetes for the first time,
but it's something that's really critical to understand in order
to ensure that your cluster operates correctly,
that your applications stay available.
That is the pod evictions or OOM
Kill recommendation or troubleshooting step.
Now, the critical background here is to
understand how the Kubernetes scheduler works.
So, when you provide a deployment and the associated pod specs,
the scheduler is going to try to find
a place to run those pods within the cluster.
But it's going to do that based not on
the unused capacity at
the time that you're making that deployment request,
so not in terms of what's actually being consumed,
as you might view within a monitoring tool, for example.
But based on what is unallocated within
the cluster from the scheduler's perspective.
So, what is unreserved in terms of capacity.
When you deploy a pod
and its associated containers into Kubernetes,
you have the opportunity to set what's called the resource request.
That allows you to reserve a set of
capacity in terms of memory in CPU and
potentially other resources for that
application to be able to run correctly.
But that's an optional field,
and so if you don't set it,
then the scheduler is going to attempt
to schedule more and more pods
onto a given node because it doesn't
actually know whether that pod
requires 100 megabytes of RAM or 500 megabytes or a gigabyte.
So it's just going to continue to schedule things there,
and then ultimately, as those applications
attempt to run and consume more memory,
they will hit the edges of what that node can
actually manage to run
and so the system will then attempt to start shutting down
pods to reclaim resources
and ensure that the node can stay healthy.
Now, there are very good use cases for that,
you may want to be running some low-priority pods where
if there is capacity available that
they should be able to consume it.
But in a lot of cases,
you actually want those pods to be running continually,
you want them to be remaining available,
and so when they're getting killed for lack of resources,
that is impacting the availability of your overall application.
And so if we detect that we are seeing a significant number of
OOM Kill operations and evictions of pods within the cluster,
that is something that we're going to surface
through the troubleshooting and diagnostics tool.
So, just to show how some of those diagnostic recommendations
are getting surfaced,
let's take a look again here in the portal,
and specifically this time we're going to drill in
on the "Diagnose and solve problems" menu
item and that's going to surface a couple of different buckets of
potential issues and you can see
some descriptions here of the types of issues
that I might be able to
diagnose based on these two different buckets.
So in this case, I'm going to drill in on "Cluster Insights".
You're going to see that it runs a series
of checks against my cluster, and again,
this is based on the telemetry that we
have about the operations of your cluster,
and so you can see a bunch of the things that have passed
here. As well as in a moment,
we'll see some of the failures and
warnings that I need to be concerned about.
And so you can see that OOMKilled Pods that we were just discussing,
and I can drill in here on more info and
get a whole bunch of detail about when this happened,
the image that was running,
the specific pods, namespace that it
might have been running, and number of restarts and so on,
and so that gives me a bunch of
diagnostic information that I can then go and connect
to my logs and troubleshoot directly against the cluster.
I'm also getting a set of pointers here out to documentation,
much as we saw with the advisor
and security center recommendations.
Interestingly, in this case,
we're actually surfacing this
particular diagnostic in two different locations,
so the set of pods being OOMKilled,
so obviously from a pod lens,
but then also in terms of the node resources.
So in this case, there's this one particular node that
is seeing significant memory pressure,
again, as a result of that same deployment
that was not setting a resource request.
So that gives me another lens into
the problem and how I may go about solving it, and in fact,
there's a couple of different recommendations here
for what I can do next as a potential action.
So again, a great way for
best practices to get surfaced in a context of a problem that you
might be seeing in the cluster and a way for you to learn how you
can apply some of
those recommendations to both get out of
the current situation, as well
as ensure that you don't get into it again.
With that, I will say thank you
for attending this virtual session.