>> Hey, friends. AKS Diagnostics
is an intelligent troubleshooting experience
that helps you identify and resolve problems in your AKS clusters.
Yun Jung Choi is here to show me how
to use it to troubleshoot cluster issues,
including how to use AKS Periscope today on Azure Friday.
[MUSIC].
>> Hey, friends. I'm Scott Hanselman,
and it's another episode of Azure Friday.
I'm here with Yun Jung Choi, how are you?
>> Doing good. How are you Scott?
>> I am having a blast time learning
all kinds of wonderful things about the Cloud
on Azure Friday and you've brought me
some fun things to talk about with AKS.
>> Today I'll be showing you how to diagnose and solve problems on
your AKS cluster using AKS Diagnostics and AKS Periscope.
>> Very cool.
>> AKS being a managed Kubernetes service,
handles a lot of the critical tasks like
health monitoring and maintenance for you.
However, once in a while you might run into
some problems and it could get really frustrating,
especially when you're running mission critical workloads.
That's why we've created AKS Diagnostics.
AKS Diagnostics is
an intelligent and interactive self-serve diagnostics experience
you can use to self troubleshoot issues on your AKS clusters.
For today's demo, I'll be using
my AKS cluster called AKS Workshop.
I've been recently having some issues with
upgrade operations on this resource,
so I'm going to try and troubleshoot
that issue using the AKS Diagnostics.
I'll click on to Diagnose and solve problems
which will open AKS Diagnostics.
Upon clicking on diagnose and solve problems,
I can immediately see there are
a couple options that I can choose from.
One's cluster insights, the other one's networking.
Since my issue is more related to the cluster operations,
I'll choose cluster insights.
When I choose cluster insights,
I can immediately see that several different checks are run
automatically on the back-end telemetry of my AKS cluster.
While that's loading, let checkout
one of the first insight that populated.
I'm going to click on More Info and expand.
It says that my AKS Kubernetes version does not meet the minimum
required which is part of
the reason why I wanted to upgrade my cluster,
so this is a very good insight and it also points me to
the recommended actions on how to update my cluster.
I'm going to go back to the observation and
see if there are other observations.
It's loading, but it's okay,
I'll check up the other insights.
Let me check on the second one this time.
It says that a Kubelet bug is detected,
meaning that I'm currently using an outdated version of a cluster,
so I'm susceptible to this Kubelet bug.
The recommended action again,
is to update my cluster to the newer version which I'll be doing.
Let's see. Let me
click on to the identity and security management.
This one seems pretty interesting.
As I read through the summary,
I can see there is a 401 Authentication error detected on
my AKS cluster indicating that
some incorrect credentials or expired secret keys are detected.
It even says that this could actually
lead to operations failing on my cluster,
including the cluster upgrades.
I think this is the issue that I was experiencing earlier.
Below it actually points out
the error details as well as some apparent recommended actions.
It says I can just update the service principle on
my AKS cluster and this should resolve the issue,
and even points to the documentation.
I click on the link.
Yeah, there's documentation details about the steps on
how to update your service principle to fix this issue.
With just a few clicks, I was able to find out
the solution for the upgrade operation failures I was having,
and then get the recommended action,
and the clear instructions.
Just to tell you what's happening behind
the scenes is that we are actually analyzing
different pieces of the back-end telemetry
to find issues you're having on your AKS cluster,
and mapping those to recommended
actions that we know of that will fix the problem,
so that troubleshooting routine is made easy for you.
Let me head back to the observations because I want to
see what these tool has to offer for me.
>> Can I ask one question about the update check though?
>> Sure.
>> Forgive my ignorance of Kubernetes, I'm still learning.
I noticed that it said that it
wanted the cluster version to be a certain thing.
But then I know that AKS is a managed service.
Help me understand, aren't version upgrades like that automatic?
What are merged versions that I manage and what
are versions that are managed by AKS?
>> That's a great question.
Let me just expand up on
the Kubernetes version check for your context.
Basically, what you would have to update on your end would
be the Kubernetes version on your cluster.
Basically those get communicated
over via our managed AKS GitHub repo.
Basically, we have these very thorough release notes
indicating that there are
certain times when you would have to upgrade to meet
the minimum supported version for your AKS cluster.
The reason for that is because Kubernetes is always moving,
and always supporting newer versions which,
as you've noticed in the other insight,
have some certain fixes that will
address issues in the versions themselves.
It's always good to keep your versions updated.
>> I see. Then if
it's a moving target as you say, it updates a lot.
So that's an urgent warning
to start to pay more attention to my versions there?
>> Definitely.
>> That makes sense. Thank you.
>> Let me go back to the observations and check out some
of the orange insights that I see over here.
There's one indicating that I'm using
the Basic SKU Load Balancer which has certain limitations.
If I expand on it,
I can actually learn more about what those limitations are,
which are the larger back-end pool,
and then the multiple node pools,
and then availability zones are not
supported for Basic Load Balancer.
Cool. This is great information.
Let me go back.
I have one last insight that I haven't
checked out yet which I think will be super interesting.
Basically it's pointing out that there is
some IO latency detected on the AKS cluster and if I
click on "More Info," I can actually
see in one of the insights that
one of my node was found to be having some IO throttling issue.
I know for a fact this is bad because IO throttling issues could
undermine the performance of an AKS cluster and even worse,
sometimes it could cause loss of availability.
So this is definitely something I want to take a look at.
It gives me the throttled VM instance,
as well as the root cause,
which is the resource starvation and some recommended
actions on how to configure monitoring for OS saturation issues.
So I can keep a close eye on these types of issues.
>> It's amazing how much detail they offer.
I'm really impressed every time I go to
the diagnostics and solve problems.
Frankly, when I started out,
I thought it'd be very simple.
It'd be like, this thing stopped or that thing is broken.
But as you scroll, it's prose,
it's links, it's suggestions,
and it's charts, it's really impressive.
>> Yeah. Then also as you've pointed out,
there's a nice graph showing you when
the spike in OS disc queue depth happened.
So you can see when the IO throttling event happened.
>> That's so cool.
>> Let me go back.
Also, if I scroll below,
I can also see
the successful checks that were run on my AKS cluster.
These are all the checks that were run on
my AKS cluster that have succeeded.
So I don't need to pay as much attention to these because I
already have quite a handful
of issues that I need to address here.
So yeah, AKS diagnose is a really easy way for you to
interactively and intelligently diagnose
issues on your AKS clusters.
With that, let me just jump over to my second demo,
which is on AKS Periscope.
For those of you who are not familiar with AKS Periscope,
which is available in CLI and also in VSCode.
AKS Periscope is a tool you can use to collect
node and pod logs on your AKS cluster.
Here, I have the Marketplace page pulled up for
the Visual Studio extension you
can download to leverage this tool.
Here you can find the installation instructions on how
to install VSCode extension onto your VSCode environment,
which is quite similar to how you would
normally install these extensions.
It's quite easy. Let me now switch over to my VSCode,
which I have already installed the VSCode extension on.
Once I've done that,
I can actually see that there's a Kubernetes icon in the VSCode.
I'll click on that and then in the bottom,
you'll see a section called Clouds.
This will open up the Cloud Explorer and if you have
your Azure account connected to this Cloud Explorer,
you'll be able to see
all the subscriptions that you have on your account.
For this demo, I'll actually run AKS Periscope on one
of the AKS resource I have called AKS Periscope.
I'm going to right-click on it,
then I just simply hit "Run AKS
Periscope" and immediately what's happening in
the background is a DaemonSet is deployed onto the Azure nodes of
my AKS cluster and collecting
all sorts of information from the node and the pod.
Just to name a few of the logs that you would see.
You would actually see
Docker and Cumulus system logs, Container logs,
and all low level logs,
such as Cluster provisioning log and Cloud-init log.
The tool's running, but once these logs are collected,
basically what will happen is these logs get stored
onto your preferred Azure Blob Storage account
that you have associated with the cluster.
It looks like the tool has finished running now
and you can see there's a big blue button called "Generate Link."
What that will do is it'll actually generate links to
your logs and also generate shareable links to the log.
So you can share those links with your colleagues
or even CSS engineers if you're working on a case with them.
I'm going to click on the "Generate Link" button, and there we go.
Right away you can see
these downloadable links and shareable links.
This is displayed per node and
I'll click onto one of the downloadable link.
I'll click "Open."
Once the zip file is downloaded,
you can simply click "Open File" and just click through
a few folders until you land onto a folder titled collector.
When you click on to this folder,
you'll be able to see all sorts of
different logs that you've collected using the AKS Periscope tool.
Now let me go back to my Visual Studio.
Also one more thing I want to point out,
as I mentioned, there's the seven-day shareable link.
You can simply just click on this icon to
copy the link and share this with anyone and you
don't even have to worry about it too
much because it'll just expire in seven days.
So no need to worry about leaking secrets and everything.
AKS Periscope is a really simple way
for you to collect logs on your AKS cluster.
Definitely try this out.
>> That's very cool. Where can
people go and learn more about this?
>> Let me actually pull up a few slides that I have ready for you.
Just to recap a little bit on what I demo today,
AKS Diagnostics is a built-in self-served diagnostics experience
available in the portal for you to use to
troubleshoot issues on your AKS cluster.
You can get recommendations for
next steps and potential solutions,
and you can use the red, orange,
and green status icons to easily
navigate between different insights.
Below we have the link to AKS Diagnostics dock.
So check it out to learn more,
and next, AKS Periscope.
AKS Periscope is a really great tool for
you to use when collecting node and pod logs
on your AKS cluster in the comfort of your favorite IDE, VSCode.
The links are below.
When you run into issues with your AKS cluster,
next time try out AKS Diagnostics and AKS Periscope,
it'll make your life a lot easier.
>> All right, I am learning all about how to
diagnose my AKS clusters
and about the power of AKS Periscope today on Azure Friday.
Hey, thanks for watching this episode of Azure Friday.
Now I need you to like it, comment on it,
tell your friends, retweet it,
watch more Azure Friday.
[MUSIC]