Welcome Everyone! Today we learn about the Proactive CPU monitoring
feature of Azure App Service platform to debug an app that is consuming
high CPU. I am Puneet Gupta and I work on Azure App Service Diagnostics
Let's jump right in. This approach applies to apps running
on the deticated SKU on the Windows stack and can be used for Apps
using any technology if the underlying technology stack is publishing
debug symbols. Recall from the previous video that we identfied
that DemoHighCPU was consuming maximum CPU resources
The tool is giving us suggestions on the next troubleshooting steps to apply
and it advising us to enable Proactive CPU monitoring.
Before enabling it, I wanted to give you all a quick overview of
the feature. When apps consume High CPU intermittently
on one or more instances, it's incredibly hard to debug the issue
because you don't the know the time or the instances where the issue may happen
But that problem is a thing of past. Our new innovate feature
collects the right data at the right time on the right instance
without any user intervention or effort. This feature
keeps a close watch on all the processes that are consuming high CPU
and whenever it detects any process crossing the configured threshold,
it collects the right data and mitigates the issue by killing the bad process
thereby recovering the High CPU situation. With a single click
enablement, you can move all the complexity from your shoulders over to
this feature. This feature makes CPU investigation so
simple that anyone can debug high CPU issues easily
using the approach shown in this video. Let's see how!
You enable this tool by clicking on this button or you can go to the
home page of the Diagnose and Solve Problems and click on Diagnostic tools
and click on Proactive CPU monitoring. When you run it for the first time
it asks you create or choose a storage account. This is where
the data collected via this tool is stored.
Enabling the tool, additional options appear, that let you configure the tool as per your
scenario. We have chosen the default configuration wisely
but you can change these if needed. As per the defaults, when any process
under the site takes more than 75% of overall CPU
for more than 30 seconds, a dump will be triggered and
a maximum of two dumps will be collected. The rule will be evaulated
every 15 seconds. Once any process
exceeds this condition, the data is collected, the bad process is terminated and the new
one starts automatically. Enabling monitor webjobs option
allows monitoring webjobs as well. You can view the summary
of the rule configured and if everything looks fine, enable the monitoring.
Once the monitroing is enabled, you can see the status of the rule and
the data collected. Shortly, instance level logs appear
that let you view the CPU monitoring activity. This data can
take some time to populate so we are fast-forwarding this video to show the points of
interest. The logs show the CPU consumption of each
process and if it is crossing the threshold. As we
can see, the first instance already cross the threshold a dump has bee
generated. This dump is getting copied to the storage account a
bad process is getting terminated. After some time, the
other instance also encounters high CPU situation and dumps are collected and
the process is killed. This way, the tool captures the data automatically when
the right condition is satisfied. The data shows up under Dumps
and Reports section. After the limit of maximum action is hit,
the rule stops and gives an option to analyze the data. Click
on the Analyze button to analyze the dumps. You can also download the
dumps and run the analysis on your local machine using Debug Diagnostic Tool.
Another option to analyze the memory dump is by using
our good old trustworthy friend - Visual Studio.
You can right click the dump and choose Open with - Visual Studio.
Once the dump is loaded
in Visual Studio, you see basic information about the dump like the process nam
and process bit-ness. When you scroll to the right, you have
options of debugging managed code or native code. Ours in an ASP.NET
application so I will select Debug with Managed Only.
Visual Studio tries to construct the callstack of the first thread in the process
Since the first thread is not managed, we see the message
indicating no managed code is running. But need not worry,
there is a still a lot you can do with this memory dump.
You can see all the threads in the process by going to the Debug Menu
choose Windows and choose Parallel Stacks. This option
categorizes all managed threads and we see there are 7 threads with similar call stacks
We see these threads are trying to match some regular expression.
The tree view above shows the different state
where some threads have progressed starting from this node.
If we double click a stack frame, Visual Studio tries to lookup the source for
that function. I wanted to simulate a scenario where you may not have debug
symbols stored anywhere and that's why you see this warning indicating that we
cannot find the source code. But need not worry, you can
click on Decompile source code and that's where the magic happens.
Visual Studio decompiles the source code from the dump and is showing you the C#
code where the threads are stuck. So the culprit
was hiding behind this one line of code and it was so easy to find
For Application Developers, this information is like a gold mine
because now they know what exactly needs to be fixed
Lets go back to the Proactive CPU monitoring page and by now the dump
analysis should have finished. When you launch the report,
the tool tries to identify patterns of known issues. In our
case, it identified that some threads are using regular expressions.
Clicking on the thread numbers shows us the call stack
Now this does look familiar, doesn't it ?
We can also see the strings being matched and the regular expression used in the code along with
the call stack. At times Debug Diag
may not identify the exact issue right on the top. You can still
navigate to the HttpContext Report and look at the long running requests
Clicking on the Thread ID next to the long running request
takes you to the thread where you can view the call stack and the thread activity
This is how you identify threads that are processing active requests
and see where they are stuck. So to summarize, Proactive
CPU monitoring feature allows you monitor all processes
belonging to your site and it not only mitgates the issue but also
collects data to identify the root cause of the problem. Thank you for
watching !