Hello Everyone, This is the second part to debugging memory issues
in Azure App Service. Today, we learn about collecting memory dumps
that help us debug memory leaks. I am Puneet Gupta and I am a Senior
Software Engineer with the Azure App Services team.
Every technology or framework provides it own set of tools to debug
memory and in this video we will debug a .net app that's running on dedidcated
Windows SKU and the same methodology applies to .NET Core as well
If you recall the previous video, we identified
that "demohighmemory" was main cause of memory consumption amongst
apps running in the app service plan. Lets see if we can
identify the application code leading to high memory
App Service Diagnostics provides rich set of solutions that help you debug a problem
For this issue, it is recommending that we collect a memory dump
if the issue is happening currently and if the issue is intermittent
it is suggesting us to use auto healing. Since our problem
is happening right now, we can follow the first solution which is to create
a memory dump. You can launch the tool by clicking on this button
over here or you can go to the Diagnose
and Solve Problems blade
, choose diagnostic tools and choose the memory dump
For the first time, the tool asks you to
choose a storage account where the memory dumps will be copied.
I can create a new account or choose an existing storage account
Post that,
we select the instances where the app is running and choose "Collect Memory Dump"
If you choose more than 50% of the instances,
the tool gives you a warning and suggests you to choose fewer
instances. I will just choose one instance and run the tool again
as the issue is happening on both the instances
In a few minutes a memory dump gets generated.
Since I have chosen "Collect and Analyze Data", the generated memory dump is
also analyzed using the DebugDiag tool and an analysis report will be
generated. It can take several minutes for the analysis report
to finish so lets look at another cool way to debug this dump.
Lets download the memory dump and try to analyze
it locally.
After the dump is downloaded,
I will open it in my favorite tool Visual Studio
Visual Studio
shows you some basic information about memory dump like the process name
, architecture, whether the dump has heap information and the OS and CLR version
I will choose "Debug Managed Memory" as we are trying to debug a
memory leak.
Clicking that, Visual Studio shows us statistics
of all the .NET Types loaded in the dump and the memory consumed by
each type and the inclusive memory consumed by that type
Inclusive bytes refer to any
types that are referenced by the current
type. Clearly we can see that we have roughly 700 objects of type ArrayList
and they are taking about 68Meg of memory themselves
The objects contained in the array list are adding up to an inclusive
byte size of 1.2 GB so that's our culprit
right there. If you scroll a little, we can see that one
specific .NET type has individual byte consumption of around
1.1 GB so it seems that this type is somehow related to
arraylist. Lets see if we can draw that correlation.
When we double click on array list
we see that there are only 2 objects of type array list responsible for
1.2 GB of total memory
and we can see that they are rooted inside HttpApplication factory
This is the class that gets invoked if you are storing
data in Application State in ASP.NET
Lets check the referenced objects by this type because we know
it is the objects that are referred by the array list object that are taking
1.2 GB of memory. So we
can keep expanding the reference tree till we come to a point
where inclusive bytes drop to a small number. We see
changed immediately to 3 MB from
1.2 GB. Lets expand and see a few more
objects. What we see in common is that each
object have a byte array thats roughly 3 MB in size
and the fact that we have too many objects of this type
is what is causing the leak. All of them are
are part of an array and that array is rooted in Application State because
of which these objects will not be cleared by the garbage collector.
and that's our problem right there. So in a matter of few minutes
we are able to identify where most of the managed
memory is coming from. Isn't that easy ?
By this time, even our memory dump anlaysis should have finished
Lets download the report and see what it has to offer
The very first thing
it shows that we have 1 GB of byte arrays (something that we already know)
and we have found a GC Root chain for them.
Let's walk through the report a bit more.
Since this is a memory investigation, I am going to minimize Crash Hang Analysis
and move to DotNetMemory Analysis. First thing
it contains is the GC Root chain and the full hierarchy of byte array
and where it is rooted.
Debug Diag
report also has other important information like .NET GC heap info
which tells you the total amount of memory in the dump in the managed heap
and this helps to identify if the leak is managed or native
Since we see 1.14 GB
in GC Heap, we can safely conclude that this is a managed leak
Next, we see top
40 most consuming .NET types, the amount of memory allocated by those types
and the count of objects of each type.
The report also has a section for objects on the large object heap.
These are objects more than 85K in size
and they need extra garbage collection cycles to clean up
so they deserve some special attention
Now this was a simpler example but I want to mention
this approach works for managed code (Which means .NET and .NET Core)
, Also memory leaks can surface in different
ways so the complexity varies from scenario to scenario
It is possible that more that one tool may be required
to debug the exact leak pattern.
So we learnt how we can collect data when the issue is happening
at that time but what about intermittent issues
You don't want to be awake just to wait for the issue to happen so that you can click
a button? Definitely Not ! Azure App Services
got your covered. You can leverage the Auto-Healing functionality
of Azure App Service to not only diagnose but also mitigate
issues when certain triggers are met.
To do that, on the Diagnose and Solve Blade,
choose Diagnostic Tools,
and then goto Autohealing.
I will enable AutoHealing for my app
, choose memory limit
and then, configure a private bytes rule.
Here, I can specify the memory threshold on which I
want to initiate an action.
Then, under action I will choose custom action and set the action to
Memory Dump. To ensure that my app is not impacted
once memory threshold is crossed, I will choose to kill the process
to mitigate the issue. This rule will now collect a memory
dump and kill the process when the threshold is reached and a new process will
be automatically spun up. Also, in the backround the dump will be analyzed
I can ensure that the rule is not kicked in immediately
after the process is just started by modifying the startup time.
The final rule configuration is then summarized.
Saving these settings will
cause an App Domain restart so you are advised to perform these
changes during non business hours.
Then you hit Save and you are all set. Whenever the right thresholds are met,
memory dumps will be captured and the process will be restarted.
You can view existing sessions by clicking on the View Sessions link
Using this feature, you can very easily diagnose
and mitigate any intermittent memory issues within your app.
So we hope you found this video useful and
you learnt something new that will help you debug memory issues in your production apps.
easily. Please like this video, tweet it, share it with your friends
and subscribe to the Microsoft Azure Channel on YouTube.
HAPPY DEBUGGING !!!