Welcome Everyone!
Today we will use the Crash Monitoring feature in Azure App Service Diagnostics to debug an app that is experiencing application crashes.
My name is Yun Jung Choi, and I am a program manager working on self- diagnostics experience for Azure services.
This video approach applies to apps running on dedicated SKU on the Windows stack
and can be used for apps that are running IIS worker processes.
Before we go into details on how to identify an application crash and configure a tool to collect rich information for troubleshooting,
let’s first learn about why these crashes happen and their impact on the workloads.
A crash happens when an exception in your code goes un-handled and reaches System Default handler which terminates the process.
These unhandled exceptions are also known as second chance exceptions.
Unlike first chance exceptions, these exceptions are NOT caught by app code or ASP.NET runtime.
Therefore, when your application crashes,
all the in-flight requests are aborted and an end user may experience an HTTP 502 error for those requests.
Now that we’ve learned about crashes and their impact, let’s start troubleshooting them. Shall we?
First, let’s identify if your app is experiencing crashes
You can do this by navigate to your App Service in the Azure Portal, and clicking on Diagnose and solve problems which opens App Service Diagnostics.
Here you can see various categories and checks you can run on your app to diagnose and solve problems with your app.
As we are trying to identify an application crash, I’m going to click on Availability and Performance.
Upon clicking on the category, a health check is run on my app and some key diagnostics are highlighted in the overview.
We can see the apps failed request rate is in red. This warrants an investigation and I’m going to click on it.
Upon clicking, more checks are performed on the backend telemetry of my app.
I can see that my app is showing a dip in availability whereas the App Service platform availability is at 100%.
Immiediatly I can rull out platfrom issues.
Also, as I scroll below, I can see that there is an Application crash detected along with an exception code as well as a 502 error, which often happen when processes are terminated.
I can either click on more info or click on Application Crashes in the left navigation for more details.
As I look at more details, n the next steps, it prompts me to enable Crash Monitoring with clear instruction.
Also, as I scroll below, I can see that there is an unhandled exception detected as well as some application event logs.
At times, you may be able to identify the root cause of crash by looking at the Unhandled exceptions table because it shows the call stack.
But if this is not helping, then we move on to the next step which is collecting data via Crash Monitoring.
As I look at more details, n the next steps, it prompts me to enable Crash Monitoring with clear instruction.
Also, as I scroll below, I can see that there is an unhandled exception detected as well as some application event logs.
At times, you may be able to identify the root cause of crash by looking at the Unhandled exceptions table because it shows the call stack.
But if this is not helping, then we move on to the next step which is collecting data via Crash Monitoring.
Note that you are advised to use one storage account per app and refrain from modifying data on the storage account while the tool is running.
Configure the start time and end time using the time picker. You can choose the number of dumps you want to collect. I am choosing 2 and will click on Start Monitor.
Note when I start the monitor, this will update the app settings to enable Crash Monitoring and restart the app. Now I just need to sit back and wait for the tool to do the work.
After some time, the tool has detected a crash and automatically collected a callstack and a memory dump at the time of the crash.
I fastforwarded a little bit to show you what you can expect. Keep in mind, in real-time, this could take up to 15-20 minutes for this insight to populate after a crash.
I can simply click on the call stack to view it on the web or download the memory dump and open it in VS Code.
When the file loads in VS Code, right away you can view the exception type and exception message.
I’m going to click on Debug with Managed Only in the right corner.
This attempts to load the PDB files and open the exact source code of the function if VS and symbols are ined up properly.
Even if they are not, the Visual Studio Debugger will show exception details as you see on the screen and you can nail down on the exact line of code where the crash happened.
In our case, the crash was due to a function started on a new thread.
Look how easily we can identify the call stack and exception message directly from the memory dump file via Crash Monitoring.
This method can be easily applied to production scenarios as well.
Now switching back to my browser, I can see there was another crash detected with a different exit code.
I’m going to quickly check the stack trace and RIGHT AWAY I can tell that this was a StackOverFlow exception caused by a recursive function.
Now that I have collected all the information I need, I’m going to disable the agent which updates the app setting to disable Crash Monitoring.
However, even after I disable the tool, we can always check back on previously collected memory dumps up to 15 days as long as we don’t delete them in the storage account.
Now let’s recap what we went over today. We learned to identify an application crash using Azure App Service diagnostics
Then, we enabled and configured Crash Monitoring to Collect memory dumps and stack trace.
Finally, we analyzed the dumps to pinpoint on the root cause of the issue.
With that, this is the end of the video. Let us know if you find this video helpful by liking this video and subscribe to the channel to receive notifications on more Azure videos.
Thank you for watching!