[MUSIC]
>> Hi there. My name is Mark Downie.
I'm a Program Manager on
the Visual Studio Production Diagnostics team.
Today, I'd like to talk to you about debugging
memory dumps with Visual Studio.
For problems that do not manifest in logs or that you
cannot investigate by debugging locally,
you might attempt to capture
a diagnostic artifact like a memory dump.
Capturing a memory dump in essence is like
taking a high-fidelity photo of your application.
It represents a single moment in time.
It's kind of the equivalent of stopping
your application at a break point.
A memory dump is typically taken when an app running in
production is exhibiting some behavior that you need to mitigate.
Unfortunately for most scenarios,
attaching a debugger like Visual Studio in
production environments is typically
not practical or even possible.
To navigate this limitation,
we can capture a memory dump and copy that file to
our local PCs and then open that file in
Visual Studio and use the same set of
first-class live debugging tools we've grown accustomed to.
Today, I'm going to show you how easy it is to get
important insights from a variety of memory dumps.
The dump I want to look at today are referred to as crash dumps.
A crash is simply when your app unexpectedly
terminates and we usually capture
a crash dump right at that critical moment.
There are many reasons an app might crash.
The most common are typically unhandled exceptions.
These occur where an exception is
raised as a first chance exception,
but your code does not handle it very well.
The exception goes up the stack and becomes what we refer to as
a second chance exception and crashes at your process.
In order to capture a memory dump of any kind,
I tend to rely on tools like ProcDump,
and this is from Sysinternals,
there's lots of documentation on this
and makes a CLI command line tool that
allows you to capture dumps under a variety of circumstances.
In fact, they have a nice set of parameters that
allow you to capture it for may be
CPU thresholds or because maybe you have too much memory.
Towards the bottom here they have a nice list of
examples for you to pull from so I can catch
a full dumps of a particular process ID.
I can capture a memory dump if it's
exceeding 20 percent of the CPU for an extended amount of time,
I can capture a memory dump
if I see a particular type of exception.
All these are great ways of capturing
a variety of memory dumps and we'll look at a few of those,
over the course of this video.
I'm going to go ahead and get started
in open a crash-stop that I've
collected by hitting Control+O in Visual Studio,
which opens up the file open window.
I've got to dump here and the extension is.dmp here,
CrashStackOverflow and to manage dump,
is one of the ones I used for examples here,
so I'm going to go ahead and open that.
That lands me immediately on the memory dump file summary page,
the mini dump file summary page.
It has a really important information,
obviously the name of the file, but the last,
basically, essentially when this dump was taking,
the last right that was taken to this file.
It gives me the process name,
the application running as.
It gives me the process architecture
and things like the exception code.
The exception code, especially
something that you can use being in search
for to find out more details about that.
Gives you the OS version and the CLR version.
Really importantly, it gives you a list and version of
the modules that were loaded by this particular process.
I can use that to identified maybe if there's
version mismatches in my assumed environment.
On the right-hand side are
the important actions you can take against this memory dump.
Certainly, starting with things like setting symbols.
As part of your build process,
you'll probably produce a bunch of
program database files which essentially help you marry
up your code to
events in memory that are occurring in the process.
Essentially, if there's an issue
I can get right to a particular line
of code that's associated with it by setting the symbol path.
The most important action for managed applications
is to debug with managed only.
Some of you will have the scenarios
with either native or mixed applications,
but for the sake of this particular demonstration,
we're going to focus on Debug with Managed Only.
I'm going to go ahead and hit,
"Debug with Managed Only," and start my debugging session.
What this is going to do is,
make it as if I'm at a break point,
right at the moment
this particular exception occurred
or this crash exception was captured.
What that sets for me is a perfect opportunity for me to
review this as an use of
all the tools I'm used to when I'm live debugging.
Let's take for example this exception help.
This is a typical exceptional,
but we would see during live debugging,
if we were to capture unhandled exception.
It's telling me quite explicitly
that we have an Exception
of type System.StackOverflowException type.
Again, this is just
like any live debugging except for obvious cause.
Of course I cannot go forward.
I cannot use F5 or anything like that.
All I can do is look at the details
of this particular moment in time.
I'm going to view details, I'm going to click on the,
"View Details," and that will pop up the quick watch window.
Again, I can get way more details about
the exceptions here if there's
additional information of the innerexception,
I can use that or additional information
in the message exception,
I can also use that.
This is just a great way of gathering
much more data about this particular scenario.
Given that this is a StackOverflowException,
I'm thinking about the fact that this is out of memory
essentially and it's run out of
essentially frames on the call stack,
because it's running out of frames on the call stack,
I want to actually go to the call stack window
and I've got my down the bottom here,
click on the, Call Stack," window.
What you'll notice here is,
we've literally run out to frames and Visual Studio has
intelligently pared down the number
of frames on confu here so I can see what
the root of this call StackOverflowException is.
Obviously it starts in main.
I've got a method called infinite recursive,
it's obviously quite deliberate.
Then it tells me how many times it's
essentially repeated the number of
frames and you can see that I've repeated these number of
frames over almost 20,000 times.
At this point, I know the origins of this StackOverflowException.
It would be great at this moment if I could look at
the code there just to double-check any assumptions right here.
I can actually double-click on the call stack.
What that will do, at this moment I don't have my symbols,
my PDB files lined up.
What I could do here is decide,
actually go get my symbols maybe from
my build machine and that's the best way,
or if I don't have those handy,
I can go ahead and decompile the source code.
That will pull me directly into the line of code.
It looks just like a live session now.
I'm sitting here at what is essentially
a recursive function calling itself.
This is the issue that I have
and now I can go ahead and resolve that and
maybe open a case with
my developers and tell them how to resolve this particular issue.
Once now that I really love using memory dump analysis for a role
revolves around growth in the memory footprint of your process.
When there is unaccounted for growth over
time and unchecked growth over time,
I like the idea of using a memory dump to
analyze where that growth might be
coming from and whether it's really healthy or not.
I may have see growth in my memory footprint of
an application over the course of hours or days.
If I don't see that memory recovering,
I may decide that this is
a great opportunity to use two memory dumps,
one taken at the beginning and one taken at the end,
and compare where the growth is
coming from and to see if it's possible that
this memory won't be reclaimed.
If it won't be reclaimed, that is something that
I then need to resolve.
Let's have a look at that process.
If you want to collect a memory dump,
I typically start with a tool-like dotnet-gcdump.
This is a great command-line tool.
You can create essentially a dump that's really
super compact and just concerns itself with the heap.
You don't get to see threads,
you don't get to see values of particular objects.
You just simply get a list of objects on the heap and their sizes.
This is just a great compact way of
comparing two memory snapshots.
I'm going to go ahead and hit, "Control+O," in
Visual Studio and open up the memory dump.
This is the actually the second
of two memory dumps that I've taken out.
I've taken one at the start after restarting
the application because I want to
understand the nature of the growth.
Then I would have taken a second memory dump once
that growth has reached
growth mechanism where I think that it's obvious that
they're having unreclaimed growth.
I'm going to open up the second of
the two dumps so that I know what
the situation is once I've leaked my memory.
This opens up the Managed Memory viewer for Visual Studio.
It shows me a list of object types
and it tells me the count of those object types.
The size of the actual object
itself is the first one, sizing bytes.
Then inclusive size includes
anything that particular object references directly.
For example, I've a list object as my first one.
It has a size of 8,000 bytes.
However, the inclusive size that
is including the things it's referencing inside the list.
It's close to 16 million bytes or 16 brugner or 16 million.
In my mind's eye, I'm immediately thinking to myself,
these are likely culprits for my growth,
so this is obvious place to start.
Now, normal application here,
it may not be this obvious,
which is why I think the ability to compare
to your original memory,
dump the one you captured first as
the baseline, is incredibly important.
I'm going to go ahead and click on
"Compare With" and go ahead and open what is the baseline,
the first memory dump I took.
This allows me to compare
the first memory dump to the second memory dump.
Again, now I have conclusively seen where the growth is.
It's definitely in data records.
I'm seeing this increased by an incredible amount,
by 1000 or so.
But the question really is,
why didn't I reclaim this?
Why wasn't this memory reclaimed?
This looked like a regular list.
Why isn't this list gone away?
I've taken these two memory dumps at two separate points
and if I go and look at the path to root,
it makes it quite obvious to me that
this list is in fact a static variable.
If you know about static variables,
you know that they actually hang around for
the entire lifetime of the process.
This won't be cleaned up unless I am deliberate about cleaning up.
Here I found what is essentially a leak in a static list.
In addition to things like crashes or memory leaks,
I also like to think about how we can use
memory dumps when your app isn't responding correctly,
when it's slow or completely unresponsive.
For that, I often like to
use the Parallel Stacks window in Visual Studio.
The Parallel Stacks window is a great way to get
the big picture view about an application that you're using.
My friends in our debugging and diagnostics team like to
think of the approach to
debugging as thinking about the big picture first.
We want to think about what the condition of our threads are.
Then once we figured out what the threads are doing,
maybe we identify a particular call stack on a particular thread,
and then from there we might dive deeper and get to code
and objects and look at the analysis from that perspective.
The way in which we can use
the Parallel Stacks window is to think from
big picture and we slowly get closer and
closer to the problem at hand. Let's dive in.
I've some great friends in the open source community
who have created this application.
It is designed to mimic the stock market in some way,
but they've deliberately created it so that
essentially it becomes unresponsive after a few moments.
I'd like to use Visual Studio to
help us understand why it's doing that.
I have started my debug action
that we've debugged with Managed Only.
I can start it up and I'm going to go
ahead and navigate because I want to see the big picture here.
I'm going to go ahead and navigate to the parallel stack window.
We can use that by going to the debug window,
debug windows and parallel stacks.
I want to see the big picture here.
Here what it does is give me a graphical overview of
all the threads currently running in this application.
I'm going to scroll here.
I'm actually going to zoom out just a little bit so I can
see a little bit more of the threads here.
You'll see it'll show the relationships between threads,
it'll show the unique call
stacks that each of these threads are running.
For example, there is a big thought
in my head, when I'm seeing this,
I'm seeing 90 threads over here on the right-hand side,
I'm seeing 89 of those threads here,
and one going in this direction.
Same similarly over here,
I have three threads and these are the unique portion.
Of these three threads here,
this is the common portion right here
and they split off and go in their different directions up here.
What's interesting to me is this little icon.
You'll notice this icon here that is representative of a deadlock.
Now, the fact that we have an unresponsive app and
parallel stacks is immediately
telling us that we have a deadlock scenario,
that's really important because now I'm starting to
think about the ways in which we get deadlocks.
From a purely hypothetical
or from a theoretical standpoint, excuse me,
you would think about a deadlock as essentially thread A,
having a lock and waiting for a lock that another thread owns.
Typically what usually happens is that thread B is then
waiting on the first thread to release a lock that it owns.
We have this deadlock situation where both threads
are waiting for the other thread to release something.
Now my job is to find out where that deadlock is originating from.
I'm going to use this window here to review the threads.
Here just looking,
what I'm seeing, especially right over here,
obviously I've got 90 threads,
so these are all going to wait,
they're going to continue to wait indefinitely.
If I look here on thread 7048,
what I've noticed is that it's waiting on a lock that is
owned by thread 28964.
I'm going to go find thread 28964 and see what it's waiting for.
If I go over to thread 28964 over here,
I've noticed that it's waiting on a lock owned by
thread 7048 and so exactly the deadlock scenario.
We have one thread
waiting on a lock that is owned by another thread,
simultaneously, that other thread is
waiting on a lock owned by the first thread.
This is where at deadlock is occurring
and parallel stacks window is shouting that out to me.
What would be interesting right now is that
I'd like to see the code that
both this Worker Thread and this Cat Grumpy thread is running.
What I can do here is "Double-click" on
the frame that is essentially doing the waiting here.
The Monitor Enter is actually
some core code from system.threading.
But this looks like a user code or is in fact user code,
so I'm going to go ahead and "Double-click" there.
If I had symbols from my build process,
I could go find them and associate them and load them here.
But I don't, so I'm going to go ahead
and decompile the source code.
Now I see the locks I've been mentioning.
I have a lock on buyer and a lock on orders.
Actually a lock on seller,
buyer, and orders.
Interesting. Let's go back to Parallel Stacks window and do
that same exercise for the other thread.
We're going to go and look
at this thread and we
"Double-click" on that frame and this time
we have orders and sellers.
That's interesting. If you remember,
the other thread was sellers and orders,
this one is orders and sellers.
Essentially, this is a typical deadlock scenario.
If we want to eliminate this problem,
we have to ensure that our locks are done in the same order.
Both threads should lock orders and sellers in that order,
or sellers and orders
in that order to avoid the deadlock scenario.
Thank you for joining today.
I hope this helped in your understanding of how Visual Studio can
be used to help you debug managed memory dumps.
Good look hunting those bugs.