Hi, I'm Mike Rousos from the .NET Customer Engagement team and today
I'm going to talk about doing performance analysis of .NET Core apps on Linux.
Now, if you've done performance analysis of .NET Core
(or any .NET applications) in the past, you're probably familiar with the PerfView tool.
PerfView is a
performance analysis tool that was created by and is maintained by the .NET team for gathering memory usage, GC behavior,
CPU usage, profiling information about .NET applications.
It's really a very useful tool for digging into sort of finding out where bottlenecks are or any
characteristics of the performance of a .NET app.
The good news is that PerfView also works with .NET Core the same as it does with .NET Framework, so
any experience you have with PerfView will transfer over to analyzing .NET Core applications.
The challenge is that PerfView runs only on Windows today.
So if you're trying to understand the performance of a .NET Core app running on Linux, it's not going to work directly there.
Instead, we have another tool which I'm going to show you today called PerfCollect. PerfCollect is a script created by the .NET team
for gathering performance traces and information about .NET Core apps on a
Linux machine and then those performance traces can then be analyzed either on the Linux machine using native tools or they can be analyzed using
PerfView on a Windows machine.
Now I'm going to focus in this demo on PerfCollect itself - what it is,
how it works, how to use it, rather than on PerfView because PerfView already has a lot of documentation available.
If you'd like to learn more about PerfView itself, you can go out to our GitHub page Microsoft/PerfView.
And I'd encourage checking out this link where we have a multi-part video
tutorial out on Channel9 on how to use PerfView. So you can use those to learn that tool. Today
we're going to look at perf collect
which is just sort of that missing piece that helps us get the performance information from a Linux machine.
So the first thing to do is
to install PerfCollect. So I'm going to
SSH to my Ubuntu VM running in Azure.
I'm actually going to do that from two
different windows because we're going to want to be able to
run our .NET Core app that we want to profile in one of these panes
while at the same time running the PerfCollect script on another pane. So we want to have two of these.
Okay
So, let's go ahead and install PerfCollect. I will create a directory for it
And installing PerfCollect itself is actually super easy because it's just a shell script, which under the covers uses other popular tools
for
analyzing performance on a Linux machine so it's going to use LTTNG and Perf for collecting the actual performance information.
So PerfCollect itself is just a wrapper shell script so installing it is as as simple as just curling down
aka.ms/perfcollect and
then, once we've done that,
make it executable and
we do have one pre-req before we run it. Since
the script depends on things like perf and LTTNG,
we need to run 'PerfCollect Install' before we're able to use the script. So, that will install the necessary dependencies.
And if you're curious to learn more about what exactly PerfCollect is doing, you can just navigate out to aka.ms/perfcollect
When we call the install command, it ends up, ends up just, under the covers,
just calling things like install LTTNG so it's going to install LTTNG tools
as necessary and its dependencies
depending on
the Linux distro that you're using.
Similarly, it installs Perf so you can see it just installs these
Linux-tools-common.
So it's really - It installs zip so that it can
zip all of the perf results up into an archive.
But these, these are sort of the underlying tools that get installed when we run
'PerfCollect install'. Now that we've installed those prerequisites,
we can start collecting information by running 'PerfCollect collect' and
what that does under the covers is it calls this DoCollect function in the shell script and
all that's happening here is,
we launch LTTNG collection and
then we run Perf
to try and collect that. You can go and look at what sorts of command arguments we pass, but it's pretty standard.
Its perf record -G and then
optionally we can specify
what process to record. By default it just records everything running on the box.
So under the covers PerfCollect takes care of running these tools for us, and then packaging the results up into a
zip file that's easy to transfer.
So now that we have PerfCollect installed and ready to go,
the next thing we need to do is we need an app that we're actually going to
profile. So I have something I call SpinSample, and this is actually just taken directly from the PerfView tutorials. If you go out to,
let's see, where is it... Here it is. Go out to our PerfView GitHub
repository and go to PerfView/src/PerfView/SupportFiles.
There's this tutorial
that's used in PerfView getting started guides as an example app to profile.
And it's very simple. It takes an argument which defaults to 5 if no argument is given and it spins for that number of seconds.
It spins by calling RecSpin
which decrements the number of seconds to spin, spins for 1 second by just having a loop that just
does some work and checks to see when a second has passed and
then calls RecSpingHelper, which is exactly the same as RecSpin, except that it calls RecSpin instead of RecSpinHelper.
So these two methods just call back and forth once a second just
looping and burning cycles in between.
And the reason we have two methods instead of one is because it makes for a slightly more interesting call graph when we analyze things
in PerfView at the end of the... at the end of the collection.
So, we come back over here to my
SSH session. You can see that I have that
That tutorial app here. I've called it SpinSample, but this is the same code that we just saw.
So let's go ahead and do a 'dotnet build' to make sure that's all built and that it builds properly
and then we can go ahead and start collecting
Now, The challenge here is less about collecting the performance information because just running PerfCollect right now
would already gather performance information on the native portions of our application on the .NET
Core host, on the GC, JIT,
execution engine, etc.
But you likely are going to want information on the managed pieces of your application, on the .NET Core framework, itself, on your code
in the application
libraries and
those sorts of
managed code are not going to
be
profiled in a useful way because we won't have any symbol information mapping the
perf data back to where in the code
the calls were being made since that's all managed code
and it's not something that perf and LTTNG have a native understanding of.
So what we need to do is we need to make sure that we're going to have perf map files available to map
from the information gathered by Perf back to our C# application, so we do that in two ways
First is we export some environment variables.
COMPlus_PerfMapEnabled=1 will instruct the .NET Core JIT to generate perf map files
for IL that it compiles at runtime.
Then we also want
COMPlus_EnableEventLog set to one because that's going to enable some other
performance diagnostic
trace emitting within the .NET framework runtime itself.
Once we've set both of those we'll be able to get perf map files for all of our JIT'd code.
But a lot of the .NET
managed code that you run will have been compiled ahead of time. This will be anything that you ran crossgen on. All of the .NET
Core framework is compiled ahead of time, and it's actually possible to use our
precompilation tool crossgen to generate perf map files for these native images, but
we need to have crossgen in order to do that.
Fortunately, PerfCollect makes this very easy because PerfCollect will call crossgen on our behalf and generate all the necessary perf map files.
But we still need to have the crossgen tool available. The easiest way to get it is just to publish our application
because when we publish it, we'll have to restore the
Microsoft.NetCore.App 2.0 NuGet package to include in the self contained publish output of our application and
when we do that, crossgen happens to be one of the tools that's included in that NuGet package.
So I'm going to publish my application standalone
for the
x64 Linux runtime and
as part of doing this, as part of this
publish process since we're not publishing in a framework dependent way, we're publishing a self-contained app we need to have all of the framework dependencies -
The .NET Core, .NET Framework all locally so that we can distribute them with our app. Since we've done that we now have,
if we go look at our NuGet
packages, we should have a
runtime.linux-x64.microsoft.netcore.app 2.0
package and inside there,
there is a tools directory which contains crossgen so now what needs to happen is -
by convention,
PerfCollect will look for crossgen next to libcoreclr.so, so we just need to copy this file
next to our libcoreclr.so.
Now that might be in a couple different locations. It might be, if we're going to run our application using the .NET
CLI, then we're gonna use the shared .NET Core installation on the machine, so
the libcoreclr.so file would be under usr/share/dotnet/shared/microsoft.netcore.app/2.0.0. I'll just type it out
so you can see. It's under
usr/share/dotnet/shared/microsoft.netcore.app/2.0.0, so we'd want to copy cross-gen there.
On the other hand if we're going to use our
self-contained
published output
that we just created by publishing our application, then we'll have a libcoreclr.so local to that publish output
which is going to be used. So in that case, we would need to copy this next to ./bin/debug/netcoreapp2.0/linux-x64/publish.
Okay, so now that's done. We can actually go take a look at what's there so you can see that that was the right location.
And if I search for libcore*,
you'll see that is where libcoreclr.so is. So that was the right place to put crossgen.
Now that crossgen's in place, PerfCollect is going to be able to generate our
perf map files for native, for native images and
we have our environment variable set to be able to get perf map files for
JIT'd manage code, so now we're ready to actually run the PerfCollect script and gather information on our, on our
application. So in order to that we're going to
hop into that PerfCollect folder.
'Sudo ./perfcollect collect', and we have to give a name to the trace that we're going to save. We'll call it SampleTrace.
Over in the other pane, we're going to run our application. May as well use our
published output that we created, so I'm going to run SpinSample over here
and so
run PerfCollect. Once perf collection has started,
I start my application, which spins for 5 seconds. Once it's done, ctrl-c breaks,
we process this, we generate native image files, save native symbols
export everything, and now it's packaged up into this SampleTrace.trace.zip file. If
you're curious, we can actually
unzip
SampleTrace.trace.zip to see what's there and
you'll see it puts it all in this folder.
And so in here we have the perf data that was collected,
we have the perf maps that were generated by the JIT,
We also have the native image perf maps that were generated by crossgen and, in fact,
These are just normal perf files, so if you're comfortable doing performance analysis on a Linux machine,
you can very easily do that here if you like. You can just run 'perf report' and
you're gonna, and you're going to get the information you would expect to see
when using perf. And so you can see, okay, here we have exclusive and inclusive time spent in these different processes
so we can,
you know, sort of look at that as you normally would. You come down say okay, here's this RecSpin
function which, you know, was 27%
inclusively of all of all of our time. It didn't actually spend much time itself,
but if we dig in we see, oh, it was called from RecSpinHelper.
We can sort of go back up.
We can see SpinForaSecond is where we were actually spending some time, see what called it, and so on you can.
Use this the same way you would any Linux performance analysis.
We actually also have some scripts in PerfCollect that will allow you to do this without
Even unzipping the the archive if you do 'perfcollect report'
or 'perfcollect view', sorry, that will give you the same sort of perf analysis directly from the zip file.
So that's an option as well. Now,
for a lot of .NET developers,
they're going to be already familiar with PerfView and would probably like to continue using that tool.
So if you want to do that, that's something that perfect collect is made for. It gives you this zip file, all you need
to do is copy that zip file onto your Windows machine and launch PerfView like normal. So let's go ahead and
copy that
file locally. We're gonna copy it from
mikerou@mikerounix.cloudapp.net. We're going to copy
/home/mikerou/perfcollect/SampleTrace.trace.zip and we'll just put this in the current directory.
It should be copying. Come on.
Not sure why this is being slow. This should be pretty fast, it's not all that large of a file
Ok, what connection closed?
Did I type something wrong? Let's give it another shot. Ok, there we go. This looks more like it.
Ok, so we now have that file locally, so what we can do now is back on my Windows machine
just from the Linux subsystem shell here
I'm launching PerfView, and I'm providing SampleTrace.trace.zip as, as the file that we want to analyze.
So we launch that, PerfView starts up. Now, again, if you aren't familiar with PerfView,
you just click on this tutorial link when you start it up, and it's gonna walk you through
sort of profiling and understanding all of the reports for a sample application.
Makes it easy to use. In our case, though, we can just hop in here to the CPU stacks.
You can see I've got all of this. Right now, it's grouped by module.
Maybe I want to see it more granularly, so I'll remove the grouping, but we can start to understand what's happening within our application.
if we sort of look,
look in here for example, we see a a lot of time is being spent in this native read MSR safe function. Well,
why are we there? Well we can come in here and we can start to drill out looking at callers until we discover
Hey, there's something called SpinForaSecond which is called from RecSpin and RecSpinHelper
and that's calling this thing where we're spending 20% of our time.
And that's just 20% of the profile time this isn't even specific to this particular process yet, we're gonna find out
it's the vast majority of this process and so we can say 'ok, well we need to go check that out.'
You also could come at it sort of from the other direction if you wanted to do
more of a top-down approach, you could sort by where you're spending your time inclusively.
We see that here we have SpinSampleMain.
Come in here, we can go look at our callees and
we can see, all right, SpinSampleMain's taking most of our time,
but where within SpinSampleMain are things actually happening? So we can jump in here,
let's see, sort of dig down until we find something
interesting. Here we go. Here's our manage main method.
And we see, oh yeah, most of the time in that main method
(maybe we'll even just drill into this in a separate window that focuses specifically on it)
Yeah, we can see 99.8% of the time that our application is running. It's in RecSpin
That's interesting.
Come in here, 80% of that's in RecSpingHelper. 20% is in this SpinForaSecond. RecSpinHelper,
though, also goes into SpinForaSecond and, all of a sudden, we start to get a picture of how our application is
spending its time.
Again, I don't intend for this to be a deep look at PerfView.
There's already a lot of very good learning materials available for PerfView.
But I wanted to just demonstrate how you can use the PerfView tool even when you're interested in understanding
performance of a .NET Core application running on Linux, and you can do that
using this PerfCollect script
to gather that information that, then, you can go ahead and analyze
either on your Linux box or back on Windows using PerfView. Hope this has been helpful.
Thanks.