LESLIE RICHARDSON: I'm here right now
with Przemyslaw Wirkus,
who's a principal engineer at Arm. How do you doing?
PRZEMYSLAW WIRKUS: Hello.
LESLIE RICHARDSON: Cool. Arm is really cool,
it's really grown a lot in recent years.
Just to kick things off
since we're going to be talking a lot about Arm.
What are some of the challenges
that come with measuring performance?
PRZEMYSLAW WIRKUS: The critical part is to
have the right tools for the right job.
Select the workloads to measure your performance,
and also understand how it all
fits into the Windows and Arm ecosystem.
Today, I would like to present
the tool that we developed in Arm called WindowsPerf.
This is the micro architecture performance analysis tool.
Together with the Arm's telemetry solution methodology,
we implemented the top down
hot spot methodology inside the tool.
We can actually follow steps that were
precisely defined by our architects
to actually measure the performance.
Today, I would like to show you and maybe execute
Windows path on our neo
verse Windows machine that I just log into.
I would like to show you how we step by step,
do the top down analysis,
how we use the new and existing tools
like top down tool to do that automatically.
Also, I will showcase how we actually
fit into the Windows on Arm performance ecosystem.
Because we don't want to reinvent the wheel,
we just build enough so you can get what you want.
Everything else is already there from Microsoft.
LESLIE RICHARDSON: I like that.
Work smart, not hard.
PRZEMYSLAW WIRKUS: That's right.
LESLIE RICHARDSON: Do just enough for these people
to get to fixing their problems.
We're going to kick things off.
It sounds like with the demo.
PRZEMYSLAW WIRKUS: Yes. Just a little bit
of the introduction.
You can go to the Arm developer webpage
to read about the telemetry solution,
top down methodology, and much more.
You can also visit our WindowsPerf web page,
just to get our software release,
binaries and also read about interesting stuff.
We also link other documentation like
Arm learning path and Arm CPU telemetric solution data.
I will later also show you
how I actually follow the top down methodology.
We actually define also metadata in
a JSON format that actually
describe events and metrics and groups of metrics,
so we can actually just
follow steps designed by the architect.
This JSON file, we also include into the WindowsPerf.
At the bottom of it,
there is a section called methodology,
and I will just follow the steps described by this.
I will just follow the cycle
accounting group of metrics,
I will see where our workload.
We're going to work actually on
a C Python native arm 64 workload.
We're going to compute Googleplex.
I'm going to follow the steps
manually to explain what's happening,
and then I'm going to run
the top down tool from
Arm to show you how we can automate that process.
After that, I will go a little bit deeper,
we're going to do a little bit of the sampling to show
you where in the source code
exactly your hot spot could be.
Let's start.
I have some helper document to help me with the commands.
LESLIE RICHARDSON: Meaning you're not
going to memorize all those?
PRZEMYSLAW WIRKUS: I don't memorize
them. They are automated.
But what's going to happen quickly is I'm going to
do the WPA start command, we're going to count.
Minus- stems for the metric.
We're going to pick up the group of metric is all
implement and baked into the WindowsPerf.
On core seven, we're going to run C Python,
and the minus C 10 to the power
of 100, we're going to evaluate this.
We're going to run this for 10 seconds.
While we are running this, I
can explain what's happening.
We are now performing the static performance analysis.
We're going to run the workload, collect the data,
and after the workload finish,
we're going to analyze actually what we learned.
LESLIE RICHARDSON: Cool.
PRZEMYSLAW WIRKUS: It's not real time analysis.
Our workload is also designed to
exploit certain algorithm or a module that we're
trying to optimize because this is
micro architecture performance analysis or profiling,
not the whole application.
LESLIE RICHARDSON: Cool.
PRZEMYSLAW WIRKUS: For that,
there are different tools.
LESLIE RICHARDSON: Awesome.
Microarchitecture profiling
is getting nitty gritty.
PRZEMYSLAW WIRKUS: That's right.
Getting inside the CPU,
nitty gritty details and understanding which parts of
the CPU are actually
lighting up when your workload is running.
You can actually passively observe if
for example, you're running vectorization,
or if the crypto instructions are running,
or maybe your load store speculation is on a heavy side.
We're going to observe this with the workload,
but the first step in the
top down analysis is to actually
look from the very high point of view on the workload.
We're going to look at the front end
and the back end of the CPU just quickly.
Front end of the CPU is like a pump,
trying to sack all the instruction in the CPU
and speculate which one actually should get in advance.
Whereas the back end is the conveyer that actually
transmits all the instructions
later and actually perform the computing.
The back end, you are mostly interested about the data,
whereas in the front end about
the instructions themselves and the decoding.
Here at the bottom, you can
see that for the telemetri solution metrics,
we see that the back end is maybe not heavily,
but it is used much more frequent than the front end.
We can actually start zooming in at the back end.
Because in adjacent, I
already checked what are the groups of
metrics that we actually should
inspect to look at the back end,
I'll just do not do that automatically from
my little tiny helping list of the commands.
LESLIE RICHARDSON: That analysis
came up pretty quickly,
too, is that the norm or?
PRZEMYSLAW WIRKUS: Yes.
That's the norm because workload
is usually very small.
Because you want to run it in a reasonable amount of
time to just saturate enough events.
Please remember, this CPUs
are running like three gigahertz,
so it's a billions of instruction every second.
It's really enough to look at what's happening.
Also, performance engineers will actually
adjust the time and the workloads independently.
But that's another set of problems that one has to solve.
Here, we have a very simple workload.
LESLIE RICHARDSON: Gotcha.
PRZEMYSLAW WIRKUS: We're not going to
worry about that that much,
but it's a very good question, actually.
We're going to run this command,
a little bit more complicated because after -Arm,
I have many more metrics.
I'm going to run exactly the same workload,
the same things will happen in the CPU.
I did not change anything.
It's also very important
when you are running a workload that
your workload has no external state.
It will not read a file,
it will not check the time,
it's always the same piece
of algorithm or a module doing exactly the same thing.
LESLIE RICHARDSON: Awesome. You're able to
analyze just the front end and just the back end?
PRZEMYSLAW WIRKUS: That's right.
Usually, you own
the code you are actually profiling.
In general, we know what we are looking at.
We just want to understand exactly
where maybe not where we make mistakes,
but where we can actually improve.
Or what we can actually change
to make that workload better.
Please also note I'm not
actually measuring any time here,
so I don't care how fast the workload is running.
It is actually continuously doing
the same computation of 10^10, 100.
I finished the analysis now of the back end metrics.
I actually use the multiplexing of windows pad.
It's a very nice feature that allows you
to measure many events at the same time.
Because in general, you only have
few registers to actually latch those events,
but we are actually multiplexing.
What we're doing here, we
can quickly just look at the list.
At the bottom, actually, we can see that
mostly load speculation, store speculation.
LESLIE RICHARDSON: Energy or DP.
PRZEMYSLAW WIRKUS: Integer double precision speculation.
These are the dominant.
We're actually expecting that because we're actually
just taking integer numbers,
and we're actually multiplying them a lot.
How Python works, we are
actually calculating very, very big number.
It will grow in memory.
Actually, we're expecting the load
and store will actually be dominant, and indeed,
they are taking around 35 percent of
all the speculative instructions
is actually very useful to understand.
Also, when you look at this set of metrics,
you can see here, for example,
that there is no for example,
scalar floating point operations is actually good.
We don't want to use floating point just to compute
integers because they are slower on every architecture.
me try to also go deeper.
Let me see what I can do.
Okay, so I already I
already understand I have few speculations.
Let me to look at. Now let's go deeper.
I will actually look at the load speculation.
Our favorite test event
that we test all the time is the load speculation.
Now I would like to understand with sampling.
I'm going to do change the way
I'm actually measuring things.
I went from counting stuff to actually sampling.
A sampling is actually a process.
I will call it it's statistical process.
Now we use maybe
some statistical methods to actually understand
where we are actually hitting in
the source code all these hot areas.
Perfect. I will just annotate at
the same time because we don't have
so much time to actually show everyone.
I will do something very similar to what I did before.
By now you can see, I will do the -e ld_spec.
I will just try to see when my event counter saturates,
where I actually hit the program counter.
I'm going to also correlate this with the lines of code.
Actually, I can talk about this when we actually do it.
I can correlate this.
Actually, I can understand where in the source code,
I'm going to actually hit some bottlenecks.
Because what we want with the hotspots
in the source code for developers.
LESLIE RICHARDSON: That's important.
We don't want to do all the trial and
error for things that have nothing to do with it.
PRZEMYSLAW WIRKUS: That's right.
We did this analysis,
because we have this C Python
build with the debug information,
I have all the information about all the symbols,
all the functions, and the D LLC is calling.
I can see that in this case,
79 percent, of
all the maybe heats when we saturate the ld_spec event.
Actually happened in the x malfunction in
the Python DLL is
actually a good news because we are multiplying.
This function, it looks like it's multiplying,
and actually, we checked, and it is multiplying.
How you can actually go even deeper.
You can actually disassemble to see
which instruction exactly is
being the one that is causing the problem.
But this is sampling in software,
so there is a little bit of inaccuracy.
We will eliminate that because we're going
to implement the RMSPE,
which is Sampliging hardware.
I can actually move to something a little bit different.
I want to show you how we implement
all this stuff with the sorry,
with the Visual Studio Gui.
We have extension that does exactly the same thing.
We can run the sampling or we can run the counting.
It's actually going to show you in your source code.
Where are the hotspots? It's very useful.
LESLIE RICHARDSON: Yeah, super useful.
PRZEMYSLAW WIRKUS: Especially when you
just have a smaller app.
You are open source.
You have a micro transaction.
It's easy for you to just run this from visuals.
It is ecosystem.
LESLIE RICHARDSON: Or if you're not someone
who likes to use the terminal. That's right.
PRZEMYSLAW WIRKUS: Not everyone likes
to use a terminal.
Why we have Windows per also as a terminal application?
Because we are actually Linux perve inspired.
We want the performance engineers to actually
join windows and arm performance analysis,
and it's a very easy way for them to actually join.
You can see here at the bottom,
we have the disassembly,
that I didn't show you from the command line
because it's a little bit hard to see.
But we have the disassembly.
What else we can do?
We also can use the WPA,
which is the Windows Performance Analyzer.
We develop a plug in that reads
the windows purve Json output.
It processes it, and we can use the WPA,
which is a Microsoft tool that is used by everyone.
That has a performance analysis.
LESLIE RICHARDSON: Oh, great.
So you have that Windows ecosystem.
PRZEMYSLAW WIRKUS: That's right. We integrate.
We don't develop something new.
Right. If there is a visualization, already available,
we just attach ourselves
to that with that adapter, and that's what we did.
LESLIE RICHARDSON: If it ain't broke.
PRZEMYSLAW WIRKUS: Yes. No one
has to learn anything new.
This is just an example of me showing you
different metrics and different events that
I just captured running a different workload, actually.
It's a different workload
and over the time with the timeline,
we can actually show how it works.
That's one of the other highlights of this solution.
I think, in general,
that's all I wanted to show.
LESLIE RICHARDSON: You're so casual about that.
That's all
I want to show. Not a deal.
PRZEMYSLAW WIRKUS: It was a fast track
for performance analysis.
Performance analysis is not easy,
even for advanced performance engineers.
I think that's all the key takeaways.
LESLIE RICHARDSON: Cool.
Thank you for sharing that.
Performance is so important to so many people,
and making it easy to
detect those performance. Bottlenecks is so important.
PRZEMYSLAW WIRKUS: That's right.
LESLIE RICHARDSON: What's next?
PRZEMYSLAW WIRKUS: What's next?
We're going to add
RMSP support, so hardware sampling.
Why we didn't add it now?
Because not every RM CPU has actually
that feature baked in.
For example, mobile devices will not have.
But my server workstation does,
so you want to have the software solution.
Just to have sampling.
Then you have one in hardware for your server solutions.
Other things that are coming is we're going to
actually integrate with the ETW,
extended traces for Windows.
You can actually inject all the data directly into
the ETW which is the right way to go.
We are again tapping into
the Windows performance ecosystem.
LESLIE RICHARDSON: Great.
Well, I'm excited to see
what's going to be happening next year's build, shoot.
Sherman, thank you so much.
PRZEMYSLAW WIRKUS: Thank you very much.
LESLIE RICHARDSON: If you're
interested in learning more
about arm, go check out the booth.
If you're here, you still have a couple hours.
If not, no worries, go to
the featured partner directory and tap
them out there. Thanks again, Sherman.
PRZEMYSLAW WIRKUS: Thank you very much.