[MUSIC PLAYING]
PIERRIC GIMMIG: Hello, and welcome to this session
dedicated to Stadia's profiling tools.
I'm Pierric, software engineer at Google.
And joining me today is Michael Anttila
from the Stadia graphics tools team.
Today we would like to present an overview
of Stadia's profiling tools.
I will first present Orbit, our CPU and system profiler,
before handing it over to Michael
to present Dive, our new GPU analysis tool.
Let's start by looking at the CPU side of things.
The very first thing I want to mention
is that if you've installed the Stadia SDK,
then Orbit is ready to use.
It looks like this.
Orbit is a feature-packed native CPU and system
profiler that connects to a Stadia instance
and displays live profiling events
as a user interacts with the game.
We'll cover most of Orbit's feature in this presentation.
But before that, I'd like to get back to basics
and outline the core idea behind the tool.
First of all, why do we profile?
Well, it's no secret that performance
is paramount in games.
And not only that, but good performance on average
is not enough.
A single frame taken more time than it should
will break player immersion.
What we're aiming for is for consistent performance.
Also, profiling allows us to make sure
that we're using the available resources
to their full potential.
Being able to visualize time during the frame
where the CPU is idle can help us guide optimization efforts
by offloading work to other cores, for example.
And finally, profiling helps gain a deeper understanding
of our own code.
It allows us to build an accurate mental model
of our complex multi-threaded application.
To me, a very good way to build this mental model
is by visualizing function calls.
We can see that the code on the left
can be represented by the diagram on the right.
In Orbit, we call this diagram a flame chart.
On a flame chart, the x-axis is time.
Sequential function calls will appear from left to right.
The width of a rectangle corresponds
to the time it took for the function to execute.
The y-axis corresponds to the function call hierarchy
where nested function calls will appear from top to bottom.
Visualizing multiple threads is also very intuitive.
We simply have multiple flame charts on the same timeline.
This can be very useful to visualize interaction
between threads to, again, help us
build our mental model of what's going on in our frames.
Now, the question is, how can we generate those flame charts?
Well, the easiest way would be to modify code.
However, this would mean that to instrument a new function,
we would have to shut down our game, change the code,
recompile, re-link, redeploy, relaunch.
This can easily take 15 minutes on a big project.
Also, performance problems in games can be highly contextual.
You might need to reproduce a long sequence of events
to trigger your performance bottleneck, adding
to this iteration time.
Another problem here is that you might
be interested in profiling modules for which you
don't have source code.
So how can we generate these flame charts
without relying on code annotations?
Orbit's answer to this question is dynamic instrumentation.
It is the number one feature of Orbit.
It allows us to generate flame charts
without having to modify a single line of code.
There's no integration time, virtually no iteration time,
and you don't even have to relaunch your game.
In Orbit, you can select functions
to dynamically instrument, or hook, and on the next capture,
the timeline will be populated with all occurrences
of the hooked function calls.
This information will be presented
as flame charts in the capture view,
as we can see on the left.
Note that this is exact information.
This is not statistical data, like what
you would get from sampling.
Under the hood, Orbit is actually
intercepting function calls and is getting a timestamp
on the entry and exit points of instrumented functions.
In addition to dynamic instrumentation,
Orbit does offer a powerful sampling profiler.
Sampling, by the way, is the process
of periodically getting a call stack for all running threads.
Sampling is always on in Orbit as opposed
to dynamic instrumentation, which requires user input.
In the diagram, we see that every sample represented
by a yellow arrow produces a call stack.
In Orbit, we can visualize this call stack data
in different ways.
First, we present every single individual sample
on the timeline as a vertical white line.
This is what we see on the sampling bars on the left.
On the right side, you will find three different ways of looking
at the same sampling data.
We have the flat, top down, and bottom up views.
The flat view is a flat list of functions
which can be ordered by how often they were
present anywhere in the call stack,
or also by how often they were actually executing
at the time of sample.
Note that it is possible to generate reports
for a selection of samples by interacting with the sample
bar.
The top down view presents sampling data
as an intuitive call graph.
So basically expanding nodes will let you traverse the call
hierarchy from parent to child.
The bottom up view, on the other hand,
allows you to see which functions
are being executed the most.
In other words, the function in which your program counter
was at sample time.
Expanding the nodes here will let
you traverse the call hierarchy in the opposite direction.
So from child to parent.
So dynamic instrumentation on its own is great,
and so is sampling.
When you combine both, you get something
that's worth more than the sum of its parts.
It unlocks a workflow that makes iteration incredibly quick,
and this combination of sampling and dynamic instrumentation
is Orbit's core value proposition.
And a good way to understand this value proposition
is by putting ourselves in the shoes of someone who
knows nothing about the binary they want to profile.
How would we go about populating a detailed per-thread breakdown
of important function calls?
Well, we start by taking a capture
without any dynamic instrumentation.
We then analyze the sampling report
to identify functions that would be interesting to instrument.
We would then hook those functions
and take a new capture.
Very quickly, we'll populate flame charts
for the different threads of our application.
We could iterate on this loop a few times
to make sure that we've identified
a good set of functions.
But in the end, we'll have generated
a detailed capture in no time.
And again, without having touched a single line of code.
Once we're happy with the set of functions
that we've identified for our profiling session,
we can save them in a preset so that we
don't have to manually hook those functions again
on every capture.
This will reduce iteration time even further.
You can then simply launch Orbit, select a preset
and capture.
Of course, if you run into a new performance
problem along the way, you can always
add functions to the preset.
So a quick note on debugging symbols.
In order to profile, we need them.
This is Orbit's only requirement.
They can live either on your instance
or on your local machine.
If they're on your instance, Orbit
will transfer them to its simple cache on your workstation.
So debug symbols are used by dynamic instrumentation
to know where functions are in memory.
They're used by sampling to do the mapping
between an instruction and a symbol,
and they're also used by our source
code and disassembly views for line information.
So now that we've covered the core functionality of Orbit,
let's have a quick look at some of the other features
it has to offer.
In the capture view, the first track that you will encounter
is the scheduler track.
At a glance, you get an idea of your core occupancy.
You can also easily see what core
your threads were running on and visualize context switches.
Related to scheduling information,
we show thread states right above the sampling bar
in the thread state bar.
A thread that is running will be shown in green.
A thread that is runnable, but not scheduled
would be shown in blue.
A thread that is sleeping would be shown in gray.
You can hover over the different sections of the thread bar
to get a description of the thread state.
To keep an eye on memory, Orbit has dedicated tracks
in the capture view to trace memory usage and page fault
information.
For more info on this, you can hover on those tracks
to have useful tool tips.
I would now like to talk a little bit
about manual instrumentation.
So we've seen that dynamic instrumentation is the number
one feature of Orbit, but manual instrumentation
can still be useful.
For example, you might be interested in profiling
sections of functions, or profiling
asynchronous operations that would start in a function
and in another one, or you could be
interested in visualizing the state of variables
directly in the Orbit timeline.
And for that, we provide a manual instrumentation API.
We took great care in making sure
that integrating manual instrumentation in your code
base is as easy as possible.
There is no library to link with.
All you have to do is include a single header
file, Orbit.h, and use our simple API in your code.
It's worth noting that our manual instrumentation API made
its way to the Unity codebase.
The scopes that you would see in the Unity Profiler
also appear in Orbit, respecting Unity's color scheme.
Similarly, you can easily add Orbit markers
to Unreal's existing profiling macros
in order to visualize Unreal's instrumented functions
in Orbit.
Another interesting feature is the disassembly view,
on which results from sampling are overlaid.
This is useful to visualize hot instructions.
Similarly, the source code view allows to map samples
to lines of code.
Coming back to the capture view, we
have function iterations that allow
you to define a frame that span consecutive function calls.
In this case, we've used vkQueuePresent
as our frame delimiter.
We can then easily navigate from frame
to frame in the capture view.
So on the subject of vkQueuePresent,
while Orbit is primarily a CPU profiler,
we do expose some GPU driver events.
We have dedicated GPU tracks where
you can see when a job was in queue to the driver,
when it was in queue to the GPU hardware
and when they were actually executed on GPU.
You can also use our Vulkan Tracing feature
to visualize Vulkan debug labels directly
in Orbit alongside all the other profiling information.
That being said, for a more in-depth GPU analysis,
we do have a dedicated GPU profiler, Dive,
which Michael will now present.
MICHAEL ANTTILA: Thanks, Pierric.
A GPU profiler is critical for developing or porting
resource intensive games on any platform,
and Stadia is no exception.
When developing Dive, our goal was
to create a profiler that would give you more information
than existing GPU profilers, both at the low level
and at the high level.
We recognize that there are two groups of game developers
in our target audience.
Those who are optimizing the underlying graphics engine,
and those who don't have any control
over what the graphics engine is doing.
I want to show that Dive can be a very useful tool to both
of these groups.
Dive was designed to be used fullscreen
on a large monitor in order to provide
the best experience possible.
So for these slides, I will only be
able to give you small glimpses into the big picture.
With that in mind, let's begin.
Dive is a single frame profiler.
You simply start your game up on Stadia,
launch Dive on your developer machine,
and hit the capture button.
Dive will capture performance information for a single frame
and then transfer that data back to your workstation
for detailed analysis.
Dive has a typical timeline view of everything
that happened during the frame and an occupancy chart which
gives you immediate insight into how busy the GPU was
at each point during the rendering of the frame.
Existing profilers have occupancy charts,
but an overall occupancy chart can only
give you so much information.
What does 50% occupancy actually mean?
Are half the compute units idle or are all compute units
being partially utilized?
These two situations require very different optimization
methods.
For game developers who need to know exactly how their graphics
engine is using the hardware, existing profilers
fall short of giving them enough information
to answer questions like this.
Dive allows you to drill down deep into the GPU,
into each shader engine, each compute unit, and each SIMD.
At each level, you can see exactly
which resources were in use and how occupied they were.
We also have instruction tracing.
So Dive can show you when each individual shader
instruction was executed and exactly what
that instruction was.
For graphics engine developers, this
is the level of detail you sometimes
need to track down an issue.
Dive shows you the state of the hardware
context in a timeline view and can automatically
highlight events that were stalled due to context rules.
This is just one of the ways that we
try to bring your attention to potential performance issues
as quickly as possible.
At a higher level, Dive also captures
which Vulkan calls were made during one frame
and displays them in an event list on the side.
One feature that sets Dive apart is
that we capture all of the Vulkan calls,
not just the ones that result in GPU work.
This extra context is critical to understand
what is happening.
Previously, developers had to perform an additional capture
of a frame in render doc just to understand
the context of what they were seeing in their profiler.
This feature has been a huge time saver.
Debug labels are captured and displayed
in both the event panel and the timeline panel
for easy cross-referencing.
Debug labels, Vulkan calls, and GPU events
are organized into a collapsible hierarchy for easy navigation.
And yes, we have a search bar, so you
can find the exact function you're looking for.
Every parameter to every Vulkan call is captured.
So there's no guesswork about what is happening.
Although Dive is not a debugger, this feature
can be useful by itself for figuring out
where things are going wrong in their graphics engine.
We capture as much information as we can.
You can view the contents of every command buffer
that was sent to the GPU.
This has proven useful when trying
to get to the bottom of some really interesting timing
issues, including allowing you to see which copy buffer
calls are going to DMA and which are being dispatched directly
to the GPU.
In addition, you can view the GPU state at any point in time
during the frame.
This can be useful to verify assumptions about exactly what
state the GPU is in.
Dive can also capture performance counters
from the GPU.
Sometimes just staring at a timeline of events
isn't enough to give you insight into where the bottlenecks are.
Performance counters can help identify draw calls
that might be problematic.
For example, sorting by the amount
of time spent in the vertex shader
will instantly take you to the call that is doing the most
vertex shader work.
This can be a very quick way to determine
if polygon counts need to be reduced for some models
in order to hit performance targets.
For developers who don't have access to the graphics engine
code, this may be a good opportunity
to go back to the art department and see if they can simplify
assets to improve performance.
In this example, looking at the counters for this call,
Dive can let you know that only 43% of the vertices
actually made it to the final output.
Maybe this is an area where some calling can happen on the CPU
or in a compute shader to avoid having the vertex shader waste
time on all those unused vertices.
Graphics profiling can be a challenge.
And we wanted to make Dive as helpful as possible.
We have a context sensitive help system
which gives you useful explanations and advice
as you hover your mouse cursor over various elements
in the UI.
In this case, hovering over a barrier
reveals what that barrier is doing
and how it could potentially be optimized by doing
things a different way.
We also have a problems tab, which
can identify certain common performance issues that we
see frequently in games.
In this case, Dive has noticed sets of barriers
with no draw calls in between them,
which means they could theoretically
be merged into a single barrier to reduce overhead.
Hovering over the mergeable barrier set
brings up context sensitive help text
that gives you an estimate of how much time you
could potentially save by merging those barriers.
In this case, 0.11 milliseconds, which
could be significant savings if you are close to the edge.
Our ultimate goal is to automatically find
as many performance issues as we can on your behalf to help
save you time and help new developers learn
about what factors can reduce rendering speed.
We plan to add more static analysis features
like this in the near future.
In addition to profiling, Dive provides
a user-friendly interface for analyzing GPU crash dumps.
Dive integrates with the graphics flight recorder
to provide as much information as possible about GPU crashes.
Graphics Flight Recorder, or GFR for short,
is another tool available in the Stadia SDK that
can log the state of the GPU as it was just before it
experienced a hang or a crash.
Using data from GFR, Dive is able to quickly highlight
which Vulkan calls completed and which did not.
In this GPU hang case, Dive was able to instantly highlight
that a copy image operation did not complete for some reason.
Drilling down to a lower level, Dive
can identify which shaders were in play when the GPU crashed
and identify which individual instructions were
in flight across the GPU at the time of the crash,
hopefully helping you narrow down
the exact cause of the crash as quickly as possible.
We hope that by giving developers access
to as much information as possible
and organizing it in a way that makes it easy to analyze,
we can make profiling and optimizing your game on Stadia
an enjoyable experience.
And we are looking forward to hearing your feedback
on these tools as you use them.
Thank you for watching.
There is an open-source version of Orbit available on GitHub.
And both Orbit and Dive are included in the Stadia SDK.
PIERRIC GIMMIG: Thank you for watching,
and please try our tools out.
[MUSIC PLAYING]