Hello, everyone. I'm Scott Carbon-Ogden.
I'm a product manager at Google based in London,
and working on the Android games team.
One big project our team is working on
is the Android Performance Tuner.
Today I'll take you through some key considerations
when adopting the Android Performance Tuner,
APT for short,
and walk you through how to interpret the data
that you get back from APT.
Just to refresh your memory, or for those of you
who perhaps haven't heard of ATP previously,
Android Performance Tuner enables you to deliver
the best possible experience to each of your users.
It does this by helping you measure and optimize frame rate stability
and graphical fidelity across Android devices at scale.
It helps you to identify performance issues in your game or app,
and also highlights opportunities to improve your graphical fidelity.
It has impact metrics to help you prioritize,
and issues are categorized in a way to help you take action.
It's now ready for integration into your game
and works with almost any C or C++ game.
If you use Unity 2017.4, it's even easier with our Unity plugin.
Although for advanced information, like GPU time,
you need Unity 2019.3.12 or newer.
For those of you on the Unreal engine,
you need our frame pacing API, which is integrated as standard
from Unreal 4.25 onwards.
So today we're going to cover how to configure the ATP for your game,
and then we'll go through an example of how to interpret the results
from the Android Performance Tuner.
So frame time is at the core of the Android Performance Tuner.
It's the variable that you're trying to optimize,
so that your game is both fun and engaging,
while still looking as visually appealing as possible.
Frame time is the time that it takes to render one frame of the scene.
It's the inverse of the frame rate.
So, a frame rate of 10 frames per second
requires a frame time of 0.1 second, or 100 milliseconds.
When frame times increase, the frame rates drop,
and the game becomes frustrating and unplayable.
This is shown in this example.
On the left, we have an animation at 10 frames per second,
and on the right is a smooth 60 frames per second.
Let's dive a little deeper now into how this
might be defined and measured.
So, what do we choose to define as frame time?
Should it be from when a frame is displayed on screen
until the next is displayed?
Should it be from when the CPU starts
rendering the first frame until it starts rendering the next?
We have three options with Android Performance Tuner.
Firstly, you can choose to use the frame times
from the Android Frame Pacing API.
Secondly, if you have your own Frame Pacing library,
you can use its output to measure frame times.
And lastly, for older versions of Unity,
the plugin will use Unity ticks.
First, let's look at using the Android frame pacing API,
or Swappy, as it's sometimes known,
to provide measurements for each frame.
The Frame Pacing API provides the raw amount
of CPU and GPU time spent working on each frame,
so that you can see if you are CPU- or GPU-bound.
It'll also help you know how much you headroom you have,
rather than just seeing the paste frame time.
To get all these details, though,
you'll need version 1.3 of the Frame Pacing API,
which is integrated into Unity 2019.3.12, Unreal 4.25,
or available for standalone download.
If you choose to use your on frame pacing,
you'll want to provide CPU and GPU times from your frame pacing system,
so you can get these raw times
and see what's actually consumed rather than just the pacing time.
If you're using an older version of Unity, APT will use Unity ticks.
This is the signal from Unity near the start of each frame.
It only looks at the time between the rendering of frames
and the CPU, so you won't get a breakdown of CPU versus GPU times.
And if your frames are paced so they render at regular intervals,
you'll only get that paced time.
Annotations provide you with a way of marking up parts of your game
so you can see how these parts affect frame times.
Loading is the most common annotation.
We provide a special loading annotation
that lets you exclude loading screens from your overall frame time metrics
so that these won't be representative of gameplay.
Similarly, you should use annotations to differentiate
any menus in the game from the actual gameplay
so you can focus your attention on where it matters.
You can look at the metrics for frame time on the menus,
but you'll be able to slice them out from the actual gameplay.
We recommend that you use different annotations for each game level.
There's a feature in our Unity plugin
to do this automatically if you use Unity scenes.
And this way, you can see if there are some levels
of performance lower than others
and take the appropriate action, such as decluttering the level.
If there are elements in your game
which you think might have a significantly negative impact on frame rate,
you can annotate these as well.
You might annotate a new character with a particularly complex mesh,
a success screen with many particle effects.
You can later slice out just these parts of the game
to see if they really are causing an issue or not.
Lastly, you might want to annotate some signals.
It'll help you identify the causes of issues.
You could add annotations for port network connections,
for sessions that had a failed device anti-abuse check,
or for sessions where the device has overheated.
Quality levels are a collection of fidelity parameters
that make a meaningful group.
You might think of them as low, medium, high quality,
with low having the lowest quality textures
and many effects turned off, and high having everything turned on
and the finest textures.
In APT you assign each quality level a number,
rather than a name, so that they're clearly ordered.
And you can have as many quality levels as you'd like.
You can see here that we have a higher quality level on the left,
with reflections and extra vegetation,
and a faster, yet lower fidelity version on the right.
The differences can be subtle.
Just some extra reflections, a few more props.
But they can have a big effect on frame rates.
For most games, you will likely only want a few of these levels
to balance the testing overheads with a broad range of device capabilities.
Once you have your list of quality levels, you'll need to set one on each device
and let APT know which quality level this device is using.
You might have dozens of fidelity parameters.
And so by grouping them into meaningful quality level,
it's easier for you to later interpret the results coming back,
and to consider which settings different devices should have.
Whereas quality levels are coarse overall bands,
fidelity parameters are the fine-grained attributes
that go together to make a quality level.
Some of these, like mesh quality or texture bit rate,
might affect other metrics as well, such as memory or loading time,
whereas some like shadows and particle effects
might largely affect frame rate.
As the developer of your game,
you're in the best position to know what levers are available to your game
to fix performance without impacting game play.
But here are some ideas to provide some inspiration.
You might consider mesh fidelity.
Effects, light reflections and shadows.
Extra props in the scene that don't affect gameplay.
Optional particle effects.
And most importantly, texture quality.
Texture quality can be a really impactful fidelity parameter.
Here you can see the same ASTC texture encoded with three different block sizes.
The lower the block size, the faster the frame time,
the lower the memory consumption, and the quicker the load time.
This can be very important for low end devices.
You can barely notice the quality difference,
but these more heavily compressed assets take up less than one third of the size.
Now that your device has a quality level with a set of fidelity parameters
and is recording annotated frame times,
it's time to think about uploading that data.
To save your users' memory and bandwidth, instead of recording every single frame,
we group them together into buckets
and upload the whole histogram of buckets together
at regular intervals.
You can configure these buckets if you'd like,
but the defaults are recommended for most games.
The data is uploaded to Play servers
and then will be collated and made available in Play Console
within a few hours.
The simplest way to configure the upload is to set an upload interval.
With this, you can set a time, say every 10 minutes,
after which, the data will be uploaded.
During development, you'll want to make this number smaller,
so that you can validate the output more quickly.
In production, you'll want to upload less frequently,
so that your users use less bandwidth.
You should also consider flushing the remaining data
when the game gets backgrounded or is closed.
It might be that, for some users, the frame rate gets so bad,
that they quit the game.
For those users, catching the final few seconds
would be particularly useful for uncovering the issue
and improving retention.
Before I move on to analyzing the data,
I wanted to make you aware of some common issues
so that you don't get stuck.
Sometimes, if you've just integrated and you go into the dashboard,
you'll get an error message saying "not enough data."
Or you might only see a few data points.
Don't worry. This might not be an issue.
It might be that you don't have enough data points
to get over our privacy thresholds.
Just wait a few more days
and make sure that the display is set to show seven days' worth of data.
If you release to an internal test track, there won't be a privacy threshold,
and you should see the data straight away.
If you release on any of the other test tracks,
you might not have enough users to cross the privacy threshold
until you do a full launch.
It's really worthwhile to set up both annotations and quality levels
in your first launch.
Otherwise, it can be hard to tease apart and analyze the data.
We'll see more on this in the interpretation section.
Lastly, if you receive a 400 or 403 error as you integrate the Performance Tuner,
it's probably an issue with your API key.
Go back to the cloud console and check all your settings.
Don't forget, you can use the included validation tool.
Check that all the right data is being uploaded.
Now that we talked through the configuration,
let's go for an example of interpreting the data
to see what it looks like.
The first steps you take with APT after integrating
is to get a lay of the land.
Open up Android vitals in the new Play Console
and look for Performance, then Insights, in the side menu.
When you do start to see some data, the first thing you want to look at
is how your frame time is compared to your target.
You can set the frame time target through this screen as well.
30, 60, 120 fps,
whatever you think is a realistic and optimal target for your game.
Games with fast frequent motion should target higher frame rates.
It's up to you and your team
how you want to interpret these statistics.
For your game, is 1% of frames running slow bad?
Or is 10%?
I think we can all agree that 90% of frames being slow isn't ideal.
Research shows that user retention starts to drop around the 10% mark.
But there may be undesirable outcomes before that point.
You can also use this high level overview to decide where to focus.
Within the chart, you can see impact.
This is the percentage of all slow frames
that are attributable to a certain problem.
If you see that devices are the main impact on slow frames,
it means that your game is running reasonably consistently,
but some devices are running slow.
This might be an indication that some devices
are running on the wrong quality level for their capabilities.
You can see in this case, this device is on quality level five,
the highest quality level, that are underperforming.
Of the 17 devices at quality level five, 11 are underperforming.
You should consider changing the quality level of those devices,
or even splitting quality level five into two quality levels,
to better cater to the two groups of devices.
If you see that annotations are the main impact on slow frames,
then it might be that your game runs reasonably consistently on most devices,
but there are a few levels or situations where the game runs more slowly.
You might want to go and reassess those parts of the game.
Sometimes there'll be situations where the slow part of the game
is running on a slow device, and so it'll count in both buckets.
This means that the impact from device model issues
and annotation issues might add up to be even greater than 100%.
This is what you can see in the example.
We also see that this particular object, character protagonist,
is present where most of the slow frames are happening.
But they're primarily only affecting three of the five quality levels.
You'll need to dig into this more in the annotation screen
to see which quality levels are affected.
It could be that this annotation uses some effect heavily
that is only found on higher quality levels.
Conversely, it could be that just the low end devices
struggle with his level of detail, and need a simpler mesh or texture.
We'll talk more about the annotations screen shortly.
The next section of the APT
is where we visualize the performance of sessions in your game.
Each dot corresponds to a device.
The bigger the dot, the more sessions around that device in the time period.
The x-axis is how long it took to create each frame.
And the y-axis is the quality level that device was on.
You can see this orange line here at the target frame time.
In this example, it's 33 milliseconds, or 30 fps.
It's a lot of information to take in, but this graph can be very powerful.
Seeing devices in this space, to the left of the orange bar, is good.
It means these devices are hitting their target,
but there's not a lot of wiggle room.
They're at the right level.
These devices down here on the bottom right
are missing the targets at even the lowest quality level.
With these devices, you have three options.
Ignore them.
And this isn't great, because you'll get bad ratings from these users.
You could exclude the devices.
But that isn't great either, because it mean fewer users.
Or lastly, you could try to make a whole new quality level
with even lower fidelity parameters.
Half their texture quality, blockier meshes, fewer effects,
and so on.
These devices over here on the far left, these are massively over-performing.
Here, they're rendering frames at maybe 10 milliseconds.
These devices could still produce fast frames at higher quality levels.
So you should consider moving them up.
If we had some here at the very top left,
it would suggest that you have room for higher quality levels
with finer textures and more effects.
There's also an equivalent graph for annotations.
So you can see how your annotations stack up compared to one another.
With the annotation view, you can also see
which annotations had the largest portion of slow frames.
You can skip over annotations, such as loading or menus,
and just focus in on the gameplay bits,
as this is where frame time matters the most.
If you find a few scene or object annotations
have a high percentage of slow frames,
these are the bits of the game you should discuss with your design team.
Maybe the game level is too complex.
Maybe the objects should be simplified in some way.
You might have added annotations in here to have temperature.
If the game frequently performs poorly because the device is running hot,
you could think about how you could reduce your thermal loads.
You can put an annotation in to indicate
that you're doing background loading during gameplay.
If this is the case, a large percentage of slow frames here
would show you that you're impacting gameplay,
and should consider if you're better off to do more loading up front.
Back in the Devices tab, if you click on one of the quality levels,
you'll be taken to the device list.
Here you can see how various SoCs and devices perform with your game.
Beneath the SoCs, you can see devices.
You can see how popular they are with the number of affected sessions.
You can see their percentage of slow frames.
And you can see their 90th percentile GPU in frame times.
If you're looking to improve frame rate, what I would recommend to do here
is filter for egregiously bad frame times.
For example, 40 to 60 milliseconds.
And then sort by the number of the affected sessions.
You might just find that you have one or two devices that perform very badly,
but are very popular.
You can then go acquire these devices
and use a profiling tool to find the issue.
If there are a range of devices that are problematic,
you might want to download the list of device models here.
This produces a CSV of the problem devices,
which you can then use to adjust the quality level of those devices.
If you're already at the lowest level,
you can look at the GPU time and frame time,
to decide if you're GPU-bound or CPU-bound.
That might help you think of some ideas for new quality levels.
Is it more that you need to simplify the simulation step?
Or are there some shades that you should simplify?
You can also consider
switching to a more flexible texture compression format,
such as ASTC.
With ASTC, you can specify the block size.
And so it's easy to create a new set of lower fidelity faster assets.
Once you fix the badly underperforming devices,
you'll want to look at where there's headroom.
You could filter for devices rendering in 1 to 20 milliseconds.
You might find that there's some new devices in there
that your QA team hasn't yet picked up and moved to a higher quality bucket.
You can do a similar CSD export
and then update all of your over-performing devices.
If you find that your over-performing devices
are at the highest quality level,
you could consider producing even higher quality assets
or turning on even fancier features for some of these devices.
So, to recap what we've talked about today,
we discussed how we define frame time,
and the three different approaches to providing data to APT.
We talked about annotations
and gave some examples of what you might like to annotate.
We touched on quality levels and how you might group fidelity parameters.
We discussed your options for uploading tools.
We looked at some common issues in the integration process.
And lastly, we walked through an interpretation of the data.
I really hope that you found this overview useful.
We think that we've made APT relevant to almost any Android game,
but especially relevant to games where frame rates are important.
If you're interested in finding out more or downloading and integrating the APT,
then check out the link in the description below.
Thank you so much for your time today,
and I hope you're able to improve your game performance
using Android Performance Tuner.