Hi, I’m Amanda, a product manager working on Android and performance libraries.
Today, I’m going to talk about measuring jank and startup
with a new Jetpack library: Macrobenchmark.
And later in the video, Carmen, a performance engineer,
will walk through some debugging.
We’re really happy to share what the team’s been working on
to help you make performant apps that your users will love.
First, let's get some definitions out of the way.
Measuring Startup is exactly what it sounds like.
How long does it take for your app to start running?
Next, what is jank?
For most devices today, the screen will refresh 60 times per second or 60 hertz.
If there’s some motion on the screen, such as scrolling, transitions or animations,
the app should create 60 frames per second to match the refresh rate.
Jank is any stuttering, juttering or just plain halting
you see when an app isn’t keeping up with the refresh rate.
Basically, jank is the result of an app taking too long to make frames.
Now that that’s out of the way,
let’s talk about why measuring performance is important.
When you're building an app,
you want users to experience the app the way that you intended.
You don’t want users to wait idle while screens slowed,
and you also don’t want them to see jank while scrolling.
Apps that aren’t performant usually have fewer people using them
and usually spend less time in the apps.
Let's take a quick look at an example.
Here are two scenarios, an app that is slow and janky,
and a really performant app running fast and smooth.
On the left, when the user tries to start up the app,
it’s slow and takes a really long time to get started.
Then when the user scrolls, the app drops frames, which leads to jank.
In this case, a user can quickly get frustrated and just abandon the app.
On the right, the app does optimize performance,
and has really fast startup times and smooth scrolling.
In this scenario, the user can quickly engage with the app
and have an uninterrupted enjoyable experience.
So obviously, building performant apps is a nice goal to have,
but there are several problems that make it difficult for developers.
One problem is that you cannot simply rely on analytics,
which allows various app metrics
including app stability, startup time, and more.
Metrics from the field are slow,
since you have to wait for the app to hit users and aggregate from them.
We want to make and measure improvements locally while experimenting.
Another problem is testing app performance is hard.
With benchmarking, we want to know how long it takes to run some chunk of code.
We want to measure it in a way where we see the improvements
reflected in those measurements.
And unlike profiling, we are not running the full application.
What we need is a tool for rapid feedback loops
so we can iterate quickly.
Benchmarking is such a tool
that can help focus developer efforts during performance testing.
There is an existing Jetpack benchmark library
that does help prevent common measuring mistakes,
but it cannot help with measuring startup or jank.
It supports micro benchmarks, which does not cover the majority of use cases.
For example, it focuses on CPUs as opposed to startup time.
Another problem is that it's difficult to measure performance in a stable way.
It's tricky.
You don’t want to measure the wrong thing at the wrong time.
It can be unstable, there's variance, how many loops should you run,
the system can be complex and noisy, and it's hard to set up correctly.
To address these issues,
we’ve made a new testing framework Macrobenchmark.
It's intended for validating your changes and identifying regressions,
both locally and through continuous integration.
We're now supporting startup and scrolling measurement at launch.
And in the future, we plan to expand the library to support other metrics.
Macrobenchmark takes some of the ambiguity of whether you’re measuring correctly
and turns them into parameters that you can set
such as different types of startup and different types of app compilation modes.
Let's dive into different scenarios
that you can build yourself using these components.
First, what you need to do is in a JUnit test, add the MacrobenchmarkRule(),
then create a test method called measureRepeated(.
MeasureRepeated( will run the block pass to it several times
and measure performance of your app.
This is a startup test,
so we’ll reset device state by pressing the home button.
Then we’ll launch an activity and specify it, by customizing the intent we pass.
In this case, we’re setting a unique action to launch our specific activity.
Now we’ll configure what we want to measure.
First, specify the package. Next, specify the metrics we want to collect.
Again, this is a starter test, so we’ll use that metric to measure launch time.
Frame timing metric is also available
for benchmarking scrolling and animation frame performance.
Next, specify how the app should be compiled.
On Android, apps may be entirely uncompiled when running the first time,
or they can be partially compiled, or fully compiled.
There are three major types of startup that you may want to measure:
hot startup, which is resuming an existing activity.
It’s already in memory, just presented again.
Warm startup is constructing and starting an activity but within an existing process.
And cold startup, which is launching from a process that isn’t alive
so the entire process must be initialized.
For example, a user launching your app for the first time that day.
The last step is to define how many times the measurement should occur.
Higher means more stability and confidence; lower is faster.
So how does it work?
At a high level, you set up your Macrobenchmark,
run the test, the results are processed, and then the results are presented to you.
Let’s take a look at what’s happening under the hood.
First in the setup step, we kill the process and compile it in the mode you requested.
Then during each iteration, we start a systrace recording,
and then run the test.
Systrace is a useful tool for seeing what the thread is doing and when.
Carmen will talk more about systrace later in the video.
Then to perform a measurement iteration, startup is triggered during a system trace.
The system trace is stored and used to compute metrics.
Each iteration will produce a systrace and metrics.
Macrobenchmark outputs this in multiple ways.
First, the results are stored as a JSON file for CI usage.
Traces are also stored to assist in debugging performance problems,
and a summary is sent to Studio for local investigation.
Now I’ll pass the Carmen to walk through analyzing the test results.
Thanks, Amanda.
I’m Carmen from the Platform Performance Team,
and I’m here to talk about understanding your Macrobenchmark results in Android Studio.
Android Studio includes support for Macrobenchmark,
starting with Studio Arctic Fox.
You can run a Macrobenchmark in Studio
and see the results in the output panel at the bottom of the screen.
You can see the min/max/average of the results running that Macrobenchmark,
and the results are actually links.
When you click that link, it’ll take you to the Studio profiler,
and you can inspect the systrace of that particular iteration.
So in this case, we’ve grabbed the trace from an app startup Macrobenchmark test.
A systrace is a recording of all activity that happened on the device during the test.
Each of these sections is a tracepoint,
which is instrumentation that records how long each section of activity took to run.
We’ll zoom in on the activityStart tracepoint in this case,
which is provided by the platform.
Now when debugging app startup, we recommend that you step through the trace
and make sure that everything you see is accounted for in your app.
If there’s spaces where you don’t know what’s going on,
or why something is happening, add more tracepoints.
So in this trace, we see resource loading, which is expected,
you can actually see in your app code when you access those resources,
view inflation, which is also expected,
and you can track down those inflates in your app code as well,
and some icon loading.
Now we know, because we’re the app devs, that we have three icons to load.
But this doesn’t look quite right, so let’s zoom in and take a closer look.
The loading icon tag is actually a custom tracepoint we added to this app.
And inside each one, we are decoding a bitmap.
This is a little bit weird because we expected to have three icons
that are pretty much the same,
but instead we have one that’s taking a super long time compared to the other two.
So let's dig into why.
Now you could start off by taking a look at your app's code
to see if you can see what’s unique about the image file
that goes with the icon that’s loaded last.
Fortunately in this case, there happens to be a tracepoint for opening this asset,
so we can take a shortcut.
So here we can see that this icon is a JPEG, which is probably an accident.
As the app developer, we know that we expected the icons to all be PNGs.
This is a pretty easy fix; you can go into your app and update the icon image file
with one that’s optimized to be like the rest.
Now that you fix the app, run another Macrobenchmark test
and scope out the trace.
You can see right away things are looking a little bit more uniform,
zoom in, and the loading icons are now consistent with each other,
and that last one is now the same size as the other two.
So mission accomplished.
Now there’s a solid collection of tracepoints
provided by the Android platform and the Jetpack libraries
that include basic app startup, activity life cycle and widget events.
They’ll get you pretty far, but it’s not everything
and where this system really shines is when you add your own.
To add a basic tracepoint,
all you have to do is include the Androidx.tracing library.
In tracing-ktx, we provide an inline trace function
that will automatically wrap a block with begin and end trace sections.
You can time anything you want just by wrapping it in a trace method.
There’s a few other functions in this library for marking tracepoints
that start and end on different threads so you don’t let that hold you back.
A few other handy tools like counters.
The Macrobenchmark library is currently available in alpha.
While we continue to stabilize it, please check it out and give us feedback.
We have a sample app on GitHub you can benchmark
and use to try this out on your own.
Thank you again for joining us; we hope you enjoyed our session
on measuring jank and startup with the new Jetpack library Macrobenchmark.