[MUSIC PLAYING]
DUSTIN LAM: Welcome everyone.
This is Fighting Regressions with Benchmarking
and Continuous Integration.
I'm Dustin, that's Chris, and we work on Jetpack libraries.
Now, early this year, we released Jetpack Benchmark,
which is an awesome library and tool that's
going to help you measure performance in app code.
And we did a talk at I/O, and we talked
a lot about how the library works
and what it can do for you.
But what we didn't talk so much about
was how we actually use this internally within Jetpack.
And it's been really helpful for us, especially
for performance-sensitive projects like Compose.
And so we wanted to share some of that value with you.
And hopefully, you come away with some useful knowledge.
So story time-- who doesn't love story time?
So here's a little lifespan of a benchmark.
So each of the blue dots represent
a build or a set of changes that might have made your benchmark
perform better or worse.
And along the y-axis, we have performance-- so how long
the benchmark took to run.
In this case, higher is worse.
The benchmark took longer to run.
And so here you can see we're developing, we're developing,
things are going great.
And of course, this wouldn't be a useful example
unless we actually hit a regression.
And as you can see from a cursory glance,
things definitely got much worse after that build,
highlighted by the red line.
Now, this is actually a real example,
so we actually look at the code.
It turns out this is the initialization benchmark
for a class called GreedyScheduler
in the WorkManager class.
And this is just a constructor.
It doesn't really matter what the class
does for the context of this example.
But you can see here that it's just some basic assignments,
we're creating a simple Java object, no harm done.
But the actual regression added this line here--
getProcessName.
Now, during code review, we thought
this would be a relatively quick thing, it'd fast, no problem.
It turns out, no, there's a huge regression, it's an issue.
Fortunately, it's only needed in this method called Schedule.
And that happens after initialization,
so we happy to just move it down, make it lazy,
problem solved, regression fixed--
yay!
But the key takeaway I really want to pass on here
is that hindsight's 20/20.
Regressions are rarely obvious.
I think we, as developers, tend to think of performance as kind
of like an afterthought.
There's this old saying where premature optimization,
is that something you want to be worried about?
And then that's kind of true.
There's some value in that.
But then what happens is, we often
neglect performance entirely.
And it ends up being really cumbersome and painful to have
to deal with it the end of a dev cycle.
Maybe your quality assurance team or someone
dogfooding the app mentions hey, there's
this key interaction or this feature
that's like kind of jank, kind of slow.
You pull out all your profiling tools,
you search up Google, what should I do?
And you have to dig through thousands and thousands
of lines of changes to figure out what went wrong.
And compare that to the example we just saw.
Maintaining performance is actually a much easier thing
to do.
And what we really want to do is run benchmarks
after every change.
And that's helped us a lot within Jetpack.
So benchmarks are great, but how do we write them?
So our solution to this is Jetpack Benchmark.
And as I mentioned earlier, we did a full, deep talk
about this at I/O. But I'm just going
to go over a very quick rundown here just for the uninitiated.
So here's a simple Android instrument to test.
It's called simpleScroll.
This is going to be a UI benchmark.
And we're going to use the activity test rule here
to launch an activity.
It's going to handle all that activity lifecycle stuff
for us.
It's going to be a UI test, so we
need to run on the UI thread.
We're going to get a handle to our recyclerView,
get the height of the last item, and then scroll by that amount.
So just a simple insert to test that
scrolls by the height of one item.
Now, to actually measure this, we're
going to get an instance of our benchmark rule, which
comes from the Jetpack Benchmark library,
and then take all the code that we want to measure and wrap it
in this measure repeated loop.
And the library's going to handle looping over this code
until it's confident that it has a stable result.
So that's great.
And because it's an instrumented test,
we can run us directly in studio and get the results directly
in our ID.
So that's nice too.
But that all seems pretty simple, right?
Why have a whole library out there?
Everybody knows how to write a while loop.
Well, we've got a bunch of baked-in tricks for stability.
So we're going to handle things like CPU throttling, thread
priority, warmup for git.
And those are just a few examples.
But we've also got a bunch of configuration checks
to make sure that you're optimizing something
that's actually going to end up paying off to your end users--
not something that's just going to get
optimized away by R8, for example.
And because we were Googlers, we get to work directly
with the studio team.
And so we've got some pretty tight-knit integration
with Studio.
And it's going to help you iterate faster
on those benchmarks.
So that's great.
But benchmarks aren't the only thing that are available to you
in your performance toolkit.
The first thing you might jump into
is something like local profiling.
So these are the kind of tools they
would use in your local deployment environment.
And they're going to take advantage of debug ability
and give you more accurate, more detailed diagnostic data.
It's going to give you insights into where your critical code
paths are.
And so some examples of this are like systrace, method profiling
pictured below.
But this is a manual process.
And that's not necessarily a bad thing.
It's just that if we want a measurement that we
can iterate on and maintain, then
we're going to need to look a little further.
So that brings us to analytics.
Analytics are another obvious thing that we can try.
And these are performance metrics from real users.
And it's easy to set up ad monitor.
And I know a lot of people who deal with analytics
are probably rolling their eyes right now.
But given an existing analytics pipeline,
adding an additional metric is usually not
that much of an additional cost.
And most people have analytics in their app.
So it's kind of nice to be just be
able to add an additional metric and get up and running.
And you can integrate directly with your existing analytics
dashboards.
But you're going to have a slower feedback
loop as a result of this.
You're going to have to wait for your app to get pushed out
for users to adopt it and then have enough of those users
to hit that critical code path until you get
a statistically-significant amount of data they
you can actually iterate on.
And the worst part of this is that your users are actually
going to hit these regressions before you're actually
able to solve them.
So if you care about users, maybe this is not the only tool
that we need to consider.
And that brings us to benchmarks.
And by benchmarks, I really mean separate thing
that you have to develop that's not
core to the functionality of your app.
It's going to measure your code in a loop.
They're kind of like the performance analogy
to correctness tests, which is a really hand-wavy thing.
Because correctness tests give you this pass-fail signal.
But benchmarks give you this measurement-- a scalar value
that you can look at.
But like tests-- these are things
that we can run side-by-side as we're
developing to maintain code.
And it's going to allow us to iterate quickly.
We can tweak things in our app, run the benchmark,
tweak another thing, run the benchmark again,
see how things changed, whether better or worse.
So that's nice.
And this is something that we can run as we're developing
and as we create builds in our continuous integration system
in our development workflow.
And we can actually monitor and fix these regressions
before they hit production.
So that solves that problem.
But there's a higher development cost here.
It's a separate pipeline that you have to invest in.
Benchmarks are a separate piece of code
that you're going to have to write and maintain.
So that's kind of the downside there.
And so the typical workflow might look something like,
you get a report from quality assurance
or maybe a user review.
Then you'll take that to local profiling tools,
dig in, figure out what your critical code paths are,
write some benchmarks around that,
iterate it, make it better, then write
some analytics to make sure that things actually
got better for your users.
But those are just some pros and cons of different tools
that are available.
But this talks about benchmarks.
And that's a little bit about why
benchmarks are useful and benchmarks in general.
And now Chris is going to tell us a little bit how
they work in CI.
CHRIS CRAIK: So Dustin talks about how benchmarks work
and what we'd like to get out of them.
So let's talk about how we actually go
about using benchmarking in CI.
So let's look at the goals here.
First and foremost, we want to maintain quality
as changes land.
Similar to running tests in CI, why
we want things to stay good instead of having to fix them
over and over again.
We want to also provide blame granularity
on regressions that's relatively detailed.
We want to be able to see, oh look,
we have these three changes that landed.
In between these two, performance got worse.
That's really what we're after here.
And then we also want to try and reuse as much
as possible infrastructure from correctness tests.
Because if you have correctness tests running in CI,
we'd like to be able to reuse much of that for benchmarking.
So let's talk a little bit about correctness tests
here, just to set up what we're talking about.
Here is an example test dashboard.
This is actually the test dashboard
for Jetpack as of a few days ago.
And you can see that, although we have tens of thousands
of tests running in CI, we actually
have a very nice, concise view of that
because correctness tests scale pretty nicely.
You really don't care about all the tests that are passing.
You really only need visibility into the ones that are failing.
And so we collapse all the packages that all
of our modules that don't have failing tests.
And we only show those that do.
And that allows you to say at a very quick glance,
there's one flaky test in this one package
and one failing test in another--
really easy to see at a glance.
However, let's talk about what this would look
like for benchmarking.
How do we get a benchmarking dashboard
that says the exact same amount of information concisely?
Well, first of all, benchmarks, as Dustin mentioned,
are a scalar value.
It is some sort of floating point number.
It is not a pass or a fail.
It is really easy to collapse Booleans in a way that
doesn't work here.
We would like a test dashboard to work
with pass-fail signals like that, but we don't have those.
So how do we get a pass-fail signal out of a benchmark?
So let's go ahead and look at all the benchmark results
from the entire suite of Jetpack libraries
and maybe we can see some patterns here.
Maybe we can discern some of this visually.
Because after all, you're going to have way
fewer benchmarks than you are going
to have correctness tests.
You only really need benchmarks around the critical parts
of your code that need to stay fast.
Now, there's a whole lot of information here,
but really, you've got room benchmarks at the very top that
are taking seconds to run.
Whereas, there are tiny little benchmarks at the very bottom
that take nanoseconds.
So let's scale this around, let's try
and normalize this a bit so that maybe we can see some patterns
pop out there.
Oh my goodness, no!
That is entirely useless.
There's just so much noise in all of these things.
And what really is happening here
and that I want to call out is that a lot of these
are individual noisy benchmarks.
But the work manager sample that Dustin showed before--
you've got tests that are stable and useful running in here.
But the noise just makes up so much visual noise
that you can't see them.
So we've got to have some sort of more automated way
to get through this.
So how do we detect performance regressions?
So let's restate the problem here.
Manually inspecting that will absolutely not scale.
Benchmarks return a scalar value.
And we need to somehow get a pass-fail from that.
So a naive, simple approach to this might be saying,
OK, well, I really only care about big regressions.
Small ones are likely going to be noise.
So let's set a threshold that's a percentage.
What if I say 20% is a failure?
If anything goes up by 20%, then that's a failure.
Well, here's a sample benchmark and that particular pattern
works pretty well.
We've got a big regression here in the middle,
plus 88%, that's a huge regression.
We've got a little bit of noise afterword.
But that's on the scale of 13% and 12%--
so, not a big deal.
In this particular case, regression is caught.
And the little minor fluke that happened there that
doesn't continue on after that--
great, we're fine.
We caught the regression and we ignored the fluke.
However, this doesn't necessarily
scale to all sorts of benchmark data that you get.
So if you have a particularly stable benchmark that
looks very solid and then just goes up by a little bit,
that might be a really critical component of your app that's
really stable in CI.
But a 15% regression is completely
ignored by this 20% threshold.
This benchmark wants a completely different strategy
for determining a regression than the previous one did.
Or here's another example--
this happens all the time in CI.
You might have seen that before on the graphs from before.
But you have a temporary spike that
occurs where some sort of interference
to the benchmark while it's running on the device
causes a big spike upward--
let's say, in this case, 115% increase in runtime.
But then it goes back down immediately.
Now, possibly someone checked in a regression
and immediately reverted it.
But what very often happens here in practice
is that you just get bad data from CI.
And that's OK.
We can work around that.
But in this case, if we try and use this naive approach,
it looks like a regression.
And if this happens across every single test,
you're telling every single test owner, hey look,
you've got a performance regression,
and wasting everyone's time.
So variance is the core problem here.
And part of the problem is that it
is inconsistent between different benchmarks.
Different benchmarks will behave differently
and have different amounts of variance
when they're running in CI.
So percentage-based thresholds don't really worked very well.
They trigger false alerts.
And they'll also miss real regressions
in stable benchmarks that could tell you much more granularly
a small regression.
And then functionally, interference spikes
are a fact of life.
Here's some more real data from a combination of recycler view,
adds identifier, and then another library that doesn't
come to my mind right now.
But completely unrelated libraries
in Jetpack that all spiked on the exact same build.
Now, there wasn't any change in here
that caused this particular problem.
But these are results from our CI.
It just went bad for a build.
How do we deal with that?
So we need to be able to handle temporary instability.
We want to avoid manually tuning for benchmark.
Because man, that would not be a lot of fun.
But we do want accuracy.
We want to be able to catch those regressions that we
talked about before-- both the large one and the small one.
So let me talk a little bit about what
we do in Jetpack-- the approach that we call step fitting.
So first and foremost, a regression
will, over time, look like a step
upward in your benchmark results.
They were relatively stable at one value.
And then they increased significantly over time.
So here in the graph--
to the right, you see more and more
builds landing, more and more numbers coming out of it.
And that highlighted build pretty clearly
looks like a regression.
So what we want to do here is, we want to test a hypothesis.
Is this particular build a regression?
So there are two aspects that we want to consider here.
First of all, the gap in between the before and after
is a significant indicator that we
are confident that this is a regression.
If there's a big gap, then we are pretty confident.
If there is variance either before or after,
then that should proportionately reduce our confidence.
So Step Fit is an algorithm that we
have that looks at a series of builds over time--
not just n and n plus 1, but a wider range
than that-- to try and gain a confidence number about how
much we think a regression has happened for a particular
build.
We want to avoid false positives from one-off slow runs
and by looking at multiple builds.
We can do that.
We inspect multiple data points all at once.
And we report a confidence of regression
across multiple builds against an alert threshold.
So let's look at what this actually
looks like running in our CI.
So over time, you get several builds landing
that take a relatively stable amount of time.
You've got several builds landing, more and more results.
Oh, you've got a big spike here.
We've got a regression, right?
Well, no, actually that was just a spike--
not a big deal.
You can ignore that.
But let's look and see how the Step Fit approach will
look and report here.
So first of all, we want to test this build--
not the most recent build, but several builds ago--
if it's a regression.
We split up the data set into "before the
build" and "after the build."
And we feed this into the Step Fit algorithm.
In this particular case, there's not a big difference
between the before and after.
So regression equals false.
We don't really worry too much about this spike.
And again, as we continue and more results land,
we say, oh no, that's not a regression.
We don't have a high confidence.
But over time as we move further and further
and we start getting more and more results that are stably
looking like a regression-- looking like that Step Fit--
once we get to that particular point,
we say, hey look, that's a regression.
I have high confidence.
So to summarize, let me talk about the workflow
again as Dustin mentioned it.
So first and foremost, when you see a problem,
start with local profiling.
That is absolutely the first place
to go to because profiling shows you what is wrong.
Benchmarking can be useful in that case,
if you want to be able to iterate really quickly--
hit the Run button on your test and see
whether it passes or fails, just like you
would do with a unit test.
But with a benchmark, you're looking
for optimizing whatever that problem is.
However, after optimizing, consider monitoring.
This is how you can prevent this problem from happening again
in the future.
And analytics are a great option for this.
But as Dustin mentioned before, if you are only
using analytics, then you are waiting
until users hit the problem before you fix it.
With benchmarks in CI, you can avoid that
by catching them early-- catching them
before you ship to your users.
So if you do want to run with your benchmarks
in continuous integration, then you
can try it out for yourself.
First and foremost, we suggest running it on real devices--
not emulators.
This helps you match the actual performance
that you'd see in the wild much more closely.
Our library outputs metrics in JSON--
a lot of information about the distribution
and various things.
And you can collect all of those,
use the stepFit algorithm in order
to detect more regressions.
If you'd like to learn more about the stepFit algorithm,
there was a blog post last week that I posted.
And you can find out about that on d.android.com/benchmark.
That's got all the information that you need--
how to get it set up, how to get it running in CI,
how to even build it if you're not using Gradle--
all sorts of information.
We've got docs for the library.
We have a sample on GitHub that shows
not just how to use a library, but how
to use it alongside recycler view, which
is a really great way to monitor performance there.
And we've also got a link to that blog post about regression
detection with sample code that shows you
exactly how we do that-- with some sample data
from Jetpack CI as well.
The 1.0 release candidate is out today.
So please give it a try.
And thank you very much.
We are going to be upstairs if you have any questions.
[MUSIC PLAYING]