>>Andrei Chirilla: Well, my timer it's already off.
So I'm Andrei Chirila.
I'm software engineer in test at Google in Zurich.
And we work with coverage for more than one and a half years, and now we are ready to
share some of the numbers we have.
So I am going to go through the code review at Google, because it's important to understand,
and given that both Karin and Ankit mentioned code review, I'll cover it.
Test coverage.
I intend to use ten minutes, and five minutes for Q&A.
So Google3 code repository, you know that we have only one big repository, except Chrome
and Android that are separate.
And development happens only at head, and we develop in multiple languages.
And with such a repository and a commit every three seconds, Ankit said this morning we
have changes that go throughout the code.
So you can have an UI, and in the morning you add something, and in the meantime, all
the components underneath have been changed.
So with this in mind, you might be asking yourself, okay, so that means that the build
for Google3 is always broken.
[ Laughter ] Well, funnily enough, no.
And this is not in small part due to the code review process.
Oh, this is the wrong button.
So code review happens in Google before check-in.
This really makes sure that we don't break stuff.
It's Web-based, streamlined, so you can use it on your commute home, early in the morning.
It doesn't matter.
It's very cool for small, incremental changes, because it shows exactly the two lines that
have changed and can go in.
And it's mandatory.
So there's no code change that goes in without code review.
And I see a couple of you nodding, yeah.
And it's helped a lot by automated analysis.
So the analysis.
What does it do?
Well, from the very small, simple stuff, like code style compliance just to make sure all
the code looks the same, you don't go in another team and you're not able to read the code
anymore, up to doing the tests and running the tests for that pass.
Again, one single repository.
There's a system that figures out the dependencies behind to run the minimum amount of tests
that would ensure that your change doesn't break the stuff.
So it also has to be kind of fast.
So in Google, first response for a review comes after four hours.
So that's average.
If you're working with someone nearby, yeah, that's half an hour or so.
But otherwise, that's average in Google.
How do we fit test coverage into this?
Well, we don't fit it yet.
We send reviews, and we have two types of reviewers.
One is for ownership.
That is person that owns that code and has most knowledge of that code.
And readability, the language style compliance.
And the reviewers will see the response of these automated tools.
And this is pretty cool, because we are able to see, as reviewers, okay, the tests failed
for this.
For that, you can send back a CL and say, sorry, I'm not going to review until you fix
the tests.
And that also speeds up the process.
And then you end up with a question, okay, how many tests should I write in order to
get something done?
How much does the reviewer have to see in order to say, yeah, okay, this is fine enough.
Go.
And here we have tools that help us.
And also for code coverage.
So for code coverage, we are speaking mostly about statement coverage in this presentation.
Function coverage, it's pretty easy to get, no matter the language support, because you
can hack into the language and get the function numbers.
Statement, it's something that has support both in native and also with some tools, like
for Java, with JaCoCo.
Branch, we haven't sorted that out yet.
So we are getting results for some languages, but for some languages we don't.
And there are other ways of measuring coverage, for example UI to code mapping.
There are some teams that are working with research in that direction.
But they're not yet there.
So based on statement coverage, we have two options in Google.
Both are opt in.
One is per project.
So we are running code coverage on the whole project once a day, for example, and the results
are looked at by the project team.
And the interesting part is that people seeing these results set objectives for the next
quarter, okay, we have to reach this number or this number.
But the question is, what is that number and how does that work?
And per commit.
That is, every change that goes has also coverage results.
And we have done this for more than one and a half years within Google, so projects that
opted in.
And we have more than 1,000 active projects monthly.
Google is huge, so we didn't manage to get everyone in, but we separated at least the
users that use the continuous test systems.
And this is our conclusion.
So, basically, what we did is, we looked at the numbers, and we say, guys, the number
to be reached is 85%.
And why are we able to say this?
Well, we have a lot of data.
But if we look at any one month in the data for the active projects, we see that the median,
it's at 78%, the 75th percentile at 85%.
So these are the projects that have the coverage-enabled solution in Google.
So the 75th percentile for coverage reached, it's 85%, and 90th percentile is 90%.
That means people really care about coverage.
But they also need a drive.
There are a couple of projects that really need to be pushed through.
I'm not able to say how many.
[ Laughter ] Sorry.
And based on this, we also did a bit of correlation.
Okay.
How does coverage relate to language strong type-ness?
And it's interesting, because C++, for example, has lower coverage than Python, for example.
The numbers are 53%, 63%, 70%, and for JavaScript and Python, they're shooting for 80.
So -- And how did we get there?
So we made coverage available exactly at code review time.
This is a snippet how it would look for a reviewer to look at the change.
And you can see the green lines where the code is covered by test and the red lines.
And based on this, they are going to say, sorry, your change is not covered good enough.
Please go and write test for this and this case, because you're not there yet.
And this worked pretty well.
So we observed a 10% increase in the coverage, computed, okay, the first time you send the
review, and knowing that change has been sent into the repository.
So people are really acting on this.
And we are trying to make this even broader available.
So that's what we kind of had to share with you, that internally, we are aiming for 85%
code coverage.
And it's absolute code coverage.
There's a difference between absolute and incremental.
Incremental, it's per change, and there we are aiming 99%.
So legacy code, okay, we understand you have not written tests.
That's the code that you worked with.
For new code, you should really aim high.
Of course, this depends on the language and the support that you have.
And there are cases which you just can't cover, you'll have to live with it.
Also, the tools that we have really help us push the numbers up.
And the better we develop the tools, the better code coverage we'll have.
And, okay, we're not saying code coverage is the ultimate metric to code health.
No, by no means.
But it's a good signal.
It's something that helps and improves code health.
And also keep in mind the language differences.
There are some languages for which if you don't execute the line, that line can do anything.
Think Python.
[ Laughter ] Thank you.
[ Applause ] If you have questions.
>>Sonal Shah: Thank you, Andrei.
Great talk.
So 85 is the magic number.
Good to know.
>>Andrei Chirilla:Well, it's 86 in the memo, but we rounded down for presentation.
>>> Two quick, easy questions.
>>Andrei Chirilla: Sure.
>>> First, in your code review tool, how do reviewers, since you mentioned it's quick
and efficient, how do they ensure they have context for what this change causes, since
that API could be called by any number of places in the stack?
>>Andrei Chirilla: That's true.
>>> Does it tie back to them so they can make sure all the different scenarios --
>>Andrei Chirila: There is possible to, for example, click on -- let's assume I was asking
a function somewhere, click on it and go to the code.
But by default, no, it's expected that if you're testing the stuff, and that bar, it's
green, like, you've covered it, it kind of works.
So you can jump to the symbols.
You can get the full context by seeing the file.
But there are small changes, like one- or two-line configuration changes, or let's turn
a knob here and there.
And for those, you're sending the changes to review to people that are familiar with
the code.
So -- >>> Thank you.
>>Andrei Chirilla: -- usually code reviewing with Google, do you it with your team most
of the time, and then you have cross team, which takes a while.
And those are longer reviews.
Over the ocean, again, time zone, takes time.
But people treat code review as part of the process.
>>> Okay.
>>Andrei Chirilla: So this is part of our job.
It's not below code writing, for example.
>>> Okay.
So then the second question is, you mentioned it shows code coverage, the coverage number,
85%.
How do you know that 85% is covering the code that actually matters?
Like, does it tie back to scenarios so that you can ensure the story is covered 85% or
100%, versus numbers magic; right?
>>Andrei Chirilla: Yeah.
Okay, the question is very good.
And the trick is the following: We are speaking very clear about code coverage.
So what you're speaking, scenario or case, that's slightly different.
We assume that we are working directly with unit tests.
So we are not considering integration tests here.
And we also assume that you have the proper setup for this.
And we are helping the teams that want to get there to get the right setup.
For example, code that is added new should get directly coverage, because it's in certain
location, in a certain configuration.
But what we are working towards is making this available no matter what your configuration
is.
So figuring out on the fly, okay, what happens, how this works, which code is relevant for
this, and what is the number that we have to deliver to you.
>>> Is the code coverage -- or is the code review tool that you're talking about here,
is it open source, by chance, that other companies could use it?
>>Andrei Chirilla: Not as far as I know.
I know that they're working on something.
But I'm not able to give any definite answer on that.
>>> Okay.
>>Andrei Chirilla: Sorry.
>>> Thanks.
>>Andrei Chirilla: No problem.