>>Yvette Nameth: We have Laura from the University of Waterloo, who is going to talk about coverage
is not strongly correlated with test suite effectiveness.
[ Applause ] >>Laura Inozemtseva: All right.
Good afternoon, everyone.
I'm going to start my talk today with an assertion: High coverage does not imply high quality.
In the next ten or so minutes, I am going to try to convince you that this is true.
But, first, let's take a step back.
Why do we use coverage at all?
Well, we want to get the bugs out of our programs, and we write tests to do this.
But if we're not finding bugs, it's not immediately clear whether the program is just very good
and there aren't many bugs to find or whether the test suite is very bad and you it's just
not doing a good job of finding those bugs.
So what we would really like is some way to estimate the fault detection ability of a
test suite.
And coverage is often recommended for that purpose.
But is it really a good way of measuring test suite quality?
Well, a number of studies have been done to try to answer this question.
The results have been mixed.
The studies indicated with the green check mark found that there is a relationship, there
is a correlation, between coverage and effectiveness.
One study indicated by the red X found no relationship and the others marked by black
tildes found mixed results.
So the relationship depended on the coverage type, the program type, or some other variable.
But there were a few other problems with these studies.
First of all, only the two most recent studies used real software.
Prior to that, they had all been using small toy programs, hundred lines of code, that
sort of thing.
So it's really not clear that any of these findings generalize to real software.
Second, some of the studies used unrealistic suites.
And I'm grouping two different ideas here.
Some of the studies used test suite generation tools to make the test suites that they evaluated.
And some of them didn't consider -- actually, no.
It was mainly just the -- yeah, the generated test suites that were the main problem here.
Although some were also 100% coverage, they were known as adequate test suites.
And those adequate ones are not common in practice.
You don't usually see suites with 100%.
So, again, it is not clear that results on adequate test suites or on generated suites
will generalize to more realistic test suites.
And, finally, some of the studies did not control for the number of tests in the suite.
And this is important because the way you increase your coverage is by writing more
tests.
And when you write more tests, you catch more bugs.
But it's really not clear if you're catching more bugs because the coverage increased or
whether it's just because there are more tests.
So we wanted to do a new study that took all these things into account.
We started by picking five realistic programs on the order of 100,000 lines of code.
And the one I'm going to talk about primarily today is the closure compiler, which many
of you are probably familiar with.
These still aren't the largest programs you'll see in industry but closure was around 700,000
lines of code at the time of the study.
So they are respectably sized.
And, like I said, we wanted to develop written test suites.
However, we wanted more than one data point per program which meant we needed some way
of creating test suites that wasn't a generation tool.
So we did this by randomly selecting test cases without replacement from the suite that
came with the program.
So first we made suites of size three by taking three test cases, then suites of size ten,
30, 100, and so on.
We made 1,000 suites of each size so we could get statistical significance out of the results.
Next we measured the coverage of each of these suites that we created using the tool CodeCover.
We measured statement coverage; decision, also known as branch, coverage; and modified
condition coverage.
MCC is not very common in practice, so you may not have heard of it.
But it's a stricter coverage criterion.
It's pretty difficult to satisfy.
So we wanted to see if there are any differences between something like MCC and something like
statement coverage.
The final step was to measure the effectiveness of the suites.
This is a little bit less straightforward.
So we did it with mutation testing.
Now, if you're not familiar with mutation testing or if Yvette's mom is still watching
-- [ Laughter ]
-- in mutation testing you make small syntactic changes to the program under test and then
you run your test suite on each of these new programs, these mutants, to see if it can
detect the fault that you injected.
If the test suite can detect a large proportion of these mutants, it's considered to be a
good suite.
Now, this might seem a little bit artificial, but it has been shown to be representative
of the suite's ability to detect real faults.
So to briefly recapitulate, we selected five large real programs, largish real programs,
made test suites through random selection, and measured the coverage and effectiveness
of those suites.
So let's go ahead and look at the data for closure.
Here we're looking at all sizes.
We're completely ignoring the effective suite size.
We have coverage on the X axis from 0 to 100% and effectiveness, so percent of mutants detected,
on the Y axis again from 0 to 100%.
The color of the points indicates the coverage type.
So the green that's furthest to the left is MCC.
In the middle, the pink color is decision.
And blue is statement.
So when we ignore size, we actually see some nice diagonal lines here.
There's a good relationship, right?
But if we look at only the size three suites, suddenly we just have a big cloud of points.
The coverage isn't telling us very much.
If we move up to size ten, all the average coverage increases.
The average effectiveness increases as you would expect because there are more tests.
But, again, we have kind of a cloud of points here.
No nice relationship is visible.
And this continues as we move on through the size 30 suites, the 100 suites, and so forth.
The point clouds become more dense as we move on because there's less variation in the larger
suites.
But at no point do we ever see that nice diagonal line.
So this brings me to the first result from our paper, which is that coverage is not strongly
correlated with effectiveness when suite size is controlled.
To really clarify this distinction, I'm going to show you another graph here.
The red line on this graph is the total revenue generated by arcades in the United States.
The blue line is the number of computer science doctorates awarded in the United States.
[ Laughter ] These two variables happen to be correlated
with a very high correlation coefficient, 0.98.
But it's almost certainly the case that there is not a causal relationship.
They probably both depend on some other available like the size of the population.
So while it's very tempting to make causal statements, it's very tempting to say, "Our
test suite has 70% coverage so it will catch a lot of bugs," this is equivalent to saying,
"Arcades made $2 billion last year, so a lot of Ph.D. students will graduate."
[ Laughter ] Now, if we go back to this graph, there's
another interesting finding in this paper.
These three different colors of points look pretty much the same.
And we can quantify that by measuring the correlation between each pair of coverage
types.
And we see that it's always above 0.9 even using a conservative non-parametric measure
of association.
So what this tells us is that the stronger coverage type provides little extra information
about non-adequate test suites.
That's suites with less than 100% coverage.
And we need that little caveat because it's possible that a suite with 100% modified condition
coverage is strictly better than a suite with 100% statement coverage.
But in the typical scenario where you have a test case, you want to learn a little bit
more about it -- sorry, test suite, you want to learn a little bit more about it so you
measure its coverage, it doesn't really matter which type you measure.
You may as well just measure statement.
So I have been a little bit negative about coverage.
What should you do instead?
Well, if you were really paying attention earlier in the talk when I talked about the
method, you might have noticed that I said, "The mutation score of the test suite is highly
correlated with the suite's ability to detect faults."
We demonstrated this last year in the paper titled here.
So this means the mutation score is exactly the metric we want.
But... and there's always a "but."
Mutation testing is a little bit impractical.
You can generate tens of thousands of mutants for a typical program, and you need to run
at least part of your test suite on each one of those mutants to see if it can detect that
the mutant is faulty, that you injected that fault.
So working on this is one of my current research interests.
As a matter of fact, my supervisor just submitted a Google faculty grant on this topic.
So, hopefully, if all goes well, next year I will be back with a really great answer
for you and a really great new tool, something that can get you the information of mutation
testing with the speed of coverage.
In the meantime, I will just leave you with the assertion that I started the talk with,
"High coverage does not guarantee high quality."
I hope that I have convinced you or that I have at least given you something to think
about.
So the next time one of your co-workers comes into your office and says, "Our test suite
is great, it is 80% coverage," you can think to yourself, "Yeah, and a lot of Ph.D. students
are going to graduate."
[ Laughter ] The official versions of the two papers I
discussed are paywalled.
But I have put them on my Web site here, lorg.org, so you can read them for free more if you
are interested in learning about those two studies.
Thank you very much.
Are there any questions?
[ Applause ] >>Yvette Nameth: Thanks, Laura.
And thanks from my mom, too.
The first question would be does low coverage guarantee low quality?
>>Laura Inozemtseva: That's interesting.
I would probably say yes because if you're not running the code, how can you possibly
find any bugs in it?
I would say that coverage is necessary but not sufficient.
What the mutation testing paper is kind of telling us is that the strength of your oracle
is what's really important about your test.
So you can execute every line of code in your program without actually checking that any
of it is correct, right?
So, yeah.
>>Yvette Nameth: How representative of real test suites do you feel your randomly generated
suites are?
Is it not more common for some areas to have good coverage and some areas to have poor
coverage?
>>Laura Inozemtseva: So -- well, we didn't exactly use randomly generated because we
didn't generate them.
We picked the test cases.
But if you just mean is that representative, it's somewhat representative and somewhat
not.
So if you look at testing textbooks, they'll say, okay, developers, this is how you are
supposed to test.
You find the area with low coverage.
You go write a test for that.
Or they give you some very deterministic method of producing tests.
And in practice, you test a little more randomly than that, right?
It's, oh, I wrote a new feature.
I will write a test for that.
I found a bug here.
I will write a test for that.
In practice, yes, you would see a lot of tests for some spots and less for others.
But, unfortunately, there's a limit to how realistic you can get when you want to have
a thousand suites for each program.
You can't get people to sit down and write a thousand suites, so...
>>Yvette Nameth: Could you give some examples of mutants used in your experiment?
>>Laura Inozemtseva: That's a good question.
I should have talked about mutation testing a little bit more.
Some things you could do are change a plus sign to a minus, like an operator swap.
You can switch the arguments to a method.
If they are the same type, you can just flip them to a different order.
You can change constants to a different value.
You can change strings to a different value.
Lots of different things that you can do actually.
>>Yvette Nameth: Have you found any metrics that are correlated with high quality?
>>Laura Inozemtseva: So the mutation testing score does seem to be.
The only downside is that it is expensive to compute.
I've seen some very new work on a new metric that's trying to sort of approximate the mutation
score.
I can't talk to you much about it because it was in a paper I was reviewing, and the
reviews are done in confidence.
But there is some interesting new work in that space.
>>Yvette Nameth: What about test-driven development approaches which can effectively produce 100%
coverage?
>>Laura Inozemtseva: I might take issue with "effectively producing 100% coverage."
[ Laughter ] But, I mean, what about them?
You can cover all the code without checking anything that's really important.
Or you can cover all the code and write really great tests that test for every possible error
condition.
So they can be good or bad just like writing tests after the fact.
>>Yvette Nameth: None of these are actually new questions.
Are there any more?
The statement that higher coverage does not correlate with effectiveness if the size is
controlled seemed like a stretch.
The data is not spread too much along either axis.
Isn't that just noise or inconclusive data?
>>Laura Inozemtseva: The fact that we had a thousand suites of each size let's us get
statistical significance.
So these findings were all significant to the 99.9% level.
So only a .1% chance that this is noise.
>>Yvette Nameth: And that seems to be the end from what I can see.
Cool.
Thank you, Laura.
[ Applause ]