>>Matt Lowrie: All right.
So now we're going to get back into lightning talks.
First up, we have Rahul from Oregon State University.
>>Rahul Gopinath: I'm Rahul Gopinath, a Ph.D. student from Oregon State University.
And this is a summary of my research, along with that of Iftekhar Ahmed from the same
university.
So this talk is about evaluating test feeds and the best methods we have in actually ascertaining
the quality of our test feeds.
And by quality, I mean the ability of a test feed to find bugs.
So before starting a question, so how many of you were present at GTAC 15?
Okay.
So in that talk there was another talk with a very similar title that said that code coverage
is not a strong predictor for test suite quality.
So this is actually an counter argument.
It was suggested previously that we should not actually look at code coverage and it's
not trustworthy, and we should look at something called mutation coverage.
So I hope to convince you otherwise, and I also hope to show that the suggested gold
standard, that's a mutation score for test suite, is not only expensive to actually compute
but also inconsistent and is very much dependent on the tool that we use to measure the mutation
score.
So essentially mutation score is not really a silver bullet, and we have to use it in
conjunction with the coverage of a test suite.
So, well, when is a program ready to ship?
Notwithstanding the comments about, you know, testing in production for the Google drive,
the Google automatic car, this is one of the decisions we as testers have to make.
And in the old fashioned way we stop testing when we are reasonably sure that the bugs
are completely over.
Not a new way, of course.
So that means that an effective test suite is very important, and that's what we really
need to find.
How do we actually make sure that our test suites are effective.
And how do we actually evaluate this test suite effectiveness?
Of course there are two main things that we all know that we can evaluate in a test suite.
How good is its coverage of the program elements?
And, secondly, what is the strength of its records.
By records, I mean just assertions, because if you don't have assertions, then it's useless.
So in the previous GTAC, Laura Inozemtseva, who actually did that talk, she suggested
that coverage should not be relied on because it rarely provides more information than just
a pure size of the test suite.
So she instituted the mutation score where the effectiveness of a test suite against
a set of mutants is validated.
How many of you actually know mutation score?
How do we evaluate mutation?
Okay.
So at least I know there are a few people that are around here, and I also heard first
day that Google is also -- some of the people in Google are also using mutation score, so
it's not out of the blue.
The unfortunate thing is that, well, as a basic idea, what we do in mutation analysis
is we start with a program and for each token in the program, we start introducing changes.
What are the changes that are valid grammatically that if you compile will not result in an
error.
So we have a large number of variants to that program.
Each variant, you compile it and run it against your test suite and see how many of these
variants are actually caught by the test suite.
The total number of variants that are caught by the test suite divided by the number much
mutations gives you the test score.
That means this is the actual effectiveness of the test suite.
Now, the unfortunate thing is after Laura's speech, programs, especially in the programmer's
community like Hacker News or Reddit, they all took it to mean that we should really
completely discard coverage.
And the comments were that coverage is meaningless in the real world.
And arbitrary lines in sand such as 90% or 100% coverage is pretty much -- they don't
provide any utility.
But is that sufficient?
I mean is it sufficient to ignore coverage?
So that is what we set out to find.
So to start from the very basics.
There are two factors, you know, assertions and coverage.
And coverage on its own is useless without assertions and assertions have to be paired
with good high-coverage test suites.
And Laura's idea is that coverage is completely determined by the size of the test suite,
and that it doesn't actually tell us anything new because it so closely correlates with
the size of the test suite.
So since test suite is so easily determinable without even actually running it, what is
the point of actually looking at coverage, and why not just use test suite size instead?
That was the point in the previous talk.
However, does the question is does it actually make sense to control for the size of the
test suite?
It's not a good measure.
We do not actually know what is the maximum number of tests that can be there in a test
suite, even for the same program, because we can have different test suites for the
same program.
And different tests are not necessarily equivalent in strength.
There are can be some tests which actually unveil a large number of problems and there
might be tests which are specific.
So size of the test suite, while it is simple, is not really a reasonable measure because
we don't really is any, you know, end points.
There's no zero percent test suite size or 100% test suite size.
So one of my points is even if coverage just measures test suite size, it's a good thing
to have because at least it tells us how large the test suite is.
And, secondly, none of us actually cares about the total number of tests that we run.
We may have a time budget.
We may have other kinds of budget, but not really related to the number of tests.
So for a practicing tester, there is only one single question.
Can I use coverage to measure the test suite effectiveness?
And this is what we set out to find in my research.
So what we did was that we started with 250 real world programs for the study from github,
and the largest was greater than 100 kilo lines of code.
So these are not toy programs.
And we checked coverage for both organic -- that is, test suites that are written by people
like us -- and also machine-generated tests that are generated by Randoop.
So we found that for these organic test suites that are written by human beings, if that
statement is covered then that statement has 80% chance -- if there is any mutant in that
particular statement, that mutant has 87% chance of being detected.
And if you have a statement coverage, then that can actually predict mutation score with
94% certainty.
That is once you control for statement coverage, mutation score actually provides for at least
6% extra availability that you can -- you know, you can evaluate.
So this is another thing.
The red -- the previous blue ones were developer written and these red ones are automatically
generated by Randoop.
So as you can expect, the records are really bad and 61% chance that if a line is covered,
that particular -- any mutant in that particular line will be caught by the test suite -- by
the randomly generated test suite.
So given this result, there's a question that we can ask.
So if coverage is so highly correlated with mutation score, what's the point of using
mutation score?
I mean, we can essential just use coverage and predict the mutation score instead.
So until now, we assume that coverage provides no extra information with respect to test
suite size.
That is assumption that Laura started with.
That was the result that Laura had.
However, the question is is that particular question -- is that assumption really true.
Remember that her results were actually from just five programs; even though very large
programs but still, only five programs, while we actually have 250 programs.
And she was also using random subsets of the same test suites again and again and again.
So is it possible that her conclusion that coverage is directly related with test suite
size may not actually generalize to the larger programs?
So this is what we actually wanted to find.
So what we did was that we eliminated the effect of test suite size statistically, and
we actually found that even if you control for the test suite size, for the remaining
amount of information that is provided by the mutation coverage, 75% of it is still
explained by the statement coverage score.
So that means that the previous assertion, or previous assertion of Laura from these
five programs, are not generalizable to a larger number of programs, especially large
programs and Morandom programs, not exactly, you know, chosen by the researcher themselves.
Which is a major issue in the software engineering research because usually there is a, you know,
problem that we rely very heavily on specific programs which are used again and again.
So even if coverage is a reasonably good criteria, it's still possible that we might actually
go for mutation analysis.
Because in some sense, mutation analysis comes very close to actual bugs, because we are
actually introducing bugs into the program and evaluating how many of these are caught.
So should we actually use mutation analysis?
Is it actually ready for the prime time?
Or the question is, should we use mutation analysis an the primary means of evaluating
quality of a test suite?
So, unfortunately, mutation analysis is beset with a large number of problems.
The biggest thing is, of course, the large number of variants.
Each of these variants have to be tested by the complete test suite.
And there is no way out of it.
We essentially have to evaluate each of them.
Even small little programs like these (indiscernible) tend to produce a large number of mutants.
And secondly, so that's -- mutation score is really costly.
And even if you ignore the size of the mutation analysis, there's another major problem.
These are called equivalent mutants.
The problem is that these changes that we introduce, the tiny little changes that we
make to the program, they need not always result in a fault.
Some of them can actually result in programs which are behaviorally the same as that of
the original program.
These are called semantic clones.
Or in terms of mutation analysis (indiscernible), these are called equivalent mutants.
The problem is that if you have a program, if you have a program which produces, say,
50% equivalent mutants, none of the test suites that you can ever write is going to have a
mutation score that is larger than 50%.
So -- And, unfortunately, there is no general way as of now, in software engineering research,
to identify how much of mutants would actually be equivalent, given a program.
Nor can we identify, given two programs, whether one or -- sorry, a mutant and a program, whether
the mutant is going to be equivalent to the program.
So this is a result from, you know, Turing computation.
This is called (indiscernible).
And that essentially says that a general way to do that is pretty much impossible.
So that means that a low mutation score is not generally indicative, it doesn't actually
indicate a low quality of test suite in general.
I mean, it can be true for most of the times.
But if your program is a unique snowflake, then it is possible that you have a low mutation
score, but you actually have a test suite that is highly effective.
And even versus something called redundant mutants.
So redundant mutants are still semantic clones.
They are semantic clones of each of them.
They have a fault in them, but there are a large number of mutants which are exactly
the same fault.
So if your test case detects at least one of them, then the rest of it faults.
So what that means is that if you have a large amount of redundant mutants and your test
suite catches one of them, then you will have a high mutation score even though the test
suite may not actually have been very, you know, effective.
So we can't really trust mutation score on that score, either.
And, finally, this is a problem with the current state of the art.
This is not a problem with mutation analysis itself.
The problem is that different mutation tools have very different -- you know, they act
on very different levels.
So you can act -- you can have a mutation score that -- mutation tool that acts on the
assembly, or in the machine core, or on the source code itself.
So here, I see -- I show two different ways in which you can have mutants, one on the
source code itself, and the other on the assembly.
And the problem is that we have on the left the mutation score that is computed by three
different tools that act on three different levels.
And the problem is that these tools do not actually agree with each other that well.
So you can see that there.
So this is the current state of the art for mutation analysis.
So it's not consistent across different tools.
So, finally, the proof of coverage is actually -- the proof of -- is in the pudding.
So if coverage is a good measure of test suite effectiveness, then a well-tested program
that we assume by using coverage should actually result in fewer bug fixes, and we should be
able to find the difference between, you know, covered and uncovered programs.
Or covered and uncovered lines in the same program.
And this is what we did next, using the same 250 programs.
So what we did was that we started with an epoch and then counted the bug fixes that
happened on that particular line until it was actually wiped out by a feature fix.
So what we actually found is very interesting.
We found that if there is a line that is covered you and compare that to any general line that
is not covered, a covered line is -- or an uncovered line is twice as likely to have
a bug fix as that of a covered line.
And that is true across statement, block, or method level.
That is, if you have a method that is at least tested or covered by some test case (indiscernible),
that method would have half bug fixes in its future compared to a method that did not even
have a single test case covering it.
We don't have sufficient data for class, so you can see that our P values are somewhat
low, and we don't have significance there.
So, yes, coverage is a very strong defense against future bugs with respect to whatever
I say.
So, in summary, coverage is still the best measure that we have for evaluating test suite
quality.
And mutation score can, at best, provide a secondary measure.
So don't rely solely on mutation score.
And also, if you are using mutation score, understand the problems of mutation score.
It is expensive.
It is cost- -- it is unreliable, and provides maybe little extra information, like 6% beyond
coverage.
And more importantly, in addition to the -- you know, the simple measures like mutation score
or coverage, if you actually look at which statements are actually not covered and try
and see why it is not covered or look at even mutants, look at the mutants that are not
killed and see why it is not killed, that is probably a very good testing technique
in itself.
Yeah.
Thank you.
[ Applause ] >>Matt Lowrie: Excellent.
Thank you, Rahul.
So the lightning talk will take one question.
Wait.
It's Laura from last year.
She's calling.
She wants a word.
She wants to talk to you.
[ Laughter ] No.
Just kidding.
So I think that covers the first one.
We'll ask -- Do you have an opinion or studies on code coverage effectiveness with property-based
test?
>>Rahul Gopinath: I will start with the first one first, because this is an ongoing argument
between us whether mutation analysis is ready for prime time use.
And you know what my opinion is.
So software engineering community is trying to come up with better tools to actually remove
equivalent mutants and redundant mutants and come up with better tools.
So that is where engineering community as of now stands.
Okay.
And do we have an opinion on (indiscernible) code convergence -- Do you mean code coverage?
Well, -- well, code coverage effectiveness is pretty good.
And I think property-based tests are really good, especially things like (indiscernible)
check.
Those actually help you quite well.
But the problem with property-based testing is that it's really hard to write that kind
of test.
I mean, you really need programmers who are well versed in that.
Yeah.
>>Matt Lowrie: All right.
Excellent.
So thank you, Rahul.
[ Applause ]