Aloha Hawaii Dave Aronson the t-rex of
Coto Soros LLC and
flew over here on my pet pterodactyl
they teach you how to kill mutants so
what are those in our universe that of
software development not comic books
there's something out of mutation
testing so what on Infinite Earths is
mutation testing you might look at the
name and think it's a way to test the
mutations used in genetic algorithms or
something but no it's actually a way to
test both our code and our unit test
suite by using mutations it's most
unusual benefit is to help ensure that
our tests are strict by finding the gaps
that let our code get away with
unintended behavior lack of this
strictness usually comes from lack of
tests of course or poorly written or
test set or poorly maintained and just
didn't keep pace with changes in the
code speaking of the code mutation
testing can also help ensure that our
code is meaningful by which I mean that
any tiny little change to it to have a
noticeable effect on its behavior lack
of meaning usually comes from code being
redundant or unreachable or otherwise
not having any real effect mutation
testing puts these two together by
checking first that some tiny little
change did indeed have a noticeable
effect and then at our unit test sweet
spots that change and fails not all the
tests have to fail but each change
should make at least one unit test fail
now that's the plus side but as Fred
Brooks told us back in 86 and multi
reminded us the other day there's no
silver bullet besides which they're for
killing werewolves not mutants
the first drawback is that it's rather
hard labor on the CPU and therefore
usually rather slow we don't want a
mutation test our entire code base on
every save maybe over a lunch break for
a small system or overnight or over a
weekend for a larger one fortunately
most of the tools do include an
incremental mode so we can test just
what we changed since last time that
maybe we can do on every save if we save
early save often like we know we're
supposed to but don't always another
drawback is that it's not at all a
beginner friendly technique it tells us
that some particular change to the code
made no difference to our test results
but what does that mean you'll find that
interpreting the results of mutation
testing involves a lot of asking so what
does that mean recursively it's it takes
a lot of interpretation to figure out
what a mutant is trying to tell us
they're almost as incoherent as zombies
but with a much larger vocabulary
they're not always on about words
they're usually trying to tell us that
our code is meaningless or our tests are
lacks or of course both but it can take
a lot of time and effort to figure out
exactly how and sometimes it's a false
alarm
for instance the mutant might not have
made any test fail but it might not have
made any actual difference in the toads
behavior either at least not in anything
we actually care about so those are the
main pros and cons but what does
mutation testing actually do it mutates
hence the name copies of our code to
create test failures otherwise known as
faults so mutation testing is a fault
based software testing technique and
this means it's sort of related to
something you might already be familiar
with chaos monkey from Netflix just like
chaos monkey uses faults to help Netflix
discover flaws in their error recovery
process mutation testing uses faults to
help us discover flaws in our code and
our unit test suite but the way this
works is kind of upside down from what
chaos monkey does chaos monkey is best
known for injecting faults into
Netflix's production network
if the customers don't notice and the
metrics still look fine then Netflix
knows that their error recovery is
working great
but mutation testing injects instead not
faults but changes it doesn't know
whether these changes will produce
faults or not we hope they all will but
that's actually up to the unit test
suite it injects them not into our
actual network but copies of our code
and it does this in our test environment
not production
and if everything still passes that
doesn't mean everything is okay that's
when we have a problem remember each
change should make at least one unit
test fail so how does it do all this
let's peel back one layer of the onion
and look at it from a high level view
first our chosen tool breaks our code
apart into pieces to test usually these
are going to be our functions for each
function it then tries to find the tests
as if there aren't any tests well would
be kind of redundant to impossible to
run its tests assuming it finds any then
the tool makes mutants out of that
particular function and to do that it
looks closely at the function to see how
it can be changed and for each way that
function can be changed the tool makes
one mutant with just that one change in
it
after the tools done creating all the
mutants it can on the given function and
it iterates over the list and now we get
to the heart of the matter
for each mutant made from a given
function the tool will run the unit
tests from the original function but
using a mutant instead and if one of
those unit tests fails this is called
killing the mutant quick sidenote
some people object to the violent nature
of this metaphor especially since in the
comic books
mutants are often used as a metaphor for
marginalized people I'm trying to come
up with a better term like maybe cover
or rescue or somesuch but I couldn't
come up with something good in time to
change this talk oh well anyway killing
the mutant means that the tiny little
change that the tool made in order to
create the mutant it indeed have a
noticeable effect on the behavior and
our test suite was strict enough to
notice that change and fail at least one
test after one test has failed the tool
will stop running a test against that
mutant and move on to the next one we
don't care how many more unit tests that
mutant might make fail like so much else
in computer science we only care about
ones and zeroes
but if a mutant lets all the functions
unit tests pass then it has the
superpower of mimicry skilled enough to
fool our tests this usually means that
our code is meaningless or our tests are
lacks or both and now it's up to us to
figure out exactly how now let's peel
back another layer of the onion and look
at some technical details of how this
works first our tool parses our code
usually into an abstract syntax tree or
AST I think those were mentioned on
Wednesday I know these boxes are a
little small to read but we don't need
to actually understand this ast in
detail after our tool makes an AST out
of our code and it traverses that tree
looking for sub trees that represent our
functions after finding them it handles
them pretty much like I described
earlier starting with looking for the
tests but how does it do that
this usually relies on us developers
either annotating our code or following
some kind of naming convention and this
is sometimes supplemented or even
replaced by the tool looking at what
functions each of the tests call
next the tool makes the mutants and to
make them from an ast subtree it then
traverses the subtree just like it did
to the whole thing this time it's not
looking for even smaller sub trees twigs
or something to extract but nodes where
it can change something maybe substitute
a different kind of node or whatever for
instance suppose our tool was traversing
the ast I showed earlier and it's gotten
down to this not equals comparison
following those red lines or each way it
can change that node it will make a
fresh copy of that entire subtree with
just that node changed in that one way
after it's done making as many copies
which are the mutants by mutating that
node it'll continue traversing this
subtree and do likewise to the rest of
the nodes
I've been talking about it making
changes so what kind of changes am I
talking about there are quite a lot it
can change a mathematical logical or
bitwise operator from one to another in
cases where you can do one of the
totally different category it can even
do that for instance in JavaScript we
can treat anything as mullions so x
times y could become x and y it could
swap the order of operands in cases
where that matters it can change a
comparison from one to another it could
insert or remove a logical negation it
can remove a condition or a loop it
could scramble or truncate the argument
lists of function calls or function
declarations it can replace a functions
entire contents with turning a constant
returning either of the arguments
deliberately raising an error all kinds
of things even nothing at all if the
language permits and JavaScript does it
could change a literal or a variable or
an expression or a function call to some
other value even one of a completely
different type like changing a number
two if I may quote smeagol string are
nothing
there are many many more but yeah I
trust you get the idea from here on
there are no low-level details I want to
add so let's look at some examples we'll
start with an easy one suppose we have a
function like this
think about what a mutant made from this
but return because that's what our unit
tests are almost certainly looking at
this doesn't seem to have any side
effects mainly such a mutant could
return results such as any of these and
many many more but I had to stop
somewhere
now suppose we have one test like so yes
this is a rather poor test but even so
the vast majority of these mutants
well the mutants that would return these
results would still would still get
killed by this test ones shown here in
crossed out green
but
addition multiplication and
exponentiation in the reverse order all
still get us the correct answer and
would therefore survive the test we know
this because when we run our tool it
usually gives us a report that will look
kinda sorta like this the exact format
and amount of context and so forth well
very enormous ly depending exactly what
tool were using but the information
should be pretty much the same and that
is that if we change the function called
power which is in file demo J s at line
42 in any of four different ways all its
unit tests would still pass and then
that those four ways are to change line
42 to swap the arguments line 43 to
change the exponentiation into addition
or multiplication or line 43 to swap the
operands so what is this set of
surviving mutants trying to tell us
good start to figuring that out is to
ask ourselves how are these mutants
surviving
and the usual answer is well they give
the same result or have the same side
effect as our original code to determine
how that happens a good start is to look
at one mutant along with one test that
it passes so let's start with the plus
mutant looking at this in combination
with the test makes it pretty clear that
this particular mutant survives because
two plus two is the same as two to the
second power
in order to kill this mutant hover it
rescue it whatever we need to have at
least one test that uses inputs such
that X plus y is not the same as X to
the Y we can either add a test or tweak
our existing test to maybe something
like this 2 plus 4 is 6 which is not 16
so this will kill the plus mutant better
yet 2 times 4 is 8 also not 16 so as a
side benefit it kills the x mutant as
well but the argument swapping mutants
still survived but that's ok we don't
need to be a superhero about it and kill
them all at once we can attack them
separately to do that we can once again
either add a test or tweak our existing
test maybe to something like this
ooh the Emmys three squared is nine
which is not eight so that will kill the
argument swapping mutants but also two
plus three is five two times three is
six
both of those are not eight so the other
mutants stay dead so we don't get any
zombie mutants with these inputs the
correct operation is the only simple
common one that will get us the right
answer this may make it may make
mutation testing sound pretty simple but
this was a downright trivial example so
we could easily think up test inputs to
make pretty much any reasonable mutant
give different results from the original
we could use for instance three to the
fifth or vice versa whatever lots of
ways to skin that florican so let's look
at a more complex example suppose we
have a function like this send message
uses send bytes to send as many bytes as
and bytes can handle over and over
picking up where it left off last time
until the message is all sent and I'm a
fairly common pattern now a mutation
testing tool could make lots and lots of
mutants out of this but the one I want
to show you is this it's an example of
removing a loop control by deleting
those two lines with the minus signs
there
now suppose that this mutant survives
our test suite which consists mainly of
this there's a little more than I'm not
going to show you quite yet dealing with
setting the size and creating the
message but even just the fact that this
mutant survives tells us something and
that is that if a mutant that only goes
through this loop body once big it has
the same result as the original code
does in the tests
and our tests are only making the code
go through the loop body once so what
does that mean it means that we're only
testing sending a message small enough
that send bytes can handle it in one
chunk the most likely cause of that in
turn is that we're just not sending a
big enough message to make it go through
body twice or more such as in this case
we might have a maximum chunk size what
send bytes can handle in one shot 10,000
bites yet we're only testing with a tiny
little three byte message and they're
the fix is pretty clear we can just
instruct a larger message take the size
add one make that bigger message and
there we go but
maybe it's not this particular cause
maybe we did best with the largest
permissible message out of a predefined
set of messages or least message sizes
for instance here we have small and
large sizes and we did test with a large
and yet the mutant survives in other
words we're still only going through
that loop body wants because send bytes
is handling the whole message in one
chunk so what's the mutant trying to
tell us now
now it's trying to tell us that a
version of send bytes with the looping
removed will do the job just fine and if
we do that and then remove everything
else that that makes redundant we wind
up with this and now it's pretty clear
at the ultimate message is that the
whole send bytes and excuse me send
message function may well be redundant
now I say may well be rather than is
just because in real-world code there
may be some logging and error handling
and whatnot we need to put in send
message but at the very least the
looping was redundant fortunately when
it's this kind of case the solution is
clear and easy just get rid of all the
extra stuff the mutant didn't have that
will also make our code more
maintainable by getting rid of
unnecessary cruft
so to summarize mutation testing is a
powerful technique to ensure that our
code is meaningful and our tests are
strict it's not so easy oh sorry one
more thing that is easy it's easy to get
started with in terms of setting up the
tools and if need be annotating our
tests which may be tedious but at least
it should be easy but it's not so easy
to interpret the results nor is it easy
on the CPU even if these drawbacks mean
that it might not be appropriate for our
current projects right now
I still think it's just a really cool
idea in a geeky kind of way
if you'd like to try mutation testing
for yourself here's a list of ones for
popular languages and platforms and some
other ones I doubt many of you are doing
fortran 77 these days as for JavaScript
the only one I know of is Stryker there
used to be one that was a plugin for the
grunt task runner but that project has
folded and it's code has been migrated
into Stryker
anybody still needs some time to take
pictures I saw some cameras up there
okay lastly a couple shoutouts first to
talk to a consulting Network I'm in
whose speaker's network helped me
prepare and practice this presentation
please use that referral link if you
what IRS are join us and secondly to
Marcus sharp
creator of mutant a mutation testing
tool for Ruby main when I've actually
used these been very willing to answer
my ignorant questions there being no
such thing as stupid questions & pratik
this presentation at least the longer
forms is the shortest one I've done so
far and if you have any questions while
we're not supposed to do Q&A but I'll be
around for the rest of the conference
and there's my contact information Oh
hallo and let's go get some lunch
[Applause]