So,
I was asked the question - how do we document what our automated tests do and
cover, without adding a lot of overhead?
And - how do we know what is not covered by automation?
So, I'm going to try and show some tips towards doing that, in this video.
So, my name is Alan Richardson. You can find me at eviltester.com.
And we're going to look at essentially, modelling.
I've written about modelling for a long time.
The first blog posts I ever wrote were all about modelling.
This is back in 2002 and I suspect I wrote this beforehand.
All coverage assessments are comparing some sort of model to some sort of thing,
to see how they map. The more they map, the more coverage we have.
Now this blog post scares people because it's got test scripting in it and
people think - Oh, test scripting - and - he wants us to write scripts.
When we do automated execution, when we write code to automate applications,
we're essentially writing test scripts.
When we write a JUnit method and it has @Test on the top that's pretty much a
test script. I'm saying, do this, do this, do this, do this, check this,
check this. That's what a test script is. But back in the day,
when I was looking at test scripting,
test scripts essentially were based on models.
I want to do a flow. I want to go from here and to here,
across to here, back to here, do these things. So,
I could generate the test scripts. Now,
there's a concept called Model Based Testing that we're not really covering
here,
where you can derive paths through a model and it will automatically execute.
But all coverage is based on a model.
When we look at code to attempt to go - what does this code cover?
I have a mental model of what that code could be doing.
When I look at the code,
I then map the code back on to my mental model to then go right,
what is actually doing? What is it covering? Was it not? We always have this.
We always have models.
And it can be hard to analyse an entire set of tests.
If I try to analyse all the tests in here, even though there's not that many,
I need a pretty big mental model. So, when we're looking at code,
we tend to work in very small chunks to keep our models very tight.
Where we want to try and analyse system coverage,
then we move over to the left hand side and then we start looking at test names,
method names, class names,
packages to get a feel for the coverage. Because looking at this,
I can see that I have some abstractions.
Abstractions are useful because they hide implementation and make test code
easier to read and understand. So here's an example. Here's an abstraction.
I've got a navigation menu on my Pulper and I'm going to set it to be a
particular version, that I can understand what that means.
There's aspects here that are not well abstracted.
So if you don't know, that there's a kind of secret flag,
in the Pulper set in the version, then this won't make much sense.
But an abstraction could make that clear.
If I had an abstraction that said - go to the Pulper for a specific version,
put the version in, then it's clear what's happening in the steps.
So abstractions are really useful for helping us read code clearly and simply,
to understand where it's doing, to map it against the model in our head.
So other things that can help us are test names.
If this test name is readable and understandable,
then I can look at it and go - right, what is this test doing?
Now I've been experimenting with JUnit5 and parameterised tests to try and make
that even more simple. So,
rather than having to rely on this test name,
which doesn't have spaces in it because we're trying to make this syntactically
correct code, I've got a display name that will be displayed.
Then this test will be run multiple times with the parameter that I'm passing
it. So,
I'm making parameter based test and you can see the output when they're executed
here. This is very readable.
So looking at the output of the test execution because I have good names or
because I'm making them readable through annotations,
it's a lot easier for me to review the output to understand the coverage,
based on a mental model that I have of the system.
When we are working on agile projects,
we might have stories and the stories are a small discrete chunk of
functionality or value that we're working on. So,
I can review the code check-ins that we've created for that particular story.
Because hopefully they've got the story ID and I can search through the version
control system and see what code we wrote to match that story.
I can review the code and the test for that against that story.
What I've got is a model which is a story and I compare the acceptance criteria
there, into the code that we've written.
And the story helps keep the reviews small enough that I can hold it in my head.
When we have models that go beyond that, then we need to do a little bit more.
We can't hold all that stuff in our head.
Other things that are useful is making the data visible.
The data that's important for the test should be visible.
The data that is unimportant,
I don't want to see in the task because it's just going to distract me. So,
here the data is coming in from a method. If I look up here,
I can see it's an integer stream going from one to the maximum version.
This is an abstraction, so it updates all the time. But if I click in there,
I can see it's 11 so I know that this is going to run from one to 11 and the
output tells me that.
This data is visible because it's the important data for the test.
If I was creating forums or typing things in,
and it was unimportant to the test,
I would want that data to be hidden in an abstraction there. So,
I don't see it and don't get distracted. So, if this said something.....
or if I was creating a book, I didn't care what data was in.
Rather than having something that says - create book,
then the title and the author's name than something else,
I would just want - create a book. And then the book comes back. So,
I want abstractions that make it clear what is happening,
so that I can review it and assess coverage,
but don't distract me with irrelevant information for that particular test.
So,
the structure of the code is also very important to support our review process.
So, I've only just started working on this project.
There's not a lot of tests in here, but looking at the structure of the code,
I can see it's all related to a particular application - the Pulper.
And I can see that I have abstractions in here.
This will eventually move into the main section so that it doesn't distract me
when I'm reviewing the test code.
And you can see that I have tests for all the versions. So,
for all the versions I'm using the navigation menu - can see that I don't have
anything for version one, two, or three specifically. So,
I know there's a lack of coverage there, for those versions.
I'd have nothing at all from versions 004 through 011. So I know I have gaps.
I can see that in the versioning, I'm checking if I can change the version.
And then the methods tell me how I am changing the version, with the admin menu,
with a direct URL via the admin page.
The naming is helping me assess the coverage.
Now if I ran the test I would have output down here and this would be even
easier for me to read.
I've got multiple places to review this in terms of coverage. So,
this isn't an automated coverage process.
This is a manual review coverage process. So,
the hard part is more - what are we not covering? So,
here it's obvious to me that I'm not covering these things because I have gaps
which are obvious from the package.
So if I can make the model clear and obvious,
then I can see that there are gaps.
But I don't want to have models in here that are hard to maintain.
I don't want to really have lots of annotations in here. Like,
I'm just gonna make this up, a story, oops story.
And then if that was a number of the story,
because then I have to keep maintaining this. This doesn't automatically update.
I'd have to click through and review the story to make sure this test applies.
When I'm working on a story by story basis.
I would rather review it at the point that we're creating the code and look at
the code check-ins, to review it. Rather than rely on an annotation.
Because this is essentially a comment and it can go out of date. So,
things I've done in the past is when I'm testing APIs or websites,
I've run the test and put it through a proxy,
then captured all the traffic in a HAR file,
which is an XML format file.
Then analyse that file to compare against a model of the application. So,
if I have a sitemap say, for a website,
I could compare all the URLs and make sure I hit them in the HAR file and that
gives me some assessment of - have I done enough coverage to check against the
sitemap? If I had an API in the past,
I've built models of the API so I know all the end points,
I know all the verbs and then I'll look through all the traffic and go - did I
hit this endpoint with all the verbs? If I haven't,
then I know I have gaps in my coverage.
You can write relatively simple things that help you find gaps in your coverage,
but you're still making a model and sometimes this has to be very custom.
So, what a lot of people do, is they use tooling to help them.
One tool that people use is Cucumber.
So any of the BDD style tools. Because what these let you do,
and I'm just using their website as an example here,
I don't have any examples in my code.
You can see here that we have a feature,
we have a scenario and this is written in English. So,
it's written at a high level domain specific language which makes it easy to
read.
The other thing it lets you do is you can have data parameterisation so here is
an example of Gherkin, using a data table. Now,
I assume when I'm seeing this that the data that's exposed in this table is
important, right?
Because this is what I'm reviewing in terms of coverage.
I assume that if the name was unimportant we would have something in the
abstraction that creates a user and with any name and we don't care about it.
What's important is that there's a user with these email addresses or these
Twitter handles, right?
When I want to use Gherkin to illustrate coverage,
I only want to expose the information that is important.
Anything that's not important will distract me in a review,
will distract me when I'm doing maintenance,
will make me think I've got potentially more coverage than I might have.
The Gherkin, when we're modelling coverage there,
should be very specific. Now,
I like the use of parameterised data tests in JUnit.
The output is very readable.
And I don't have to bring in another tool.
But I can still achieve a lot of the same aims which are making readable code,
making the data that we're using visible,
making the high level conditions that were checking and asserting for visible in
the output.
People tend to go overboard when they start using tools like Gherkin and do
everything in Gherkin.
Use Gherkin to support you when you're modelling automated
execution code;
when you need something that is to be reviewed by people that either don't know
how to code, or won't learn how to code,
or people who are going to review it without anyone that knows code being with
them.
Now you can see here it's very easy for someone who doesn't know how to code to
review the coverage here because we're outputting it in English language.
Someone can review that.
I could train people to review the test code here.
And if I really wanted someone who didn't know how to code to review this,
I would rely on more abstractions in here to make my code more readable.
Abstractions can help you move away from having to use tools like this for
automated execution and then you don't run the risk of being shouted at because
you start saying we're doing BDD when you're not doing BDD.
It's perfectly valid to use Gherkin in combination with Cucumber to support you
for making some of your coverage visible through a domain specific language.
Just don't use it for everything and certainly don't think that you're doing BDD
when you do this. So, that's a high level overview of what we're doing.
Because remember, what we've got, is we've got,
trying to use code that is packaged in such a way;
that makes it easy to review,
that exposes gaps so we can see where the coverage is,
that outputs in a form that is easy for us to review and understand,
that when we want to work at a lower level in terms of looking at the story
acceptance criteria or the acceptance criteria in a test the test itself is very
readable. So we'd improve this to support that type of coverage review.
We can extend our automated execution to run through proxies or to log more
information and capture that and then process it after the fact,
to compare it to custom models that we create,
that can give us some assessments outside the code here and help us look for
gaps. But that takes a little bit more work.
Or we can use tools like Gherkin to model aspects of our execution,
but not everything, just to make certain parts visible.
But modelling is the key thing we're doing here. Now,
modelling doesn't mean it has to be a diagram because a graph,
the underlying mathematical model from this is not a diagram.
This is a diagrammatic form of the model. So,
we create models that we can compare with something else and all coverage is
based around models. And if you're reviewing the code to assess coverage,
you're using a mental model or you may have stuff written down that you're
comparing with it. So, someone's mental model has been put on paper.
The more that we can automate the coverage,
the better off we are it because we won't let things slip through and we won't
make as many mistakes. This data is visible.