>>Nan Li: Hello, everyone.
My name is Nan Li.
I got my Ph.D. from George Mason University this year.
Currently, I'm a test engineer at Medidata Solutions where I'm leading the research and
software testing and developing test frameworks.
Today I'm going to talk about model-based testing and how to do test automation in model-based
testing.
In model-based testing, testers derive tests from behavioral models.
Generally, there are two reasons for us to use model-based testing.
First, models are widely used in industry to design complicated software systems.
For instance, UML state machine diagrams are used to describe behaviors of software systems
such as control software used at high-speed trains, cars and planes.
So it's natural for us to use design models as test models.
Second, we use models to decide what to test.
At the international conference on software testing, verificationand validation 2014,
the director from Lockheed Martin, Tom Wissink, gave a keynote speech.
In his speech he showed how a software system evolved for fighters developed by Lockheed
Martin.
We were able to see software is very complicated with milliions of lines of code, with lots
of complicated software components.
So we cannot test everything.
We have to decide what to test.
So we use models to get abstraction of the system at some level.
In his conclusion, he thinks model-based testing is the future.
Next, I'm going to show basic steps of model-based testing.
In this model-based testing research, I generated criteria-based tests.
That means I used coverage criteria to generate test requirements.
Coverage criteria are rules about how you generate test requirements.
Test requirements are artifacts that have to be covered in tests.
I use the UML state machines to run the models, so these diagrams are treated as general graphs.
Therefore, I applied graph coverage criteria to the models to generate the test requirements,
which are subpaths on a graph.
Test requirements have to be covered in tests, especially in abstract tests first.
Abstract tests are defined in terms of the model, which are -- the test paths that starts
from an initial node and ends with a final node.
The abstract tests cannot be executed directly because they are not concrete tests.
They have to be converted into concrete tests such as jUnit tests.
To transform from abstract tests to concrete tests, testers have to provide additional
information such as mappings and test value.
When you have concrete tests, they can be executed and reported.
In the next talk I'm going to focus on how to automate the mapping process from abstract
tests to concrete tests.
I'm going to show an example of how we do model-based testing.
This is a finite state machine for a simplified vending machine system.
State one is the first stage when the vending pre-system starts, and state four is the final
state.
There are three basic actions.
Service persons add chocolates into the vending machine.
Customers then insert coins and then buy chocolates.
There are some constraints defined in this vending machine system.
For instance, in state one, credit has to be equal to zero and the number of chocolates
in stockhas to be equal to zero.
We can apply various coverage criteria to this model.
If we use node coverage, all states have to be covered in tests.
If we use edge coverage, all transitions have to be covered in tests.
If we use a more complicated coverage criterion, the test requirements may be a set of subpaths
with various lengths.
So a problem is that how should we generate test paths to cover all the test requirements
with minimum cost?
Cost can be defined in terms of the total number of the test paths, the total number
of nodes in the tests, etc..
This problem has proven to be a set of NP-complete problems.
If you have more interest, you can read my ICST 2012 paper and my dissertation.
Now, abstract tests have actions.
For each action we have to find corresponding test code for them.
In the meanwhile, constraints in the tests have to be covered as well.
So how to transform abstract tests to concrete tests is called the mapping problem.
In this test path, this test path has adhoc action appear twice, Coin action appear twice,
and GetChoc appear once.
If we have a complete set of test paths, we find out these three actions will appear multiple
times.
If we generated concrete tests for this abstract test set,
testers may write lots of redundant test code.
The reason is the test code for the same action may be the same or similar, so we want to
automate this mapping process.
The problem of previous research is that they use a lot of additional models such as object
diagrams, class diagrams and use case diagrams.
And these models have to be highly consistent.
That means, a parameter used in one model has to be recognized in another model.
In practice, this is really hard for testers to maintain the highly consistent relationships
among them.
So my solution is that allow the testers to use a test automation language to create mappings
from the model to executable test code by using only one behavioral model, such as a
state machine diagram.
For state machine diagrams, there are two identifiable elements that need mappings:
transitions and constraints.
Testers can create element mappings for transitions and constraints.
If the test code of element mappings require objects, testers can create object in object
mappings to initialize objects.
Next I'm going to show you the definitions of element mapping and object mapping.
Each element mapping has to have a unique mapping name, following the keyword mapping.
All keywords are in red.
Each element mapping can be defined for a constraint or a transition.
If the test code of element mappings require objects and we can use additional object mappings
to initialize these objects.
(indiscernible) square brackets means this part is optional because this part is using
only for element -- element mappings for constraints.
Constraints can be used as preconditions, post conditions, and the state invariants.
Test code is included in the curly brackets.
Similarly, object mapping has to have a unique mapping name, and it also has to specify the
type of class and the object name.
If the test code for this object initialization also requires additional parameters, we can
use additional object mappings to initial parameters.
Similarly, test code is included in the curly brackets.
Next I'm going to show a few examples about how we write element mappings and object mappings.
This is an example of element mapping for transition coin.
In the test code, first we need to initial the vending machine system and then we need
to insert a dollar in the vending machine system.
If the vending machine object initialization is also used by other element mappings, we
can extract this part and put it into a separated object mapping.
And we give a mapping name vmlnit.
Now, the previous element mapping becomes this, which requires the object mapping in
it.
This is another element mapping for a constraint, which requires the credit has to be greater
than zero.
This constraint is used as state invariants in state two and state four.
Now that we have mappings for constraints and transitions, and then concrete test generation
can be automated.
For the test automation, we have to select the proper mappings for transitions so that
constraints can be satisfied.
The constraints in state three are satisfied, we need to select a mapping for coin so that
constraints in state four are also satisfied.
If selective mapping cannot find the constraints in state four, we have to select another mapping.
If all mappings cannot satisfy the constraints, testers will be notified and they need to
provide additional mappings.
If testers are not able to provide any mapping to satisfy the constraints, then that means
either the model or program has something wrong, or there is a mismatch between the
model and program.
In summary, testers can use the test automation language to automate the mapping process from
abstract tests to concrete tests by using only one behavioral model.
From the study of model-based testing research, I learn one lesson.
I was able to do some test automation techniques for the mapping process and the test generation.
The basis is that requirements are included in a format that can be understood by computers.
In this case, requirements are described in state machine diagrams.
So we want to apply this idea to practice at Medidata.
At Medidata Solutions, we are processing and transforming terabytes or even petabytes clinical
trial data for reporting purposes.
If requirements are written in plain English, it's really hard for us to do any test automation,
so we plan to use a different test automation language or called a specification language
to transform original requirements into some format.
Then our test framework can read this newly formatted requirements and evaluate the transformed
data against original data, and then the whole validation process will be automated.
Thank you for your listening.
[ Applause ] >>Sonal Shah: Thank you very much, Nan.
Do we have any questions on the moderator link?
No.
Any live questions?
All right.
>>> So model-based testing is interesting for small applications with a small state
space.
But as soon as you have a more complex application, modeling that application becomes much more
difficult.
And are you doing anything different here to make that easier?
>>Nan Li: Yes.
You are absolutely right.
When software systems are complicated, we have to design at which level we generate
models.
If we use too many parameters.
We will have too many states.
And then we will have too many tests.
>>> So are you doing anything differently?
Are you doing anything to make it simpler to model more complex application?
Because otherwise, you end up recreating the entire application in the model test space.
>>Nan Li: Yes.
So for my research, my focus was on how to derive a test from the model, given a different
model.
But next, at Medidata, we plan to do some model-based testing for real in practice,
we will figure out how we generated models.
And that's going to be our future research project.
>>Sonal Shah: Thank you, Nan.
>>Nan Li: Okay.
>>Sonal Shah: Great talk.
[ Applause ]