So far, we have seen how we can easily create tests for our models based on our recent requirements. How to
manage, validate, and document. But how do we know if we have fully tested our design
with these test. Well, model coverage helps us validate our model tests by
measuring how completed the model objects are tested. For model objects, I mean our Simulink
blocks and states and transitions of Stateflow charts. Model coverage calculates
how much the test cases exercise simulation pathways to a model.
It is a measure of how fully the test cases test a model using a
pathways that the test cases are able to exercise.
And there are many industry-standard metrics, such as decision, condition, modified condition,
decision coverage, MCDC that we can use.
Lets see how we can achieve, see, and analyze these metrics easily in Simulink.
Before starting the test that we have created for BMS state machine, we have the possibility to
enable the test manager some options, that during the test run, can collect
coverage metrics that I said before. In the test file, in the coverage settings section,
I can enable to metrics we want before running the test.
You can see that I already enabled decision, condition,
MCDC before starting the test. This means that I can already see the results.
If we go back to the results tab, selecting each individual test, I can see the
the metrics calculated on that individual test. More important of
course, is to see the metrics on the full test code. And we can do this
by selecting the container of all test. In this case, we have reached
73% MCDC. How to analyze in the model what was covered and what was not?
By clicking on this button, the coverage will be highlighted on the model the colors red
and green. Green means total coverage, and red means not total coverage. We therefore need to do a more
in depth analysis. For example, we may have to go into this transition. But in talking about MCDC
we may have tested only some of the combinations of conditions and
operations. If you want more details here, you can open the coverage view, which will open
a new window on the right. Here, through hyperlinks, we can browse coverage and model, and see the details.
In this case, we discover that the condition of fault present has never been tested. So we have to add a new test
for this. Speaking of reporting, useful for certifications. Obviously it is done
automatically by clicking on this arrow, and an HTML file will be generated. Enclosing the details of what has been updated.
We know from many customers that as you define test and measure coverage,
it is a difficult challenge to reach 100% of coverage. And it is
frequently the last few percentages that take most of the efforts. This brings us to a new topic. The
automatic generation of new tests. Initial situation is this: the creation of test
cases to obtain coverage, which most of the time is partial. The test
generator in Simulink Design Verifier can help us in the workflow by taking
incomplete coverage as input, so that new tests are specifically targeted to missing
coverage. And when combined with initial test gives you a suite that can achieve the complete
coverage. Lets see how to activate this feature. If you remember, we had a partial
MCDC coverage of 73%. Simply using this button, we can
input the test generator and enter some information to say how and where to create
our new test. The test generator, starting from the starting coverage, will calculate some
goals and try to reach the total coverage to new test. For the models
I'll cover path. A new test harness with the new test will be automatically created and added
to the test manager. The new test can be seen from here. The
test generator has added 13 new iterations to this arrow. We can open the
test harness that contains that. And in choosing the creation of a similar builder we can find and
analyze the new 13 tests inside. Back to the test manager, we
can run this new test and see the results in terms of model coverage. The new tests cover
47% MCDC model coverage. The interesting part is to match the coverage of the
old test and the new ones, which we can calculate in this way. And we can see that coverage
is up from 73% to 87%. And that again,
we can analyze the transition still in red.
One question that might come naturally is why didn't we get 100% of the coverage? Well, in general there are several reasons
that could lead us to no full coverage on the model. The first one that comes to mind is
missing test. The second is missing requirements. In the previous case, since we invoked the automatic
test generator, the most likely cause are design errors. For example, dead logic.
Dead logic in our design, even the code generator will surely fail to achieve
100% coverage. Design Verifier gives us the ability to detect hidden
runtime errors statically without having to do any simulation. Support detection of
errors such as integer overflow, division by zero, array out of bounds,
range violations, and dead logic. These scenarios usually indicate bugs in your
designs. When we run these, are highlighted in green, and construct violations in red.
If you want more details, please refer to the short video on YouTube: How to Automatically Detect Design Errors in Your
Simulink Models.