Maximizing Coverage

14. Maximizing Coverage#

Reactis Tester and Validator work hard to generate tests that maximize covered targets. Nevertheless, you may encounter models that present challenges to Reactis: generated test suites may not provide adequate coverage. This chapter discusses strategies for overcoming such situations.

When attempting to improve coverage, the following general strategy is often useful:

Load the model in Reactis and start Simulator.
Load a test suite.
Make sure Coverage > Show Details is selected, and run all tests in the suite.
Inspect uncovered targets. Are unexercised targets coverable?

No:
Perhaps the uncovered targets are in fact unreachable. The unreachable targets might be intended (for example, the upper or lower bound of a saturation block that you never expect to be exercised) or unintended due to a modeling error.

Yes:
If you determine that the uncovered targets could be covered, then try applying one of the techniques described in the following sections to enhance the coverage of the test suite.

14.1. Constraining Input Values#

In many cases coverage can be improved if tighter bounds are imposed on input values by inport constraints in the .rsi file. First, any range information for inputs should be entered. For example, for vehicle speed, rather than allow any double precision floating point value, we narrow the set of acceptable value to between 0 and 200. This greatly reduces the set of values for this input that Reactis must consider when generating tests. In some cases the allowable set of inputs can be reduced even further to a set of enumerated values. For example, a flag may be restricted to the set including only 0 and 1. In general, the goal is to reduce the number of possible values Reactis must consider when generating tests.

Probability weights may also be used to affect the likelihood with which different conditions become true. For example, in some models, large portions of the model become unreachable if an input flag has a certain value (e.g. this input might indicate the system being modeled should shut down). Assigning low probabilities to those values can increase coverage. The Reactis Info File Editor chapter describes how to use the Reactis Info File Editor to set constraints for top-level inports.

When modifying the .rsi file in this fashion, it is good practice to save the modified .rsi file as a new file, so that the original settings are not overwritten. This can be done using the File > Save Info File As… menu item in the top-level Reactis menu, as described in The Reactis Info File Editor chapter. Note that this operation automatically associates the new .rsi file with the model. The link to the previous .rsi file can be restored using the File > Select Info File… menu item.

14.2. Modifying Test-Generation Parameters#

Another approach for improving coverage is to change the parameters used by Tester and Validator. These parameters may be set in the launch screens of the respective tool and may be categorized as follows:

Preload Files.: This parameter consists of a list of test suites that Tester / Validator should load initially and then try to extend. When a test suite provides insufficient coverage, a good strategy to try is to re-run Tester / Validator pre-loading the inadequate suite and instructing Reactis to try to enhance it.
Run for.: Letting Tester run for a longer period of time can often increase coverage. For large models overnight or even multi-day runs might be helpful. For long runs, specifying a run time in the Run for section is usually easiest (as opposed to giving a number of steps to be taken).

14.3. Using Timer Targets#

Certain model constructs that we term interruptible timers present a particular challenge to Reactis. These timers/counters require the application to hold some condition true for a given length of time or number of simulation steps. If the condition becomes false before the threshold is reached the timer/counter resets.

Reactis includes a mechanism called Timer Targets that enables you to identify those elements of your model which are timers or counters. When they are tagged, Reactis can often generate a test automatically to cause a timer/counter to tick up to its threshold. Please see the Reactis Validator chapter for details of how to instrument your model with a timer target directive.

The general procedure for using timer targets is as follows:

Identify a timer or counter in your model.
Instrument your model with a timer target that identifies a model item (Simulink block or Stateflow variable) as a timer or counter and specifies its start, end, and increment values.
Run Tester or Validator to generate a test which causes the timer to tick from its start to end value.

14.4. Interactively Tuning Test Suites#

If the previously described techniques still do not result in a test suite with a satisfactory level of coverage, then some hand-tuning of the suite may be necessary. Reactis Simulator and Tester / Validator contain several features designed to facilitate this process, including test-suite pre-loading (Tester), and test-suite execution, breakpoints and test-suite editing (Simulator).

The general strategy to be pursued may be summarized as follows:

Build an initial test suite.
Load the test suite into Simulator and add a test that causes a “hard-to-cover” target to be covered.
Pre-load the modified test-suite into Tester / Validator and rerun.

The idea is to help Tester cover things it finds difficult while still letting Tester do as much work as possible.

Coverage problems often arise because of difficult-to-satisfy conditions on Stateflow transition segments or within Simulink. Conditions might be difficult to satisfy for several reasons:

A condition might become true when a timer expires, meaning that control must reside in the source state of the transition segment labeled by the condition for a number of simulation steps. If other segments’ conditions become true in the meantime, then the state may be exited before the timer has a chance to expire.
A condition might be satisfied by a relatively small subset of the values its variables might be able to assume. An example would be a comparison of a floating-point variable to a constant: only one value for the variable makes the condition true.

The strategy for coping with these problems can be summarized as follows:

Identify an uncovered transition-segment whose source state has been covered by a previously constructed test suite.
Using the test suite and breakpoints, advance execution until the source state is active.
Using user-guided simulation (the User Input Mode section), perform simulation steps that cause the uncovered item to become covered.
Add these simulation steps to the test suite, save the suite, and re-run Tester preloading the modified suite.

The specific steps needed to implement this strategy are as follows (It is assumed that Simulator is already running and that the earlier test suite has been loaded):

Click on the Coverage menu and make sure that Show Details is enabled.
Designate the (covered) source state of the uncovered item as a breakpoint by right-clicking on it in the hierarchy panel of Simulator and toggling the breakpoint indicator.
In the Source-of-Inputs dialog governing the source of inputs, select All.
Run the tests by clicking the Fast Simulation button. Simulator will execute until the breakpoint is reached. Because execution breaks immediately prior to entry into the breakpoint state, hit the step button to cause the state to be entered.
Select User Guided Simulation in the Source-of-Inputs dialog.
Using your knowledge of the system, select (a sequence of) input values that drive execution to cover the uncovered item.
Select Test Suite > Add/Extend Test to cause a new test to be added to the current test suite, which can now be seen in the Source-of-Inputs dialog.
Save the current test suite by selecting Test Suite > Save.

The newly saved test suite may now be pre-loaded into Tester using the Preload Files selection box, and Tester rerun. This strategy may be repeated until an acceptable test suite is constructed. Be sure to save the resulting test suite so that it will not accidentally be overwritten by new runs of Tester.

14.5. Model Decomposition#

If the steps described above still result in unsatisfactory coverage, consider testing subsystems of your model independently. Reactis makes this possible through a facility to extract subsystems from a model. The idea is to extract a subsystem, along with portions of the model involved in triggering the subsystem, and store the results as a new model in a separate .slx file.

The subsystem extraction utility is invoked by loading a model into Reactis, then right-clicking on the Simulink subsystem or Stateflow chart to be extracted, and selecting the Extract Subsystem entry in the pop-up menu. In the resulting dialog a filename under which to save the extracted subsystem may be specified. Reactis will then extract the subsystem, save it under its new name, open a new Reactis window, and load the extracted model. Note that the extracted subsystem is saved as a standard .slx file. That makes it easy to edit the extracted model using Simulink if changes to the triggering mechanism or inputs or outputs of the model are necessary.

If the extracted subsystem is a triggered subsystem or located within a triggered subsystem or contains triggered subsystems whose triggers are connected to something outside the extracted subsystem, Reactis will keep all blocks and charts that are part of the triggering mechanism. This works recursively, i.e. if the triggering block or chart is triggered itself, Reactis will trace back that trigger, too.

To determine which subsystems might be candidates for independent testing, you can use the following strategy:

Generate a test suite for the full model.
Load the test suite into Reactis Simulator and select Coverage > Show Details.
Run the entire test suite on the model.
Inspect the subsystems within the model to look at the coverage obtained. Subsystems having poor coverage are good candidates for being tested independently.