10. Reactis Validator

Reactis Validator searches for defects and inconsistencies in your program. The tool enables you to formulate a property that program behavior should have as an assertion and perform an automated search for a violation of the assertion. If Validator finds an assertion violation, it returns a test that leads to the problem. This test may then be executed in Reactis Simulator to gain an understanding of the sequence of events that leads to the problem.

Validator also offers a mechanism called user-defined targets to specify specific test scenarios you want to create. User-defined targets may be seen as user-defined extensions to the built-in coverage metrics (statement, decision, condition, MC/DC, etc.) supported by Reactis: when generating test data, Tester will attempt to generate tests that satisfy the indicated scenarios.

Validator is particularly useful in requirements validation. Given a list of requirements on program behavior, you can formulate assertions to check whether the requirements are being satisfied and user-defined targets to define test scenarios intended to “stress” the requirements.

You use Validator as follows. First, you insert assertions and user-defined targets into your source code using the reactis_assert and reactis_target macros. We refer to assertions and user-defined targets as Validator objectives. Note that when using the instrumentation code mechanism described in the Instrumentation Code section, you can insert Validator objectives from within the Reactis GUI without modifying your application source code. The objectives and their location within the source code is managed by Reactis and stored in the .rsm file. Reactis Tester is then invoked on the instrumented code to search for assertion violations and paths leading to the specified coverage targets. The output is a test suite that includes tests leading to objectives found during the analysis.

The reactis_assert and reactis_target macros are defined in reactis_validator.h so this header file must be included in the C files that use the Validator macros.

As an alternative to reactis_assert, you can use the assert macro that is part of the ANSI C standard. To make Reactis treat assert macros as if they were reactis_assert, simply select the Create coverage targets for assert() statements setting in the C code tab of the Reactis Settings Dialog.

If you include NDEBUG as a Define in an .rsm file, then any assertion checking will be turned off for any assert, reactis_assert, and reactis_target macros in the C files and libraries of the component.

10.1. Validator Objectives

A Validator objective is a target created by Validator. Coverage of Validator objectives is tracked in the standard manner, namely through highlighting in the main Reactis panel, the Coverage-Report Browser, and generated reports. Also, in Simulator, hovering over an objective shows its coverage status and, if covered, the test and step when it was first covered. There are two types of Validator objectives, assertions and user-defined targets.

10.1.1. Assertions

Assertions specify properties that should always be true for a program under test. If the argument to an reactis_assert macro evaluates to a non-zero value we say it holds; if the argument evaluates to zero we say it is violated or does not hold. For a given assertion, Validator searches for a simulation run that leads to a configuration in which the assertion does not hold. An assertion is considered “covered” when a violation is found. So in contrast to targets, where being covered is considered good, covering an assertion is bad. We therefore highlight covered assertions in red and yellow, in contrast to targets where uncovered targets are highlighted in red.

10.1.2. User-Defined Coverage Targets

User-defined coverage targets extend the Reactis built-in targets described in the Reactis Coverage Metrics chapter (Statement, Decision, Condition, MC/DC, etc). Reactis will consider a user-defined target to be covered when the argument to a reactis_target macro evaluates to a non-zero value.

10.2. Use Cases of Validator Objectives

Consider a cruise control application that has the following requirement:

The cruise control shall remain inactive if the speed of the vehicle is less than 30 mph.

To properly test this requirement we need to execute a test in which an attempt is made to activate the cruise control while the speed of the vehicle is less than 30 mph. We can capture this as a simple user-defined target:

reactis_target(on && activate && (speed < 30))

This objective would be inserted at a location of the code that executes during every execution of the entry function such that on monitors whether the cruise control is turned on, activate monitors whether an attempt has been made to activate the cruise control, and speed is the speed of the vehicle. When this expression becomes true, we have found a test to test our requirement.

We can test whether the application’s response to this test is correct with the following assertion:

reactis_assert(!(active && (speed < 30)))

This assertion is inserted so that active monitors whether the cruise control is active. If this expression ever becomes false, then the requirement is violated.

The assertion and user-defined target that capture the simple requirement of the previous example depend only on the values of program variables at a given point in time. Note, however, that more complex properties involving timing can be constructed by creating Validator objectives that maintain state in between simulation steps. Consider another requirement for a cruise control:

When active, cruise control shall not permit actual, desired speeds to differ by more than 1 mph for more than 3 seconds.

This requirement can be checked by an assertion that monitors the vehicle speed, whether the cruise control is active, and the speed at which the driver has set the cruise control. A convenient way to implement such an assertion is as a C function with three input parameters active, speed, and desiredSpeed and returning a zero value if the requirement is violated.

The C function that implements the assertion is shown in Figure 10.1. It implements a simple state machine in which the static local variable state records the current state. If state SPDCHECK_ACTIVE_ERROR is ever reached, then the assertion is considered violated.

Fig. 10.1 The implementation of the speed maintenance assertion.

10.3. Instrumentation Code

There are two ways to add Validator objectives to a program under test. One way is to insert reactis_target and reactis_assert statements directly within the source file (typically enclosed with #ifdef __REACTIS__ … #endif). Another way is to add instrumentation code from within Reactis. A block of instrumentation code is just one or more lines of C code that is “virtually” inserted at some location within your application source code. Each block of instrumentation code and its location are stored in the build file and integrated with the source code at compile time. This allows you to add Validator objectives (and possibly other test code) to the program under test without modifying the source code.

10.3.1. Advantages and Disadvantages of Instrumentation Code

Instrumentation code has the following advantages over directly modifying the source code.

1. If you normally save the Validator objectives in the code under test, using instrumentation code will eliminate the risk that the Validator objective code will affect the code under test outside of Reactis, such as causing a compilation failure or changing runtime behavior.

2. The instrumentation code is highlighted within Reactis, making it easy to distinguish from the rest of the code.

3. If you normally add the same Validator objectives each time a new revision of the code is tested, Reactis will automatically insert the instrumentation code at the appropriate locations with each revision if possible.

The main disadvantage of instrumentation code is that if a source file is modified, there is a chance that Reactis won’t be able to automatically relocate instrumentation code within the file. In this case you will need to manually place the instrumentation code in the correct location; but, in such cases Reactis will flag instrumentation code with a warning symbol to alert you that it may require relocation.

10.3.2. Identifying Instrumentation Code

By default, instrumentation code is highlighted in the main panel with a colored background, as shown in Figure 10.2. Each line of instrumentation code is identified by the number of the line in the source file which immediately precedes the block of instrumentation code, followed by a plus sign (+) and then the number of the line within the block. For example, the lines for the instrumentation code in Figure 10.2 are identified as 106+1, 106+2, … 106+5. This double line number is displayed when hovering on the line number of a non-empty line of instrumentation code, as shown in Figure 10.2.

Fig. 10.2 The Reactis Tester progress dialog.

10.3.3. Adding Instrumentation Code

When Simulator is disabled, instrumentation code can be added by right-clicking on the line number of a regular source code line (i.e., any line which does not contain instrumentation code), and selecting Add Instrumentation Code. This will open an editor window similar to the one shown in Figure 10.3.

Fig. 10.3 The C Code Editor is used to edit instrumentation code.

10.3.4. Editing Instrumentation Code

Instrumentation code can be edited by right-clicking on the line number of any line of instrumentation code and selecting Edit/Move Instrumentation Code. This will open an instance of the C Code Editor for the block of instrumentation code which was clicked on, as shown in Figure 10.3. The following numbered items appear in Figure 10.3:

1. The text of the instrumentation code is edited within this panel.

2. This drop-down list is used to determine if the code will be placed before or after the line number shown to the right (item 4).

3. This button opens the help dialog for the C Code Editor.

4. This entry box contains the line number where the instrumentation code will be placed. The placement will be either before or after the line depending on the setting of item 2.

5. This button saves all changes and closes the editor.

6. This button discards all changes and closes the editor.

10.3.5. Cutting, Copying, and Pasting Instrumentation Code

Instrumentation can be cut, copied, or pasted directly within the main panel of the Reactis top-level window. Before a cut, copy, or paste operation can be performed, one or more lines of instrumentation code must be selected. To select a single line of instrumentation code, left-click on the line number. To select multiple lines of instrumentation code, left-click on a line number, then shift left-click on a second line number within the same block of instrumentation code. This will select all lines between (and including) the lines which were clicked on. Selected lines are highlighted with a dark blue background.

Once one or more lines of instrumentation code have been selected, right-clicking on any line number and selecting Copy Instrumentation Code will copy the selected lines to the clipboard. Right-clicking on any line number and selecting Cut Instrumentation Code will copy the selected lines to the clipboard and then delete them.

Any time the clipboard is not empty, right-clicking on a line and selecting Paste Instrumentation Code will insert the instrumentation code containing the contents of the clipboard immediately before the line which was clicked on.

10.3.6. Verifying the Location of Instrumentation Code

Since instrumentation code is not stored in a source file, it is possible that changes to a source file will cause the location of instrumentation code within the source file to change. Reactis stores information about the surrounding context of each block of instrumentation code and automatically adjusts the starting location each time the file is loaded. If the lines immediately surrounding the instrumentation code have not changed, then the relocation will typically succeed (there will be a warning message stating that the location has changed). The new location of the instrumentation code does not take full effect until you select Edit -> Save Instrumentation Code.

After some source file modifications, Reactis will be unable to accurately determine the new location of an instrumentation code block. In this case there will be a warning message stating that the location could not be determined and the current location is unreliable. Blocks of instrumentation code with unreliable location are shown with red text in the .rsm file Editor (see the Instrumentation Code section). In the main window, instrumentation code with an unreliable location is indicated by a yellow background and warning symbol in the line number column. The location can be verified by either moving the instrumentation code (see the Moving Instrumentation Code section), or right-clicking on a line number within the block and selecting Keep Instrumentation Code Here.

10.3.7. Moving Instrumentation Code

The easiest way to move instrumentation code to a new location is to right-click on any line number within the instrumentation code block and select Edit/Move Instrumentation Code). The location can then be changed within the C Code Editor (items 2 and 4 of Figure 10.3.

Alternatively, you can also move a block of instrumentation code by selecting the entire block and then cutting and pasting it to a new location.

10.3.8. Removing Instrumentation Code

The easiest way to remove a block of instrumentation code from a source file is to right-click on any line number within the instrumentation code block and select Remove Instrumentation Code.

Alternatively, you can also remove a block of instrumentation code by selecting the entire block and then cutting it.