Chapter 10 Reactis Validator
Reactis Validator searches for defects and inconsistencies in your program. The tool enables you to formulate a property that program behavior should have as an assertion and perform an automated search for a violation of the assertion. If Validator finds an assertion violation, it returns a test that leads to the problem. This test may then be executed in Reactis Simulator to gain an understanding of the sequence of events that leads to the problem.
Validator also offers a mechanism called user-defined targets to specify specific test scenarios you want to create. User-defined targets may be seen as user-defined extensions to the built-in coverage metrics (statement, decision, condition, MC/DC, etc.) supported by Reactis: when generating test data, Tester will attempt to generate tests that satisfy the indicated scenarios.
Validator is particularly useful in requirements validation. Given a list of requirements on program behavior, you can formulate assertions to check whether the requirements are being satisfied and user-defined targets to define test scenarios intended to “stress” the requirements.
You use Validator as follows. First, you insert assertions and user-defined targets into your source code using the reactis_assert and reactis_target macros. We refer to assertions and user-defined targets as Validator objectives. Note that when using the instrumentation code mechanism described in Section 10.3, you can insert Validator objectives from within the Reactis GUI without modifying your application source code. The objectives and their location within the source code is managed by Reactis and stored in the .rsm file. Reactis Tester is then invoked on the instrumented code to search for assertion violations and paths leading to the specified coverage targets. The output is a test suite that includes tests leading to objectives found during the analysis.
The reactis_assert and reactis_target macros are defined in
As an alternative to reactis_assert, you can use the
If you include
10.1 Validator Objectives
A Validator objective is a target created by Validator. Coverage of Validator objectives is tracked in the standard manner, namely through highlighting in the main Reactis panel, the Coverage-Report Browser, and generated reports. Also, in Simulator, hovering over an objective shows its coverage status and, if covered, the test and step when it was first covered. There are two types of Validator objectives, assertions and user-defined targets.
Assertions specify properties that should always be true for a program under test. If the argument to an reactis_assert macro evaluates to a non-zero value we say it holds; if the argument evaluates to zero we say it is violated or does not hold. For a given assertion, Validator searches for a simulation run that leads to a configuration in which the assertion does not hold. An assertion is considered “covered” when a violation is found. So in contrast to targets, where being covered is considered good, covering an assertion is bad. We therefore highlight covered assertions in red and yellow, in contrast to targets where uncovered targets are highlighted in red.
10.1.2 User-Defined Coverage Targets
User-defined coverage targets extend the Reactis built-in targets described in Chapter 7 (Statement, Decision, Condition, MC/DC, etc). Reactis will consider a user-defined target to be covered when the argument to a reactis_target macro evaluates to a non-zero value.
10.2 Use Cases of Validator Objectives
Consider a cruise control application that has the following requirement:
The cruise control shall remain inactive if the speed of the vehicle is less than 30 mph.
To properly test this requirement we need to execute a test in which an attempt is made to activate the cruise control while the speed of the vehicle is less than 30 mph. We can capture this as a simple user-defined target:
This objective would be inserted at a location of the code that executes during every execution of the entry function such that on monitors whether the cruise control is turned on, activate monitors whether an attempt has been made to activate the cruise control, and speed is the speed of the vehicle. When this expression becomes true, we have found a test to test our requirement.
We can test whether the application’s response to this test is correct with the following assertion:
This assertion is inserted so that active monitors whether the cruise control is active. If this expression ever becomes false, then the requirement is violated.
The assertion and user-defined target that capture the simple requirement of the previous example depend only on the values of program variables at a given point in time. Note, however, that more complex properties involving timing can be constructed by creating Validator objectives that maintain state in between simulation steps. Consider another requirement for a cruise control:
This requirement can be checked by an assertion that monitors the vehicle speed, whether the cruise control is active, and the speed at which the driver has set the cruise control. A convenient way to implement such an assertion is as a C function with three input parameters active, speed, and desiredSpeed and returning a zero value if the requirement is violated.
The C function that implements the assertion is shown in Figure 10.1. It implements a simple state machine in which the static local variable state records the current state. If state SPDCHECK_ACTIVE_ERROR is ever reached, then the assertion is considered violated.
10.3 Instrumentation Code
There are two ways to add Validator objectives to a program under test.
One way is to insert reactis_target and reactis_assert
statements directly within the source file (typically enclosed with
10.3.1 Advantages and Disadvantages of Instrumentation Code
Instrumentation code has the following advantages over directly modifying the source code.
The main disadvantage of instrumentation code is that if a source file is modified, there is a chance that Reactis won’t be able to automatically relocate instrumentation code within the file. In this case you will need to manually place the instrumentation code in the correct location; but, in such cases Reactis will flag instrumentation code with a warning symbol to alert you that it may require relocation.
10.3.2 Identifying Instrumentation Code
By default, instrumentation code is highlighted in the main panel with a colored background, as shown in Figure 10.2. Each line of instrumentation code is identified by the number of the line in the source file which immediately precedes the block of instrumentation code, followed by a plus sign (+) and then the number of the line within the block. For example, the lines for the instrumentation code in Figure 10.2 are identified as 106+1, 106+2, … 106+5. This double line number is displayed when hovering on the line number of a non-empty line of instrumentation code, as shown in Figure 10.2.
10.3.3 Adding Instrumentation Code
When Simulator is disabled, instrumentation code can be added by right-clicking on the line number of a regular source code line (i.e., any line which does not contain instrumentation code), and selecting Add Instrumentation Code. This will open an editor window similar to the one shown in Figure 10.3.
10.3.4 Editing Instrumentation Code
Instrumentation code can be edited by right-clicking on the line number of any line of instrumentation code and selecting Edit/Move Instrumentation Code. This will open an instance of the C Code Editor for the block of instrumentation code which was clicked on, as shown in Figure 10.3. The following numbered items appear in Figure 10.3:
10.3.5 Cutting, Copying, and Pasting Instrumentation Code
Instrumentation can be cut, copied, or pasted directly within the main panel of the Reactis top-level window. Before a cut, copy, or paste operation can be performed, one or more lines of instrumentation code must be selected. To select a single line of instrumentation code, left-click on the line number. To select multiple lines of instrumentation code, left-click on a line number, then shift left-click on a second line number within the same block of instrumentation code. This will select all lines between (and including) the lines which were clicked on. Selected lines are highlighted with a dark blue background.
Once one or more lines of instrumentation code have been selected, right-clicking on any line number and selecting Copy Instrumentation Code will copy the selected lines to the clipboard. Right-clicking on any line number and selecting Cut Instrumentation Code will copy the selected lines to the clipboard and then delete them.
Any time the clipboard is not empty, right-clicking on a line and selecting Paste Instrumentation Code will insert the instrumentation code containing the contents of the clipboard immediately before the line which was clicked on.
10.3.6 Verifying the Location of Instrumentation Code
Since instrumentation code is not stored in a source file, it is possible that changes to a source file will cause the location of instrumentation code within the source file to change. Reactis stores information about the surrounding context of each block of instrumentation code and automatically adjusts the starting location each time the file is loaded. If the lines immediately surrounding the instrumentation code have not changed, then the relocation will typically succeed (there will be a warning message stating that the location has changed). The new location of the instrumentation code does not take full effect until you select Edit -> Save Instrumentation Code.
After some source file modifications, Reactis will be unable to accurately determine the new location of an instrumentation code block. In this case there will be a warning message stating that the location could not be determined and the current location is unreliable. Blocks of instrumentation code with unreliable location are shown with red text in the .rsm file Editor (see Section 5.6.1). In the main window, instrumentation code with an unreliable location is indicated by a yellow background and warning symbol in the line number column. The location can be verified by either moving the instrumentation code (see Section 10.3.7), or right-clicking on a line number within the block and selecting Keep Instrumentation Code Here.
10.3.7 Moving Instrumentation Code
The easiest way to move instrumentation code to a new location is to right-click on any line number within the instrumentation code block and select Edit/Move Instrumentation Code). The location can then be changed within the C Code Editor (items 2 and 4 of Figure 10.3.
Alternatively, you can also move a block of instrumentation code by selecting the entire block and then cutting and pasting it to a new location.
10.3.8 Removing Instrumentation Code
The easiest way to remove a block of instrumentation code from a source file is to right-click on any line number within the instrumentation code block and select Remove Instrumentation Code.
Alternatively, you can also remove a block of instrumentation code by selecting the entire block and then cutting it.