Automated Testing and Validation with Reactis®

 Reactis for C User's Guide   Contents  |  Index
 Chapters:  1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17

Chapter 10  Reactis Validator

Reactis Validator searches for defects and inconsistencies in your program. The tool enables you to formulate a property that program behavior should have as an assertion and perform an automated search for a violation of the assertion. If Validator finds an assertion violation, it returns a test that leads to the problem. This test may then be executed in Reactis Simulator to gain an understanding of the sequence of events that leads to the problem.

Validator also offers a mechanism called user-defined targets to specify specific test scenarios you want to create. User-defined targets may be seen as user-defined extensions to the built-in coverage criteria (statement, decision, condition, MC/DC, etc.) supported by Reactis: when generating test data, Tester will attempt to generate test runs that satisfy the indicated scenarios.

Validator is particularly useful in requirements validation. Given a list of requirements on program behavior, you can formulate assertions to check whether a requirement is being satisfied and user-defined targets to define test scenarios intended to “stress” the requirement.

You use Validator as follows. First, you insert assertions and user-defined targets into your source code using the reactis_assert and reactis_target macros. We refer to assertions and user-defined targets as Validator objectives. Reactis Tester is then invoked on the instrumented code to search for assertion violations and paths leading to the specified coverage targets. The output is a test suite that includes tests leading to objectives found during the analysis.

The reactis_assert and reactis_target macros are defined in reactis_validator.h so this header file must be included in the C files that use the Validator macros.

As an alternative to reactis_assert, you can use the assert macro that is part of the ANSI C standard. To make Reactis treat assert macros as if they were reactis_assert, simply select the Create coverage targets for assert() statements setting in the C code tab of the Reactis Settings Dialog.

If you include NDEBUG as a Define in an .rsm file, then any assertion checking will be turned off for any assert, reactis_assert, and reactis_target macros in the C files and libraries of the component.

10.1  The Meaning of Validator Objectives

We now explain in more detail how Reactis interprets Validator objectives. Coverage of these objectives may be tracked in the standard manner, namely through highlighting in the main Reactis panel or in the Coverage-Report Browser. Also, in Simulator, hovering over a covered objective shows the test and step number when it was first covered.

10.1.1  Assertions

Assertions specify properties that should always be true for a program under test. If the argument to an reactis_assert macro evaluates to a non-zero value we say it holds; if the argument evaluates to zero we say it is violated or does not hold. For a given assertion, Validator searches for a simulation run that leads to a configuration in which the assertion does not hold. An assertion is considered “covered” when a violation is found. So in contrast to targets, where being covered is considered good, covering an assertion is bad. We therefore highlight covered assertions in red and yellow, in contrast to targets where uncovered targets are highlighted in red.

10.1.2  User-Defined Coverage Targets

User-defined coverage targets extend the Reactis built-in targets described in Chapter 7 (Statement, Decision, Condition, MC/DC, etc). Reactis will consider a user-defined target to be covered when the argument to a reactis_target macro evaluates to a non-zero value.

10.2  Use Cases of Validator Objectives

Consider a cruise control application that has the following requirement:

The cruise control shall remain inactive if the speed of the vehicle is less than 30 mph.

To properly test this requirement we need to execute a test in which an attempt is made to activate the cruise control while the speed of the vehicle is less than 30 mph. We can capture this as a simple user-defined target:

reactis_target(on && activate && (speed < 30))

This objective would be inserted at a location of the code that executes during every execution of the entry function such that on monitors whether the cruise control is turned on, activate monitors whether an attempt has been made to activate the cruise control, and speed is the speed of the vehicle. When this expression becomes true, we have found a test to test our requirement.

We can test whether the application’s response to this test is correct with the following assertion:

reactis_assert(!(active && (speed < 30)))

This assertion is inserted so that active monitors whether the cruise control is active. If this expression ever becomes false, then the requirement is violated.

The assertion and user-defined target that capture the simple requirement of the previous example depend only on the values of program variables at a given point in time. Note, however, that more complex properties involving timing can be constructed by creating Validator objectives that maintain state in between simulation steps. Consider another requirement for a cruise control:

When active, cruise control shall not permit actual, desired speeds to differ by more than 1 mph for more than 3 seconds.

This requirement can be checked by an assertion that monitors the vehicle speed, whether the cruise control is active, and the speed at which the driver has set the cruise control. A convenient way to implement such an assertion is as a C function with three input parameters active, speed, and desiredSpeed and returning a zero value if the requirement is violated.

The C function that implements the assertion is shown in Figure 10.1. It implements a simple state machine in which the static local variable state records the current state. If state SPDCHECK_ACTIVE_ERROR is ever reached, then the assertion is considered violated.

Figure 10.1: The implementation of the speed maintenance assertion.