2. Reactis Tester

Reactis Tester dramatically reduces the time and effort required to generate comprehensive test suites.

As shown in Figure 2.1, Reactis Tester automatically generates tests for C programs. The test suites generated by Tester provide comprehensive coverage of a variety of test coverage metrics, including the Modified Condition/Decision Coverage (MC/DC) test coverage mandated by the US Federal Aviation Administration (FAA) in its DO-178/B guidelines and by the ISO 26262 automotive safety standard. In addition to maximizing coverage, Tester-generated test suites are automatically minimized to eliminate redundancy, saving testing time. Each test case in a test suite consists of a sequence of program inputs and a sequence of program outputs generated in response to those inputs.

Fig. 2.1 Reactis Tester automatically generates comprehensive yet compact test suites.

The automatically-generated test data may then be used for a variety of purposes, including the following:

Finding runtime errors

The tests help uncover runtime errors in C programs, including memory errors, overflows, and divide-by-zero errors.

Regression testing

When a program is modified, tests generated from the older version may be run on the newer version to understand the impact of the changes on program behavior.

Reactis Tester enables engineers to maximize the effectiveness of testing while reducing the time actually spent on testing.

Fig. 2.2 Reactis Tester automatically generates comprehensive yet compact test suites.

The embedded software applications that Reactis for C initially targets are reactive control systems, which typically operate as depicted in Figure 2.2. Such systems use a control loop to interact with their external environment, which consists of the following steps:

1. Read one or more input values from environment.

2. Perform internal calculations based on the input values and the internal system state.

3. Update outputs to control the environment.

4. Repeat.

To support this framework, Reactis for C lets you specify a test harness consisting of an entry function, a set of inputs, and a set of outputs. The entry function is a C function which acts as interface between the application and the test environment. Each argument of the entry function acts as either an input or an output. In addition, global variables can be selected for use as inputs or outputs. When testing a program, Reactis for C repeatedly performs the following steps:

1. Calculate a set of inputs for the harness. Record the values of all inputs.

2. Call the entry function with the selected inputs.

3. When the entry function returns, record the values of all outputs.

Fig. 2.3 Structure of a Reactis for C-generated test.

Figure 2.3 shows the structure of a Tester-generated test. Tests are comprised of simulation steps. Each step is a single call to the program entry function, including the inputs (arguments passed to the entry function and global variables) and outputs (value returned by the entry function, final values of any arguments passed by reference and global variables). A test can be represented by a matrix in which each row contains the sequence of values for a specific input or output and each column contains all the input and output values for a single step. A test suite consists of a set of tests, as shown in Figure 2.3. When a test suite is executed the program is reset to its initial state before each test begins.

Fig. 2.4 Structure of a Reactis for C-generated test suite.

Test suites are constructed by executing a program while recording the input and output values at each step. Outputs are computed by the program at each step, but there are several ways in which the input values that drive the testing can be chosen. Input data could be manually constructed or recorded during field testing, but these are expensive tasks. Alternatively, input data can be randomly generated. Unfortunately the latter approach produces tests with poor coverage.

Reactis Tester tracks several different classes of coverage targets. A coverage target is a program element that should be exercised during testing. A coverage criterion or coverage metric defines how to calculate a set of coverage targets from the syntax of a program. For example, the Statement Coverage metric identifies all statements in a program as coverage targets, i.e. every statement should be executed to satisfy Statement Coverage. The coverage criteria tracked by Tester include Statement, Decision, Condition, and MC/DC coverage. Decisions are boolean expressions which determine the path of program execution. Conditions are atomic predicates contained within decisions. Modified Condition/Decision Coverage (MC/DC) targets are used to ensure that each condition independently effects the outcome of the enclosing decision.

Reactis Tester employs a novel, patented approach called guided simulation to generate quality input data automatically. Guided simulation uses algorithms and heuristics to automatically select inputs which exercise coverage targets that have not yet been covered during each simulation step.