9. Reactis Tester

Tester automatically generates test suites from C programs. Tests are formulated according to a test harness constructed by the user as described in The Reactis Harness chapter. Among other items, a harness includes an initialization function, an entry function, as set of inputs, and a set of outputs. Each test in a suite consists of a sequence of steps formulated as follows:

1. If this is the first step, call the initialization function.

2. Select a value for each input.

3. Execute the entry function based on the inputs.

4. Record the values computed for the outputs.

The test suites are intended to maximize different coverage objectives while minimizing the total number of execution steps in the overall suite. The remainder of this chapter describes the workings of Tester.

9.1. The Tester Launch Dialog

Figure 9.1 contains an annotated screenshot of the Tester launch dialog, which may be opened from the Reactis for C top-level window using the Test Suite > Create… menu item. This section describes the labeled items in the figure.

Fig. 9.1 The Reactis Tester launch dialog.

1. Specify how long Tester should run. There are three options to choose from:

   • A fixed amount of time

   • A fixed number of steps

   • A specified number of random and targeted tests/steps.

2. The Preload Files section lets you optionally specify a list of test suites to be executed before generating new test data. The coverage achieved by the preloaded tests will be tracked and subsequently generated tests will aim to exercised targets not covered by the preloaded tests. If no suites to be preloaded are given, new test data will be generated from scratch.

3. Clicking this button opens a file-selection dialog that enables you to specify an .rst file to be added to the preload list.

4. Clicking this button removes the currently selected .rst file from the list of test suites to be preloaded.

5. When running Tester for a fixed length of time, the number of hours and minutes is entered here. If 100% coverage of all targets is reached prior to the time limit, Tester will terminate early.

6. When running Tester for a fixed number of steps, the number of steps is entered here. Tester will decide how many of these steps will be random or targeted. Because of pruning, the number of steps in the final test suite will typically be less than the number entered here.

7. When running Tester for a specified number of random and targeted steps, the number of tests in the random phase is entered here. Because of pruning that occurs at the end of the random phase, some tests may be eliminated entirely, leading to a smaller number of tests at the end of the random phase than what is specified here.

8. When running Tester for a specified number of random and targeted steps, the number of steps to take while constructing each test of the random phase is entered here. Upon completion of the random phase, unimportant steps are pruned from the ends of tests, so the lengths of the final tests will usually be shorter than the length specified here.

Note

Specifying too many steps in the random phase can cause Reactis for C to run out of memory. The upper bound on the number of steps possible depends on the size of the program under test, but in general much more time should be spent in the targeted phase which is more optimized for memory usage.

9. When running Tester for a specified number of random and targeted steps, the number of execution steps to take during the targeted phase is entered here. The targeted phase uses sophisticated strategies to guide the simulation to exercise parts of the program under test not visited during the preload or random phases. The value entered specifies an upper bound on the number of simulation steps executed during the targeted phase.

10. The Coverage Objectives section enables you to specify the coverage metrics used to guide test generation. Each coverage metric defines a set of target elements in programs under test to exercise. The metrics supported by Tester are described in the Reactis Coverage Metrics chapter.

11. This entry box enables you to pass one or more of the following parameters to Tester :

    • -a1 turns inputs abstraction on, -a0 turns inputs abstraction off. Inputs abstraction usually improves the performance of Tester and should be left on (default). In rare cases, turning it off may improve coverage. If coverage problems are encountered with inputs abstraction on, it may be beneficial to take a test suite produced with abstraction on, preload it into Tester, turn abstraction off, and then run Tester again.

    • -c n sets the maximum number of input variables that may change during an execution step to n, which must be a positive integer. The default is that every input variable can change at every step. Restricting the number of input variables that can change can lead to easier-to-understand test suites.

    • -C n directs Reactis for C to use n cores during test generation. Currently supported values for n are 1 and 2 . Leveraging multi-core architectures speeds up test-generation in many cases.

    • -s randomSeed seed for the random number generator. This is useful for replaying a previous run of Tester. The random seed used to create a .rst file can be found in the test-suite log (which may be viewed in the Test Suite Browser described in The Reactis Test-Suite Browser chapter), after the -s in the Created by Tester: line.

12. The name of the .rst file to be generated.

13. Clicking this button opens a file-selection dialog for specifying the name of the .rst file to be generated.

14. Clicking this button displays Tester help.

15. Clicking this button opens a file selection dialog to specify an .rtp file from which to load Tester launch parameters. Reactis for C may be configured from the Settings dialog to generate an .rtp file for each Tester or Validator run.

16. Reset the Tester launch parameters to the default values Reactis for C ships with.

17. Scroll backward in the parameter history.

18. Scroll forward in the parameter history.

19. This button is visible only if you went back to this screen by clicking the Back button in the Tester progress dialog. Clicking it will bring back the progress dialog with its previous results.

20. Clicking this button starts test-suite generation.

21. Clicking this button closes the Tester launch dialog without starting test-generation.

9.2. The Progress Dialog

Fig. 9.2 The Reactis Tester progress dialog.

The Tester progress dialog, shown in Figure 9.2, is displayed while Tester is running. What follows describes the labeled items in the figure. When Tester completes, buttons 5–12 become enabled.

1. Status message describing current Tester activity.

2. Time elapsed (real time) since start of test generation, and estimated time until test generation completes.

3. Total number of simulation steps to be taken during test generation, and the percentage thereof taken so far. Note that since many steps are pruned from final test suite, this number will typically be much larger than the cumulative number of execution steps in the generated test suite.

4. Progress bars that report percentages of reachable targets covered for each enabled metric. Coverage metrics are described in the Reactis Coverage Metrics chapter.

5. When warnings occur, this button may be clicked to view descriptions of the warnings.

6. View help for the progress dialog.

7. Open the Reactis Test Suite Browser and load the newly generated test suite.

8. Enable Reactis Simulator and load the newly generated test suite.

9. Enable Reactis Simulator, load the newly generated test suite, and close the Tester progress dialog.

10. View the program under test, highlighted with coverage results from the newly generated test suite.

11. Go back to the Tester launch dialog.

12. View coverage results for the newly generated test suite in the Coverage-Report Browser.

13. Interrupt test generation, save tests generated so far, and close the Tester progress dialog.