Automated Testing and Validation with Reactis®

 
 Reactis User's Guide   Contents  |  Index
 Chapters:  1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | A | B | C

Chapter 12   Executing Tester-Generated Test Suites

Once created with Reactis, test suites may be executed in a variety of different ways. You may:

  1. Load them into Reactis Simulator and run them.
  2. Load them into the Simulink simulation environment of The MathWorks and run them, with or without the help of the graphical rsRunTests tool included in the Reactis distribution.
  3. Export them as CSV files for processing by custom test harnesses.

This chapter describes these options in more detail.

12.1  Executing Test Suites Using Reactis Simulator

Reactis Simulator includes facilities for running Tester-generated tests on Simulink / Stateflow models. These were detailed in Chapter 7. Here we briefly outline several of the most common use cases for executing tests in Simulator. Note that in all cases, Reactis will check the values stored in test suites for outputs and test points against those computed by the model. Also recall from Chapter 7, that Simulator offers many different ways to track data item values and coverage as tests execute.


images/simToolbarC_web.png
Figure 12.1: The Reactis Simulator toolbar.

The following uses Figure 12.1 to explain the steps needed to load and run a .rst file named cruise.rst.

  1. Select File -> Open Model... and use the resulting dialog to select cruise.slx for loading.
  2. Click button 2 to enable Simulator.
  3. Click button 13 and use the resulting dialog to select cruise.rst for loading. The name of the file will be written in the title bar of the Simulator window, and the tests will be listed in Source-of-Inputs dialog (window item 11).
  4. Once a test suite is loaded, some possible ways of running tests include:
    • To run a single test:
      1. Click on the Source-of-Inputs dialog button (window item 11).
      2. Select the test to be run. It will appear in the Source-of-Inputs dialog button.
      3. Use buttons 3–9 to run the simulator as described in Section 7.1. The inputs for each simulation step will be taken from the test and the values computed for test points and top-level outputs will be compared against those stored in the test suite.
    • To run all tests, one after another:
      1. Click on the Source-of-Inputs dialog button (window item 11).
      2. Click the All button at the bottom left of the dialog.
      3. Use buttons 3–9 to run the simulator as described in Section 7.1. The inputs for each simulation step will be drawn from the tests (beginning with Test 1) and the values computed for test points and top-level outputs will be compared against those stored in the test suite. When each test is completed, the simulation is “reset” so that the next test begins in the initial configuration. Coverage information, however, is not reset, so the final cumulative statistics show the set of targets covered by any test in the suite.
    • To run all tests and generate an HTML report:
      1. Click on the Source-of-Inputs dialog button (window item 11).
      2. Click the All button at the bottom left of the dialog.
      3. Select Simulate -> Fast Run with Report... The resulting report lists any runtime errors uncovered (divide-by-zero, overflow, memory errors, missing cases, assertion violations, etc.) or differences between outport values stored in the test suite and those computed by the model. If a runtime error is encountered, the remaining steps of the current test are skipped and Simulator continues execution with the following test. Optionally, you can also direct Reactis to include detailed coverage information (if and when each target was covered) in the report.

12.2  Executing Test Suites in Simulink .m files

Test suites generated by Reactis may also be used to guide simulations in The MathWorks’ Simulink environment. To run tests in Simulink, a test suite must first be exported into a format readable by MATLAB . Test suites may be exported from Reactis Simulator (described in more detail in Chapter 7) by selecting the Test Suite -> Export menu item. For running tests in Simulink, Reactis supports three export formats which are listed in the export dialog as follows.

Format 1
MATLAB script (.m)
Format 2
MATLAB binary data (.mat)
Format 3
MATLAB binary data for ’FromWorkspace’ blocks (.mat)

In all cases, when executed, the file initializes matrices that can be used as inputs to drive a Simulink simulation. We describe four ways to load an exported test suite into Simulink and then read input values from the tests during a simulation. The four methods are listed below along with the export file formats with which they work.

 Compatible
MethodFile Formats
Using the rsRunTests utility distributed with Reactis.1, 2
Using the Simulation Parameters Dialog.1, 2
Editing the .m files .1
Replacing top-level inports with ’FromWorkspace’ blocks3
Note: When executing Tester-generated test suites in Simulink, interpolation must be turned off for all top-level inports of the model. You may turn interpolation off for top-level ports in one of two ways:
  1. By using the interpolateoff utility distributed with Reactis. The utility is invoked from the MATLAB command line. If given a filename as an argument, interpolateoff will open the specified model file and set the inports to non-interpolate. When invoked with no argument the current model is modified. If the model does not have any top-level input ports or all top-level input ports are already set to non-interpolate, the script does nothing. To make the changes permanent, the model must be saved (to a different file if you wish to retain the old settings).
  2. By right-clicking on each port in a Simulink window, selecting block parameters, and making sure the interpolate check box is not checked.

12.2.1  Executing Tests in Simulink with the rsRunTests Utility


images/runTestSuite_web.png
Figure 12.2: The rsRunTests utility for running test suites in Simulink.

The rsRunTests tool in the Reactis distribution can be used as follows to run Reactis-generated tests from within Simulink:

  1. Ensure that the Reactis MATLAB API folder has been added to the MATLAB path by selecting the File -> Set Path... from the top-level MATLAB window. If you used the default settings during the Reactis installation then the Reactis MATLAB API directory is
    C:\Program Files\Reactis V2017\lib\api\MATLAB\reactis
  2. Start the GUI by entering rsRunTests on the MATLAB command line. You will be presented with the window shown in Figure 12.2.
  3. Use the Load Model button to specify a Simulink model to load.
  4. You may now click the Show Model button to view the loaded model.
  5. Use the Load Test File button to specify a .m files or a .mat files (exported by Reactis) to load.
  6. Select a test in the drop-down box.
  7. Click the Run Test(s) button to execute the selected test.
  8. Click the Save Log button to save a copy of the contents of the Test execution log to a user-selected text file.
  9. Any significant differences between the model’s outputs as computed by Reactis (and stored in the test suite) and those computed by Simulink will be listed in the log. A difference between outputs is listed if it exceeds the tolerance specified for the output . The Tolerance pull-down may be used to manipulate tolerances. The menu includes a Default entry as well as an entry for each top-level outport. If the setting for an outport is Inherit the Default tolerance is used.

    The initial values of the Tolerance pull-down are imported from the test suite (which was exported by Reactis). You may manipulate both the default and port-level tolerances for a model from the Reactis Info File Editor and Reactis will store these values in the .rsi file associated with the model. Reactis will then export the tolerances from the .rsi file to exported suites (.m or .mat files) for use by rsRunTests.

    The process used to flag differences is described in Section 5.6.

12.2.2  Executing Tests in Simulink with the Simulation Parameters Dialog

An alternative to the rsRunTests utility is running the tests manually within MATLAB as follows.

  1. Load the .m file (or .mat file) by entering the file name without the .m (.mat) file suffix on the MATLAB command line. 1 For example:
          >> cruiseTests
    
  2. Load the model in Simulink and select the Simulation -> Configuration Parameters... menu item.
  3. In the Solver panel of the Configuration Parameters dialog, adjust the simulation time to use the entire test. By convention, Reactis stores the start and stop times for test n, where n is the number of the test (1, 2, 3, etc.), in variables named testn_start and testn_stop, respectively.
  4. In the Data Import/Export panel of the Configuration Parameters dialog, enable the Input check box and enter the name of the test matrix in the text box next to it. The test matrices in a Reactis-exported .m file (.mat file) are by convention named test1, test2, test3, ... For example, enter test1 in the text box if you want to execute the first test.
  5. Start a simulation run in Simulink. The simulation inputs will be taken from the specified test.

12.2.3  Running Tests in Simulink by Editing the .m files

This method also allows tests to be run from within MATLAB . The difference with the previous method is that it allows the “batch processing” of multiple tests.

  1. Open the .m file in the MATLAB editor (by selecting File -> Open... from the top-level MATLAB window).
  2. In line 2 of the .m file make sure modelName is assigned the name of the model you would like to simulate.
  3. Scroll to the bottom of the file and you will find a number of commented-out lines, one for each test. Uncomment the lines of tests you want to execute and save the file.
  4. Load the file by entering its name on the MATLAB command line, and all uncommented tests will be executed.

12.2.4  Running Tests in Simulink by Replacing Top-Level Inports with FromWorkspace Blocks.

Executing tests exported in format 3 requires some modifications to be made to a model before the tests may be run in Simulink. For each test in a test suite, the exported file contains a variable named testn (where n is the test number). The variable stores an array of structures ready for use in a FromWorkspace block. Each element of the array stores the values that one inport assumes during the test. A model may be modified to use these structures to execute test n as follows:

  1. Load the model in Simulink.
  2. For each top-level inport,
    1. Delete the inport and replace it with a FromWorkspace block.
    2. Double-click on the new FromWorkspace block.
    3. In the Data field, enter testn(i) (where n is the test number and i is the inport number. For example test3(5) specifies test 3, inport 5).
    4. Uncheck the Interpolate field.
    5. Select Holding final value in the Form output after final value by field.
  3. In the Solver tab of the Simulation Parameters dialog, adjust the simulation time to use the entire test. By convention, Reactis stores the start and stop times for test n, where n is the test number, in variables named testn_start and testn_stop, respectively.
  4. Start a simulation run in Simulink. The simulation inputs will be taken from the specified test.

12.3  Executing Test Suites on Source Code

Tester-generated test suites may also be run on source-code implementations of Simulink / Stateflow models; this feature helps automate the important task of determining whether deployed code will conform to models. Reactis currently supports several methods for running tests on source.

Via Reactis Simulator
You may use the S-function feature of MathWorks’ Simulink environment to embed your source code inside a Simulink block. This wrapper model may then be loaded and run in Simulator with differences between S-function outputs and outputs stored in tests flagged. This is especially useful when using the Reactis for C Plugin, since it allows inspection of the C code as the tests execute and tracking the coverage of the C code.
Via Simulink:
You may use the S-function feature of MathWorks’ Simulink environment to embed your source code inside a Simulink block. Then, as described in Section 12.2, a number of methods for executing tests are available, including the rsRunTests utility provided by Reactis which will execute the tests and automatically flag any differences between the values stored in the tests and those computed by Simulink.
Via user-written test harnesses:
Reactis is also capable of outputting a test suite as a comma separated value (CSV) file for easy processing by external tools. Test steps are listed in an easy-to-process fashion. You may write your own test harnesses to read these files, run your program on the test data, and compare the outputs of the program to those stored in the tests.

Both the .m files and CSV files may be obtained using the Reactis Simulator export facility; see Section 7.7.1 for more on how to do this.

In order to write test harnesses that process Reactis-exported CSV files, one must know about the format of these files. Figure 12.3 contains a partial listing of a .csv file generated from the cruise-control example used in Chapter 3. See Section 7.7.1 for a description of the CSV file format.


onOff,accelResume,cancel,decelSet,brake,gas,inacThrDelt,drag,___t___,active,thrDelt,speed,"|InitialSpeed"
0.0,1.0,1.0,1.0,0.0,1.0,-0.1,-0.01,0.0,0.0,0.0,0.0,0.0
1.0,1.0,0.0,1.0,1.0,1.0,0.0,-0.00959914797712953,1.0,0.0,0.0,0.0,0.0
0.0,1.0,0.0,0.0,1.0,0.0,0.1,-0.00866942376836091,2.0,0.0,0.0,0.0,0.0
1.0,1.0,1.0,1.0,1.0,0.0,0.1,-0.00829318652470756,3.0,0.0,0.0,0.191330576231639,0.0

1.0,1.0,1.0,0.0,0.0,1.0,0.1,0.01,0.0,0.0,0.0,50.0,50.0
0.0,1.0,0.0,1.0,1.0,0.0,0.0,0.00942639853457365,1.0,0.0,0.0,50.21,50.0
0.0,0.0,0.0,1.0,1.0,1.0,-0.1,0.00884905410912495,2.0,0.0,0.0,50.2194263985346,50.0

0.0,0.0,1.0,1.0,0.0,1.0,0.1,-0.01,0.0,0.0,0.0,4.21309604376837,4.21309604376837
0.0,1.0,1.0,1.0,0.0,1.0,-0.1,-0.00955227172719837,1.0,0.0,0.0,4.40309604376837,4.21309604376837
1.0,1.0,1.0,0.0,0.0,0.0,0.1,-0.00866896857231075,2.0,0.0,0.0,4.19354377204118,4.21309604376837
0.0,1.0,1.0,1.0,0.0,0.0,0.1,-0.00855659477502143,3.0,0.0,0.0,4.38487480346887,4.21309604376837
1.0,0.0,1.0,0.0,0.0,0.0,-0.1,-0.00773304191244197,4.0,0.0,0.0,4.57631820869384,4.21309604376837
0.0,0.0,1.0,1.0,0.0,1.0,0.1,-0.00700061209826044,5.0,0.0,0.0,4.3685851667814,4.21309604376837

0.0,1.0,0.0,1.0,0.0,1.0,0.1,0.0,0.0,0.0,0.0,22.8683123376438,22.8683123376438
Figure 12.3: A sample fragment of an exported test suite (CSV format).


1
Note that if an .m files and an .mdl files have the same base name, then entering that base name on the MATLAB command line causes the .slx file to be loaded.