Automated Testing and Validation with Reactis®

 
 Reactis User's Guide   Contents  |  Index
 Chapters:  1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | A | B | C

Chapter 3  Getting Started with Reactis

This chapter provides a quick overview of Reactis. It contains a brief description of each main component of the tool suite in the form of an extended “guided-tour” of the different features of the tool suite. Each “stop” in the tour is indicated by a §. The tour uses as a running example a simple Simulink / Stateflow model of an automobile cruise control that is included in the Reactis distribution.

3.1  A Note on Model Preparation

The cruise-control example included with the distribution does not require any special processing before you run Reactis on it; you may load the file immediately and start the guided tour. However, there is an important step that you should undertake when you are preparing models for use with Reactis. This section describes this preparatory step and discusses the Simulink operations needed to perform it.

Reactis supports a large portion of the discrete-time subset of Simulink and Stateflow. As it processes a Simulink / Stateflow model, it also interacts with MATLAB in order to evaluate MATLAB expressions, including references to workspace data items, that a model may contain.

In order for Reactis to process a model, Reactis must be able to automatically set up the MATLAB workspace data used by the model. For this reason, any workspace data items that a model uses must be initialized within one of the following locations (for more details see Section 14.1.1):

  • Any Simulink model callback or block callback that is executed when loading or running the model (PreLoadFcn, PostLoadFcn, InitFcn, StartFcn).
  • A “startup.m” file located in the folder where the model file is located.
  • The Callbacks pane of the Reactis Info File Editor.

Below we describe how the workspace initialization was established in the cruise-control model file included in the Reactis release. In that example the MATLAB file cruise_constants.m defines two workspace variables that are used in the Simulink / Stateflow model file cruise.slx. The cruise_constants.m file was attached to cruise.slx using the following steps.

  1. Load cruise.slx into Simulink.
  2. In the Simulink window, select the File -> Model Properties -> Model Properties menu item1.
  3. In the resulting dialog, select the Callbacks tab.
  4. In the PreLoadFcn* section, enter cruise_constants; (note that the .m suffix is not included).
  5. Save the model.

This saved cruise.slx file is distributed with Reactis, so you do not need to undertake the above steps yourself in order to load and process this file in Reactis. The above steps are cited only for illustrative purposes.

3.2  Reactis Top Level


images/reactisTop_web.png
Figure 3.1: The Reactis window.

The Reactis top-level window contains menus and a tool bar for launching and controlling verification and validation activities. Reactis is invoked as follows.

§1 Select Start -> All Programs -> Reactis V2017 -> Reactis from the Windows Start menu (or double-click on the desktop shortcut if you installed one).


You now see a Reactis window like the one as shown in Figure 3.1. A model may be selected for analysis as follows.

§2 Click the  images/openBtn_web.png    (file open) button located at the leftmost position on the tool bar. Then use the resulting file-selection dialog to choose the file cruise.slx from the examples folder of the Reactis distribution.2


Loading the model causes the top-level window to change as shown in Figure 3.2. The panel to the right shows the top-level Simulink diagram of the model; the panel to the left shows the model hierarchy. In addition, the title bar now reports the model currently loaded, namely cruise.slx, and the Reactis info file (.rsi files ) cruise.rsi that contains auxiliary model information used by Reactis. Info files are explained in more detail in the next section (Section 3.3).

It is also worth noting that if during installation you choose to associate the .rsi file extension with Reactis, then you can start Reactis and open cruise.slx in a single step by opening cruise.rsi in Windows Explorer.


images/modelViewerWin_web.png
Figure 3.2: Reactis after loading cruise.slx.

A node in the hierarchy panel may be tagged with a plus sign (+) to indicate the node contains subsystems not currently displayed. The hierarchy may be expanded or collapsed by clicking the + or - tags. Subsystems in the hierarchy panel are also tagged with icons indicating whether they are Simulink (SL), Stateflow ( SF), Validator objectives (  images/assertionIconBtn_web.png  ,  images/targetIconBtn_web.png  ), virtual sources (  images/virtualSourceIconBtn_web.png  ), or configuration variables (  images/cfgvarIconBtn_web.png  ). Validator objectives, virtual sources and configuration variables are discussed in more detail later. Additionally, if you are using the Reactis for C Plugin (see Chapter 16 for details), then you may also see C source files (C), C libraries (LIB), and S-Functions (FN) in the hierarchy panel.

Reactis provides a signal-tracing mechanism which allows the path taken by a signal to be quickly identified. To trace a signal, left-click on any part of the signal line.



§3 In the model hierarchy panel, left-click on the CruiseMDL subsystem, and then, in the main panel, left-click on the on the signal line emerging from the decelSet inport (inport 4). The result should be similar to Figure 3.3. Click the  images/viewParentBtn_web.png  button twice to see how to trace the signal through different levels of the model hierarchy.


As shown in Figure 3.3, signals are highlighted in yellow when left-clicked on. The route of the highlighted signal can then be easily identified. To turn off the highlighting, left-click on empty space in the main window.


images/sighighlight_web.png
Figure 3.3: Highlighting a signal.

3.3  The Info File Editor

Reactis does not modify the .slx files for a model. Instead the tool stores model-specific information that it requires in a .rsi file. The primary way for you to view and edit the data in these files is via the Reactis Info File Editor, which is described briefly in this section and in more detail in Chapter 5.

The next stop in the guided tour explains how this editor may be invoked on cruise.rsi.

§4 Select the Edit -> Inport Types... menu item.


This starts the Reactis Info File Editor, as shown in Figure 3.4. Note that the contents of the .rsi file may only be modified when Simulator is disabled. When Simulator is running, the Info File Editor operates in read-only mode as indicated by “[read only]” in the editor window’s title bar.


images/infoViewerWin_web.png
Figure 3.4: The Reactis Info File Editor.

.rsi files contain directives partitioned among the following panes in the Info File Editor.

Inport Types.
A type for each top-level inport is used by Reactis to constrain the set of values fed into top-level inports during simulation, test-case generation, and model validation. For example, you can specify an inport range by giving minimum and maximum values.
Configuration Variables.
Configuration variables are user-selected workspace data items which can change in between tests in a test suite (but not during a test).
Test points.
Test points are internal signals which have been selected (by the user) to be logged in test suites.
Outport Tolerances.
Each outport of a model may be assigned a tolerance. When executing a test suite on a model in Reactis Simulator, the tolerance specifies the maximum acceptable difference between the value computed by the model for the outport and the value stored in a test suite for the outport.
General.
Various settings that determine how a model executes in Reactis.
Error Checking.
Specify how Reactis should respond to various types of errors (e.g. overflow, NaN, etc.).
Coverage Metrics.
Specify which coverage metrics will be tracked for the model and let you adjust the configurable metrics (CSEPT, MC/DC, MCC, Boundary).
Excluded Coverage Targets.
Specify targets to be ignored when measuring the coverage of a model.
Validator Objectives.
Validator objectives (including assertions, user-defined targets, and virtual sources) are instrumentation that you may provide to check if a model meets its requirements. You can double-click on an item in this list to highlight the location of the objective in the model.
C Code.
If the Reactis for C Plugin is installed, this tab will contain information about C-coded S-functions, C code embedded within Stateflow charts and C library code called from either an S-function or Stateflow.
Callbacks.
These are fragments of MATLAB code that Reactis will run before or after loading a model in Simulink. Note, that these callbacks are distinct from those maintained by Simulink.
Search Path.
The model-specific search path is prepended to the global search path to specify the list of folders (for the current model) in which Reactis will search for files such as Simulink model libraries (.slx), MATLAB scripts (.m), and S-Functions (.dll, .mexw32, .m).
File Dependencies.
A list of files on which the current model depends. The file-dependency information enables Reactis to track changes in auxiliary model files so that information obtained from them for the purposes of processing the current model may be kept up to date. Typically, files listed here would include any .m files loaded as a result of executing the current model’s pre-load function. For example, a .m file mentioned in the pre-load function might itself load another .m file; this second .m file (along with the first) should be listed as a dependency in order to ensure that Reactis behaves properly should this .m file change. Dependencies on libraries (.slx files) are detected automatically and need not be listed here.



§5 To modify the type for an inport, select the appropriate row, then right-click on it and select Edit..., and make the desired change in the resulting dialog. Save the change and close the type editor by clicking Ok or close without saving by clicking Cancel.


The types that may be specified are the base Simulink / Stateflow types extended with notations to define ranges, subsets and resolutions, and to constrain the allowable changes in value from one simulation step to the next. More precisely, acceptable types can have the following basic forms.

Complete range of base type:
By default the type associated with an inport is the Simulink / Stateflow base type inferred from the model. Allowed base types include: int8, int16, int32, uint8, uint16, uint32, boolean, single, double, sfix*, and ufix*.
Subrange of base type:
t[i, j], where t is a Simulink / Stateflow base type, and i and j are elements of type t, with i being a lower bound and j an upper bound.
Subrange with resolution type:
t [i:j:k], where t is either double or single, i is a lower bound, j is a resolution, and k is an upper bound; all of i, j and k must be of type t. The allowed values that an inport of this type may have are of form i + n × j, where n is a positive integer such that i + n × jk. In other words, each value that an inport of this type may assume must fall between i and k, inclusive, and differ from i by some integer multiple of j.
Set of specific values:
t{ e1, … en }, where t is a base type and e1, … en are either elements of type t, or expressions of form v:w, where v is an element of type t and w, a positive integer, is a probability weight. These weights may be used to influence the relative likelihood that Reactis will select a particular value when it needs to select a value randomly to assign to an inport having an subset type. If the probability weight is omitted it is assumed to be “1”. For example, an inport having type uint{0:1, 1:3} would get the value 0 in 25% of random simulation steps and the value 1 in 75% of random simulation steps, on average.
Delta type:
t delta [i,j], where t is either a base type, a range type, or a resolution type, and i and j are elements of the underlying base type of t. Delta types allow bounds to be placed on the changes in value that variables may undergo from one simulation step to the next. The value i specifies a lower bound, and j specifies an upper bound, on the size of this change between any two simulation steps. More precisely, if a variable has this type, and v and v′ are values assumed by this variable in successive simulation steps, with v′ the “later” value, then the following mathematical relationship holds: iv′ − vj. Note that if i is negative then values are allowed to decrease as simulation progresses.

Table 3.1 gives examples of types and the values they contain. For vector, matrix or bus inports, the above types can be specified for each element independently.


RSI TypeValues in Type
double [0.0,4.0]All double-precision floating-point numbers between 0.0 and 4.0, inclusive.
int16 [-1,1]-1, 0, 1
int32 {0,4,7}0, 4, 7
uint8 {0:1,1:3}0, 1
double [0.0:0.5:2.0]0.0, 0.5, 1.0, 1.5, 2.0
int16 [0,3] delta [-1, 1]0, 1, 2, 3; inports of this type can increase or decrease by at most 1 in successive simulation steps.
Table 3.1: Type examples.

If any change is made to a port type, then “[modified]” appears in the title bars of the Info File Editor and the top-level Reactis window. You may save changes to disk by selecting File -> Save from the Info File Editor or File -> Save Info File from the top-level window.

If no .rsi file exists for a model, Reactis will create a default file the first time you open the Info File Editor, or start Simulator or Tester. The default type for each inport is the base type inferred for the port from the model. If you add or remove an inport to your model you can synchronize your .rsi file with the new version of the model by selecting Tools -> Synchronize from the Info File Editor.



§6 Select the File -> Exit menu item on the editor’s tool bar to close the Reactis Info File Editor.


3.4  Simulator

Reactis Simulator provides a rich array of facilities for viewing the execution of models. To continue with the guided tour:

§7 Click the  images/simOnBtn_web.png  (enable Simulator) button in the tool bar to start Reactis Simulator.


This causes a number of the tool-bar buttons that were previously disabled to become enabled.

Simulator performs simulations in a step-by-step manner: at each simulation step inputs are generated for each top-level inport, and resultant outputs reported on each outport. You can control how Simulator computes top-level inport values using two different mechanisms:

  • Virtual sources enable you to specify input sequences for one or more top-level inports (for example, using a Signal Builder block).
  • The Source-of-Inputs dialog lets you control the inputs not controlled by a virtual source as follows. Simulator may be directed to:
    1. generate values randomly (this is the default),
    2. query the user,
    3. read inputs from a Reactis test suite. Such tests may have been generated automatically by Reactis Tester, constructed manually in Reactis Simulator, or imported from a file storing test data in a comma-separated-value format.

To set the input source for an inport not controlled by a virtual source, use the Source-of-Inputs dialog located to the left of  images/viewerBtn_web.png  in the tool bar (see Figure 3.5). The next part of the guided tour illustrates the use of each of these input sources; interspersed with this discussion are asides on coverage tracking, data-value tracking, and other useful features.


images/inputsSource3_web.png
Figure 3.5: The Source-of-Inputs dialog allows you to control how Reactis Simulator generates values for top-level inports (not controlled by a virtual source) when executing a model.

3.4.1  Generating Random Inputs

As the random input source is the default, no action needs to be taken to set this input mode.

§8 In the model hierarchy panel on the left of the Simulator window, click on CruiseMain/CruiseMDL/Mode in order to display the Stateflow diagram in the main window. Then, click the  images/playBtn_web.png  (Run/Pause Simulation) button.


During simulation, blocks, signals, states, and transitions in the diagram are highlighted in green as they are entered and executed. The simulation stops automatically when the number of simulation steps reaches the figure contained in the entry box to the left of the Source-of-Inputs dialog. Before then, you may pause the simulation by clicking the  images/playBtn_web.png  (Run/Pause Simulation) button a second time. Note, that simulation will likely pause in the middle of a simulation step. You may then click:

  •  images/miniStepBtn_web.png  to continue executing a block at a time, or
  •  images/stepBtn_web.png  to complete the step, or
  •  images/backBtn_web.png  to back up to the start of the step

3.4.2  Tracking Model Coverage

While Simulator is running you may also track coverage information regarding which parts of your model have been executed and which have not. These coverage-tracking features work for all input-source modes. The next portion of the guided tour illustrates how these features are used.

§9 Make sure menu item Coverage -> Show Details is selected (as indicated by a check mark in the menu). Select menu item Coverage -> Show Summary.



images/simWin2_web.png
Figure 3.6: The Reactis coverage tracking features convey which parts of a model have been exercised. Note that the coverage information you see at this point will be different than that shown above. This is because a different sequence of inputs (from the random ones you used) brought the simulation to the displayed state.

A dialog summarizing coverage (which blocks/states have been entered and which transitions have fired) now appears, with elements of the diagram not yet exercised drawn in red, as shown in Figure 3.6. Note that poor coverage is not uncommon with random simulation. You may hover over a covered element to determine the (1) test (in the case being considered here, the “test” is the current simulation run and is rendered as . in the hovering information) and (2) step within the test that first exercised the item. This type of test and step coverage information is displayed with a message of the form test/step. You may view detailed information for the Condition, Decision and MC/DC   coverage metrics, which involve Simulink logic blocks and Stateflow transition segments whose label includes an event and/or condition, as follows.



§10 Perform the following.

  1. In the model hierarchy panel on the left of the Reactis window, click on CruiseMain/CruiseMdl/Mode to display the Stateflow diagram in the main panel. Note, this was done in a previous step, so it may not be necessary here.
  2. Right-click on the transition from state Init to state Active, and select View Coverage Details. A Coverage Details dialog will appear.
  3. In the Coverage Details dialog, select the MCC tab to display and inspect information related to Multiple Condition Coverage.
  4. Right-click on the bottom cell in the Coverage column of the MCC table and then left-click on Track Coverage in the pop-up menu. An Exclude Target dialog will appear.
  5. In the Exclude Target dialog, select Exclude this target from coverage and click on Ok to close the dialog. The status of the MCC target will change to excluded.
  6. Right-click on the bottom cell in the Coverage column of the MCC table and open the Exclude Target dialog a second time. Select Track coverage for this target and click on Ok to close the dialog. The status of the MCC target will change back to its original value.




images/mcdcTable_web.png
Figure 3.7: The Coverage Details dialog includes two tabs: Decision and MCC. The Decision tab includes details of decision coverage, condition coverage, and MC/DC. The MCC tab shows multiple condition coverage.

The Coverage Details dialog, shown in Figure 3.7, has two tabs: Decision and MCC. The Decision tab displays details for decision coverage, condition coverage, and MC/DC. The table in this figure gives information for the decision:

set == 1 && deactivate == 0

This decision contains two conditions:

  • set == 1
  • deactivate == 0

Conditions are the atomic boolean expressions that are used in decisions. The first two columns of the table list the test/step information for when the decision first evaluated to true and when it first evaluated to false. A value -/- indicates that the relevant situation has not yet been encountered. The third column lists the conditions that make up the decision, while the forth and fifth columns give test/step information for when each condition was evaluated to true and to false.

MC/DC Coverage requires that each condition independently affect the outcome of the decision in which it resides. When a condition has met the MC/DC criterion in a set of tests, the sixth and seventh columns of the table explain how. Each element of these two columns has the form bb:test/step, where each b reports the outcome of evaluating one of the conditions in the decision during the test and step specified. Each b is either T to indicate the condition evaluated to true, F to indicate the condition evaluated to false, or x to mean the condition was not evaluated due to short circuiting.

The MCC tab of the Coverage Details dialog displays details of multiple condition coverage (MCC) which tracks whether all possible combinations of condition outcomes for a decision have been exercised. The table includes a column for each condition in the decision. The column header is the condition and each subsequent row contains an outcome for the condition: True, False, or x (indicating the condition was not evaluated due to short-circuiting). Each row also contains the outcome of the decision (True of False) and, when covered, the test and step during which the combination was first exercised.

It should be noted that the previous scenario relied on randomly generated input data, and replaying the steps outlined above will yield different coverage information than that depicted in Figure 3.7

An alternative way to query coverage information is to invoke the Coverage-Report Browser by selecting Coverage -> Show Report. This is a tool for viewing or exporting coverage information that explains which model elements have been covered along with the test and step where they were first exercised.

A simulation run, and associated coverage statistics, may be reset by clicking the Reset to First Step button (  images/resetBtn_web.png  ) in the tool bar. Note that in the Coverage Summary window, not all percentages will be zero, since some items are covered in the model’s initial configuration.

3.4.3  Reading Inputs from Tests

Simulation inputs may also be drawn from tests in a Reactis test suite. Such a test suite may be generated automatically by Reactis Tester, constructed manually in Reactis Simulator, or imported from a file storing test-data in a comma-separated-value format. By convention, files storing Reactis test suites have a .rst file-name extension. A Reactis test suite may be loaded into Simulator as follows.

§11 Click the  images/openBtn_web.png  button in the tool bar to the right of the Source-of-Inputs dialog and use the file-selection dialog to select cruise.rst in the examples folder of the Reactis distribution.


This causes cruise.rst, the name of a test suite file generated by Reactis Tester, to appear in the title bar and the contents of the Source-of-Inputs dialog to change; it now contains a list of tests that have been loaded. To view this list:

§12 Click on the Source-of-Inputs dialog (located to the left of  images/viewerBtn_web.png  in the tool bar).


Each test in the suite has a row in the dialog that contains a test number, a sequence number, a name, and the number of steps in the test. Clicking the “all” button in the lower left corner specifies that all tests in the suite should be executed one after another. To execute the longest test in the suite:

§13 Do the following.

  1. Select the longest test in the test suite in the dialog.
  2. Click the  images/fastPlayBtn_web.png  (Run Fast Simulation) button.



If you look at the bottom-right corner of the window, you can see that the test is being executed (or has completed), although the results of each execution step are not displayed graphically. When the test execution completes, the exercised parts of the model are drawn in black. If the Run Simulation button (  images/playBtn_web.png  ) is clicked instead, then the results of each simulation step are rendered graphically, with the consequence that simulation proceeds more slowly.

You may specify that a subset of tests should be run by holding down the control key and clicking on each test to be run with the left mouse button. The tests will be run in the order they are selected. As tests are selected the sequence number column is updated to indicate the execution order of the tests.

You may also use the Source-of-Inputs dialog to change the name of a test. To do so, select the test by single clicking on it, then click on the name and, when the cursor appears, type in a new name and press return.

Whenever tests are executed in Simulator, the value computed by the model for each top-level outport and test point is compared against the corresponding value stored in the test suite. The tolerance for this comparison can be configured in the Info File Editor. An HTML report listing any differences (as well as any runtime errors encountered) can be generated by loading a test suite in Simulator and then selecting Simulate -> Fast Run With Report.

3.4.4  Tracking Values of Data Items

When Simulator is paused3, you may view the current value of a given data item (Simulink block or signal line, Stateflow variable, or C variable) by hovering over the item with your mouse cursor. You may also select data items whose values you wish to track during simulation using the watched-variable and scope facilities of Simulator.

§14 In the model hierarchy panel on the left of the Reactis window, select the top-level of the model (“cruise”) for display. Then:

  1. Right-click on the outport “active” and select Add to Watched from the resulting pop-up menu.
  2. Hover over outport “active” with the mouse; a pop-up displays its current value.



The bottom of the Simulator window now changes to that indicated in Figure 3.8.


images/watched_web.png
Figure 3.8: The watched-variable panel in Simulator displays the values of data items, as does hovering over a data item with the mouse.

The watched-variable panel shows the values of watched data items in the current simulation step, as does hovering over a data item with the mouse. Variables may be added to, and deleted from, the watched-variable panel by selecting them and right-clicking to obtain a menu. You may also toggle whether the watched-variable list is displayed or not by selecting the View -> Show Watched Variables menu item.

Scopes display the values a given data item has assumed since the beginning of the current simulation run. To open a scope:

§15 Perform the following.

  1. In the model hierarchy panel, select CruiseMain/CruiseMdl.
  2. Right-click on inport “speed” and select Open Scope from the resulting pop-up menu. A new window like the one in Figure 3.9 (a) appears and plots the speed of the car during the simulation run.
  3. Right-click on the “dSpeed” signal that exits the “DesiredSpeed” subsystem and select Add to Scope -> cruise.CruiseMain.CruiseMDL.CruiseMDL|speed. This causes the desired speed to be plotted against the actual speed on the previously opened scope, so that the scope now appears similar to Figure 3.9 (b).




images/scopesAandB_web.png
Figure 3.9: Two scope windows plotting actual speed (yellow) and desired speed (green). Note that the plots you see will be different than that shown above. This is because a different sequence of inputs brought the simulation to the displayed state.

Scopes also have a zoom feature which is particularly useful for viewing the details of long signals. The zoom feature uses the mouse to select an initial region of interest, which can be subsequently repositioned using the mouse. See Section 7.5.3 for more details on scopes.



§16 Perform the following.

  1. Select a region of interest within the scope by left-clicking and dragging while holding the mouse button down. Release the button. This zooms the scope in to view the selected area.
  2. Select a second region of interest within the scope by left-clicking and dragging. This zooms the scope in a second time.
  3. Hold down the control (Ctrl) key and left-click within the scope. The cursor will change to a plus sign (+). Move the mouse while holding down the mouse button. This will shift the region displayed within the scope. Release the mouse button.
  4. Right-click on the scope. This zooms the scope out so that the region selected during step 1 is now displayed.
  5. Right-click on the scope. This zooms the scope out to view the area selected before step 1 was performed. In this case, the entire signal will be visible.





§17 Close the watched-variable panel by selecting the View -> Show Watched Variables menu item.



images/guidedSim_web.png
Figure 3.10: The Next Input Values dialog enables you to hand-select input data.

3.4.5  Querying the User for Inputs

The third way for Simulator to obtain values for inports is for you to provide them. To enter this mode of operation:

§18 Select User Guided Simulation from the Source-of-Inputs dialog.


Upon selecting this input mode, a Next Input Values dialog appears, as shown in Figure 3.10, that allows you to specify the input values for the next simulation step. Each top-level inport of the model has a row in the dialog containing five columns; these determine the next input value for the corresponding inport as follows.

  1. The inport number and name.
  2. This pull-down menu has three entries that determine how the next value for the port is specified:
    Random
    Randomly select the next value for the inport from the type given for the inport in the .rsi file.
    Entry
    Specify the next value with the text-entry box in column three of the panel.
    Panel
    Open a sub-panel to specify the next value. This input mode is primarily used when the port reads vector values.
  3. If the pull-down menu in column two is set to “Entry”, then the next input value is taken from this text-entry box.
  4. If the pull-down menu in column two is set to “Entry”, then clicking the history button (labeled H) displays recent values the inport has assumed. Selecting a value from the list causes it to be placed in the text-entry box of column three.
  5. The arrow buttons in this column enable you to scroll through the possible values for the port. The arrows are not available for ports of type double or single or ranges with a base type of double or single.

When Run Simulation (  images/playBtn_web.png  ) or Run Fast Simulation (  images/fastPlayBtn_web.png  ) is selected, the inport values specified are used for each subsequent simulation step until the simulation is paused.

3.4.6  Controlling Inputs with Virtual Sources

Previous sections described three ways to control the model inputs during simulation: random generation, reading values from tests, and querying the user. Virtual sources offer an alternative mechanism to control one or more inputs with a Simulink subsystem (outputs of the virtual source feed into inputs of the model). Signal Builder blocks are especially convenient for implementing virtual sources. Virtual sources are created using Simulink and saved in a separate library. They are then wired into your model within Reactis, leaving your model unmodified by the testing instrumentation.



§19 Perform the following:

  1. Turn off Simulator by clicking the  images/simOffBtn_web.png  button in the toolbar.
  2. Select menu item File -> Select Info File...
  3. In the resulting dialog, select file cruise_vs.rsi
  4. After dismissing the dialog, display the top level of the model and hover with the mouse over the ActiveCheckScenario virtual source (on the left side of the panel).



After selecting the new .rsi file, two new items will appear at the top level of the model (to the left): a virtual source ActiveCheckScenario and an assertion ActiveCheck as shown in Figure 3.11. The virtual source is implemented with a Signal Builder block that specifies sequences of values for the first six inputs of the model and a sequence of expected responses to this scenario for the active output of the cruise control. The assertion then compares the expected value of output to the actual value computed by the model.


images/VS1_web.png
Figure 3.11: Virtual sources offer a mechanism to control one or more model inports. Any Simulink / Stateflow construct supported by Reactis may be used to implement the virtual source. Signal Builder blocks are especially useful. A Validator assertion can monitor an output of the virtual source, relieving the user of the tedious task of checking expected values against actual responses of the model.

The following steps show how to enable and disable virtual sources:

§20

  1. Right-click on ActiveCheckScenario and select Disable. This causes the virtual source and the virtual source symbol on inports to be shaded in gray to indicate that the virtual source is disabled and will no longer control the inports.
  2. Right-click on ActiveCheckScenario and select Enable to turn the virtual source back on.
  3. Right-click on inport brake and select Disable control by virtual source. This means brake will no longer be controlled by the virtual source.
  4. Right-click on brake and select Enable control by virtual source.



Perform the following to observe how a virtual source controls inports during simulation:

§21

  1. Click the  images/simOnBtn_web.png  (enable Simulator) button in the tool bar to start Reactis Simulator.
  2. Enter 100 in the text-entry box to the right of  images/fastPlayBtn_web.png  .
  3. Click  images/fastPlayBtn_web.png  to execute 100 fast simulation steps.
  4. Right-click on inport decelSet and select Open Scope. A scope like that shown in Figure 3.12 will appear. Note that the value of the decelSet during simulation matches that generated by the Signal builder block that implements the virtual source controlling the inport.




images/VS2_web.png
Figure 3.12: The Signal Builder block shown above implements a virtual source to control the decelSet inport (among others). The scope for decelSet, opened after executing 100 simulation steps, matches the signal defined in the Signal Builder block to demonstrate how the virtual source controls the inport.

3.4.7  Other Simulator Features

Simulator has several other noteworthy features. You may step both forward and backward through a simulation using toolbar buttons:

  • The  images/stepBtn_web.png  button executes a single execution step.
  • The  images/miniStepBtn_web.png  button executes the next “mini-step” (a single execution step consists in general of several mini-steps).
  • The  images/backBtn_web.png  button, which causes the simulation to go back (undo) a single step.
  • The  images/fbackBtn_web.png  button causes the simulation to go back multiple steps.
  • If using the Reactis for C Plugin with a model that includes C code, additional buttons appear for stepping through C code:  images/stepOutRBtn_web.png  ,  images/stepOverRBtn_web.png  ,  images/miniStepRBtn_web.png  ,  images/stepOverBtn_web.png  ,  images/stepOutBtn_web.png  . See Chapter 16, for a description of how these buttons work.

You may specify the number of steps taken when  images/playBtn_web.png  ,  images/fastPlayBtn_web.png  , or  images/fbackBtn_web.png  are pressed by adjusting the number in the text-entry box to the right of  images/fastPlayBtn_web.png  . When a simulation is paused at the end of a simulation step (as opposed to in the middle of a simulation step), the current simulation run may be added to the current test suite by selecting the menu item Test Suite -> Add/Extend Test. After the test is added it appears in the Source-of-Inputs dialog. After saving the test suite with Test Suite -> Save, the steps in the new test may be viewed by selecting Test Suite -> Browse. A model (or portion thereof, including coverage information) may be printed by selecting File -> Print....

Breakpoints may be set by either:

  • right clicking on a subsystem or state in the hierarchy panel and selecting Toggle Breakpoint; or
  • right clicking on a Simulink block, Stateflow state, or Stateflow transition in the main panel and selecting Toggle Breakpoint. Additionally, if you are using the Reactis for C Plugin, then a breakpoint may be set on any line of a C file that contains a C statement by right-clicking just to the right of the line number and selecting Toggle Breakpoint.

The  images/stopSignBtn_web.png  symbol is drawn on a model item when a break point is set. During a simulation run, whenever a breakpoint is hit, Simulator pauses immediately.

3.5  Tester

Tester may be used to generate a test suite (a set of tests) automatically from a Simulink / Stateflow model as shown in Figure 3.13. The tool identifies coverage targets in the model and aims to maximize the number of targets exercised by the generated tests.


images/testerOverview_web.png
Figure 3.13: Reactis Tester takes a Simulink / Stateflow model as input and generates a test suite.

To start Tester:

§22 Select the Test Suite -> Create menu item.


This causes the window shown in Figure 3.14 to appear. Tester employs a three-phase algorithm to generate test suites. In the Preload Phase, a set of previously constructed test suites may be loaded; Tester then attempts to extend the pre-loaded suite(s). In the Random Phase, Tester uses a fast random search strategy to generate test cases. In the Targeted Phase, the tool uses sophisticated algorithms to build tests that exercise previously uncovered parts of the model.


images/testerWin_web.png
Figure 3.14: The Reactis Tester launch dialog.

The Tester launch dialog consists of several sections. The first section (Preload Files) lists previously-generated test suites to be pre-loaded. The second section (Run for) determines how long tester should run. The third section (Coverage Objectives) lists the metrics which Tester will focus on during the targeted testing phase. In the fourth section, you specify the name of the output file in which Tester will store the new test suite (Output File).

There are three options provided in the Run for section. The default option, as shown in Figure 3.14, is to run Tester for 20,000 steps, with Tester choosing how many of these steps will be spent during random and targeted testing. Alternatively, you may choose to run Tester for a fixed length of time by clicking on the top radio button in the Run for section, after which the steps entry box will be disabled and the hours and minutes entry boxes will be enabled. These values are added together to determine the total length of time for which Tester will run, so that entering a value of 1 for hours and 30 for minutes will cause Tester to run for 90 minutes.

The bottom radio button in the Run for section lets you specify how many steps Tester spends on random testing and targeted testing. When selected, there are three parameters which can be edited. The first two (tests in random phase and steps per random test) determine the number of tests and steps per test to be generated in the random phase. It should be noted that, upon completion of the random phase, redundant steps are pruned from the tests, so the lengths of the final tests are usually shorter than the value entered for steps per random test. Note that the random phase is far less optimized than the targeted phase; therefore, the random phase should be allocated no more than a few thousand steps, leaving the vast majority of test-generation time for the targeted phase. The third parameter (steps in targeted phase) specifies the total number of execution steps which Tester will take during the targeted testing phase.

The Coverage Objectives section contains check boxes which are used to select the kinds of targets Tester will focus on during the targeted testing phase. Chapter 6 describes the different types of coverage tracked by Reactis.

To generate a test suite in the guided tour, retain the default settings and:

§23 Click the Create Suite button.



images/testerRunning_web.png
Figure 3.15: The Reactis Tester progress dialog.

The Tester progress dialog, shown in Figure 3.15, is displayed during test-suite generation. When Tester terminates, a results dialog is shown, and a file cruise.rst containing the suite is produced. The results dialog includes buttons for loading the test suite into the Test-Suite Browser (see below), Reactis Simulator, or a coverage viewer.

3.6  The Test-Suite Browser

The Test-Suite Browser is used to view the test suites created by Reactis. It may be invoked from either the Tester results dialog or the Reactis top-level window.


images/tsBrowser_web.png
Figure 3.16: The Reactis Test-Suite Browser.



§24 Select the Test Suite -> Browse menu item and then cruise.rst from the file selection dialog that pops up.


A Test-Suite Browser window like the one shown in Figure 3.16 is then displayed. The Test Data tab of the browser displays the test selected in the button/dialog located near the center of the browser’s tool bar. The main panel in the browser window shows the indicated test as a matrix, in which the first column gives the names of input and output ports of the model and each subsequent column lists the values for each port for the corresponding simulation step. The simulation time is displayed in the output row labeled “___t___”. The buttons on the tool bar may be used to scroll forward and backward in the test. The Test History and Suite History tabs display history information recorded by Reactis for the test and the suite as a whole.

The Filter entry box on the right side of the toolbar lets you search for test steps that satisfy a given condition. You enter a boolean expression in the search box and then select the filter check box to search for test steps in the suite for which the expression evaluates to true. For example, to see each step where the cruise control is active enter “active == 1” and then select the Filter check box.

The Test-Suite Browser may also be used to display the entire set of values passing over a port during a single test or set of tests.

§25 Perform the following in the Test-Suite Browser window.

  1. Left click on the row for inport drag.
  2. Right click on the row and select Open Distribution Scope (current test).



A dialog similar to that shown in Figure 3.17 appears. In the figure, each value assumed by the inport drag is represented by a yellow dot.


images/distributionScope_web.png
Figure 3.17: Values passing over inport drag during a test.

3.7  Validator

Reactis Validator is used for checking whether models behave correctly. It enables the early detection of design errors and inconsistencies and reduces the effort required for design reviews. The tool gives you three major capabilities.

Assertion checking.
You can instrument your models with assertions, which monitor model behavior for erroneous scenarios. The instrumentation is maintained by Reactis; you need not alter your .slx file. If Validator detects an assertion violation, it returns a test highlighting the problem. This test may be executed in Reactis Simulator to uncover the source of the error.
Test scenario specifications.
You can also instrument your models with user-defined targets, which may be used to define test scenarios to be considered in the analysis performed by Tester and Validator. Like assertions, user-defined targets also monitor model behavior; however, their purpose is to determine when a desired test case has been constructed (and to guide Reactis to construct it), so that the test may be included in a test suite.
Concrete test scenarios.
Finally, you can place virtual sources at the top level of a model to control one or more top-level inports as a model executes in Reactis. That is, you can specify a sequence of values to be consumed by an inport during simulation or test-generation. Virtual sources can be easily enabled and disabled. When enabled, the virtual source controls a set of inports and while disabled those inports are treated by Reactis just as normal top-level inports.

Conceptually, assertions may be seen as checking system behavior for potential errors, user-defined targets monitor system behavior in order to check for the presence of certain desirable executions (tests), and virtual sources generate sequences of values to be fed into model inports. Syntactically, Validator assertions, user-defined targets, and virtual sources have the same form and are collectively referred to as Validator objectives.

Validator objectives play key roles in checking a model against requirements given for its behavior. A typical requirements specification consists of a list of mandatory properties. For example, a requirements document for an automotive cruise control might stipulate that, among other things, whenever the brake pedal is depressed, the cruise control should disengage. Checking whether or not such a requirement holds of a model would require a method for monitoring system behavior to check this property, together with a collection of tests that would, among other things, engage the cruise control and then apply the brake. In Validator, the behavior monitor would be implemented as an assertion, while the test scenario involving engaging the cruise control and then stepping on the brake would be captured as a user-defined target.

3.7.1  Manipulating Validator Objectives

This section gives more information about Validator objectives.



§26 Do the following.

  1. Disable Simulator by clicking the  images/simOffBtn_web.png  button in the top-level tool bar. (Validator objectives may only be modified or inserted when Simulator is disabled.)
  2. In the model hierarchy panel, select the top-level of the model cruise for display in the main window.



After performing these operations, you now see a window like the one depicted in Figure 3.18. The three kinds of objectives are represented by different icons; assertions are denoted by a lightning bolt  images/assertionIconBtn_web.png  , targets are marked by a cross-hair symbol  images/targetIconBtn_web.png  , and virtual sources are represented by  images/virtualSourceIconBtn_web.png  .


images/validatorObjectives_web.png
Figure 3.18: The top-level Reactis window, with Validator objectives.

Validator objectives may take one of two forms.

Expressions.
Validator supports a simple expression language for defining assertions, user-defined targets, and virtual sources.
Simulink / Stateflow diagrams.
Simulink / Stateflow diagrams may also be used to define objectives. Such a Simulink subsystem defines a set of objectives, one for each outport of the subsystem. For assertion diagrams, a violation is flagged by a value of zero on an outport. For user-defined targets, coverage is indicated by the presence of a non-zero value on an outport. For virtual sources, each outport controls an inport of the model.

To see an example expression objective, do the following.



§27 Right click on the Brake assertion and select Edit Properties. When through with this step, click Cancel to dismiss the dialog.



images/exprObjDialog_web.png
Figure 3.19: The dialog for adding or modifying a Validator expression objective.

A dialog like that shown in Figure 3.19 now appears. This dialog shows an expression intended to capture the following requirement for a the cruise control: “Activating the brake shall immediately cause the cruise control to become inactive.” Note that the dialog consists of five parts:

  1. Name. This name labels the assertion in the model diagram.
  2. Enabled. This check box enables and disables the expression.
  3. Expression. This is a C-like boolean expression. The interpretation of such an assertion is that it should always remain “true”. If it becomes false, then an error has occurred and should be flagged. In this case, the expression evaluates to “true” provided that at least one of brake and active is false (i.e. the conjunction of the two is false). Section 9.3.1 contains more detail on the expression notation.
  4. # steps to hold. For assertions, this entry is an integer value that specifies the number of simulation steps that the expression must remain false before flagging an error. For user-defined targets, the entry specifies the number of steps that the expression must remain true before the target is considered covered.
  5. Inputs. The entry boxes in the right column of this section list the variables used to construct the expression. These can be viewed as virtual input ports to the expression objective. The pull-down menus to the left of the section specify which data items from the model feed into the virtual inputs. Note that, although you can manually specify the inputs and wiring within this dialog, it is simpler to:
    1. Leave this section blank when creating an objective.
    2. After clicking Ok to dismiss the dialog, drag and drop signals from the main Reactis panel onto the objective.

To see an example diagram objective:

§28 Perform the following.

  1. Using the model hierarchy panel, open the CruiseMDL subsystem located within the CruiseMain subsystem.
  2. In the main panel, right-click on the assertion SpdCheck and select Edit Properties.




images/diagObjDialog_web.png
Figure 3.20: The Properties dialog for a diagram objective (the assertion SpdCheck).

A dialog like the one in Figure 3.20 now appears. This dialog contains five sections.

  1. Name. The name of the objective.
  2. Enabled. This check box enables and disables the objective.
  3. Library. The Simulink library containing the diagram for the objective. In the case of SpdCheck, the target resides in cruise_validator.slx.
  4. System. The system in the library that is to be used as the objective. In this case, the system to be used is SpeedMaintenance.
  5. Parameters. If the subsystem selected for use as an objective is a masked subsystem, then this panel is used to enter the relevant parameters. In the case of SpdCheck, no parameters are required because the indicated subsystem is not masked.
  6. Inputs. The wiring panel is used to indicate which data items in the model should be connected to the inputs of the objective. In this case, SpdCheck has three inputs: speed, dSpeed, and active. The wiring information indicates that the first input should be connected to (the output of) Relational Operator1; the second should be connected to the dSpeed output of subsystem DesiredSpeed; and the third input should be connected to the speed inport of the current level of the model. In general, this panel contains pull-downs describing all valid data items in the current scope to which inputs of the objective may be connected.

    Note, that, although you can manually specify the wiring within this dialog, it is simpler to:

    1. Leave the wiring unspecified after inserting an objective.
    2. After clicking Ok to dismiss the dialog, drag and drop signals from the main Reactis panel onto the objective.

Wiring information may be viewed by hovering over a diagram objective in the main Reactis model-viewer panel.

Diagram-based objectives may be viewed as monitors that read values from the model being observed and raise flags by setting their outport values appropriately (zero for false, non-zero for true). A diagram-based assertion in essence defines one “check” for each of its outports, with such an outport check being violated if it ever assumes the value zero. Similarly, a diagram-based target objective in essence defines one “target” for each outport; such a target is deemed covered if it becomes non-zero.

To view the diagram associated with SpdCheck:

§29 Perform the following.

  1. Close the Properties dialog for SpdChk by clicking the Cancel button at the bottom of dialog box.
  2. Double-click on the SpdChk icon.



These operations display the Stateflow diagram shown in Figure 3.21. This diagram encodes the following requirement: “While it is engaged, the cruise control shall not allow the desired and actual speeds to differ by more than 1 mile per hour for more than three time units.”


images/spdCheck_web.png
Figure 3.21: A Validator diagram assertion encoding a requirement that the cruise control properly maintain the speed of the car when engaged.

To understand how this diagram captures this requirement, note that the SpdCheck diagram has two top-level states, one corresponding to when the cruise control is active and one when it is inactive. The active state has three child states:

OkDiff
Control resides here when the difference between actual and desired speed is within the accepted tolerance.
BigDiff
Control resides here when the difference between actual and desired speed is greater than the accepted tolerance, but three time units have not yet elapsed.
Error
Control resides here when the difference between actual and desired speed has been greater than the accepted tolerance for more than three consecutive time units.

Note that the transition action executed as state Error is entered sets the outport ok to 0. This is how an assertion violation is flagged.

Diagram objectives give you the full power of Simulink / Stateflow to formulate assertions and targets. The objectives may use any Simulink blocks supported by Reactis, including full Stateflow. The diagrams are created using Simulink and Stateflow in the same way standard models are built; they are stored in a separate .slx file from the model under validation.

Diagram wiring is managed by Reactis, so the model under validation need not be modified at all. As this information is stored in the .rsi file associated with the model, it also persists from one Reactis session to the next. After adding a diagram objective to a model, the diagram is included in the model’s hierarchy tree, just as library links are. See Chapter 9 for more details on using Reactis Validator.

3.7.2  Launching Validator

To use Validator to check for assertion violations:

§30 Select the Validate -> Check Assertions... menu entry.



images/validatorWin_web.png
Figure 3.22: Launching Validator brings up this screen.

A dialog like the one in Figure 3.22 now appears. The dialog is similar to the Tester launch screen in Figure 3.14 because the algorithms underlying the two tools are very similar. Conceptually, Validator works by generating thorough test suites from the instrumented model using the Tester test-generation algorithm and then applying these tests to the instrumented model to see if any violations occur. Note that when a model is instrumented with Validator objectives, the test-generation algorithm uses the objectives to direct the test construction process. In other words, Reactis actively attempts to find tests that violate assertions and cover user-defined targets. Validator stores the test suite it creates in the file specified in this dialog. The tests may then be used to diagnose any assertion violations that arise.


1
Prior to R2012b, the dialog was invoked via menu item File -> Model Properties...
2
Reactis supports the older .mdl model file format in the same way support is described here for .slx files.
3
A simulation is paused if Simulator is not actively computing simulation step(s).