Reactis Coverage Metrics

7. Reactis Coverage Metrics#

Reactis for C uses a number of different coverage metrics to measure how thoroughly a test or set of tests exercises a program. In general, coverage metrics record how many of a given class of syntactic constructs, or coverage targets, that appear in a program have been executed at least once. The metrics discussed in this chapter may be visualized using Simulator and are central to test generation and C code validation using Tester and Validator.

7.1. Statement Coverage#

Statement coverage records for each statement in a C program, whether or not the statement has been executed at least once.

7.2. Function Coverage#

The Function metric records whether or not each function in a program as been called at least once. One target per function is created and a target is covered the first time the function is called.

7.3. Function Call Coverage#

The Function Call metric records whether or not each particular call to a function has been executed at least once. One target per call to any function is created (i.e., if a function is called from multiple places, then a separate target is created for each call). A target is covered the first time the function is called from the location associated with the target. Targets are not created for calls via a pointer to a function.

Note that a single statement or condition may contain calls to more than one function, so function call coverage is not a subset of statement coverage.

7.4. Decision Coverage#

Decision coverage tracks whether each decision in a program has evaluated to both true and false. The program elements that Reactis for C identifies as decisions can be configured via the Decision metric setting in the C Code tab of the Reactis for C Global Settings dialog. There are three possible Decision metric settings:

Expressions which control if/while/for statements.: Only expressions which appear within the parentheses following the if, while, or for keywords are decisions. Boolean expressions appearing in other contexts, such as the expression x < 100 in the assignment statement b = x < 100, are not decisions.
Expressions which control if/while/for statements or the ? operator.: In addition to expressions which appear within the parentheses following the if, while, or for keywords, the first argument of the C ? operator is also a decision. For example, in a program which contains the statement x = y<0? -y : y, y < 0 is a decision.
All non-trivial Boolean expressions except static initializers.: Under this criterion, any Boolean expression which does not appear in a static initializer and also is not a simple assignment of one Boolean variable to another, is a decision. For example, if a function f contains the assignment b = x < 100 (where b is a local variable of f), then x < 100 is a decision.

Note that the argument of a switch statement is not a decision.

7.5. Condition Coverage#

Condition coverage tracks whether each condition in a program has evaluated to both true and false. A condition is a Boolean-valued subexpression of a decision which does not contain any Boolean operators. For example, if x < 100 || !(y > 100) is a decision, then x < 100 and y > 100 are conditions.

7.6. MC/DC Coverage#

The MC/DC coverage metric is somewhat more complex to define. It was introduced by John J. Chilenski of Boeing in the early 90s; the definitive research paper was published by Chilenski and Steve Miller, of Rockwell-Collins, in 1994. MC/DC is the level of testing mandated by the Federal Aviation Administration (FAA) in its DO-178/B guidelines for the “most safety-critical” components of aviation software. The MC/DC targets in a program are the conditions; a condition C in decision D is covered by a test suite if there are two test steps X and Y (not necessarily consecutive) in the suite such that:

C evaluates to a different truth value (true or false) in step X than in step Y; and
each condition other than C in D evaluates to the same truth value in both step X and step Y; and
D evaluates to a different truth value in step X than in step Y.

In other words, each condition must be shown to independently affect the outcome of its enclosing decision.

7.7. MCC Coverage#

Multiple Condition Coverage (MCC, also known as Condition Combination Coverage) tracks one target for each combination of values to which the conditions in a decision evaluate. For a decision with \(N\) conditions this will create \(2^N\) MCC targets. A condition D = C1 && C2 && C3 will need 8 MCC targets to track all possible combinations of C1, C2 and C3 (each row in the table below represents one MCC target):

C1	C2	C3	D
false	false	false	false
false	false	true	false
false	true	false	false
false	true	true	false
true	false	false	false
true	false	true	false
true	true	false	false
true	true	true	true

However, in virtually all programming languages (including C), Boolean expressions are always evaluated using short-circuiting for reasons of efficiency. A short-circuited operator avoids evaluating all conditions of a decision if the outcome is determined after evaluating only a subset of the conditions. For example, a short-circuited “and” operator would not evaluate its second argument if the first evaluates to false. Without going into the technical details, it can be said that achieving full MC/DC coverage is easier if short-circuited Boolean operators are used. For MCC coverage, short-circuited expressions generate a reduced number of coverage targets compared to non-short-circuited expressions. The D = C1 && C2 && C3 decision creates only 4 (as opposed to 8) MCC targets (the x characters in the table below represent “don’t care” terms - conditions that were not evaluated due to short-circuiting and whose value therefore can not be recorded):

C1	C2	C3	D
false	x	x	false
true	false	x	false
true	true	false	false
true	true	true	true

7.8. Boundary Value Coverage#

Boundary value coverage tracks whether a data item assumes values of interest for a particular variable. The boundary values tracked for an input or configuration variable are determined by its associated type [1] 1 as shown in Table 7.1. If an input or configuration variable has a type not shown in the table, then it has no boundary value targets.

Table 7.1 Boundary values associated with each type.#
RSI Type	Boundary Values
\(t[i, j]\), where \(t\) is `double`, `single`, `sfix`, or `ufix`	if \(i < 0\) and \(j > 0\) then \(i, 0.0, j;\) otherwise \(i, j\)
\(t[i, j]\), where \(t\) is and `integer` type	if \(i < 0\) and \(j > 0\) then \(i, i+1, 0, j−1, j;\) otherwise \(i, i+1, j−1, j\)
\(t{ e_{1}, … e_{n} }\)	\(e_{1}, … e_{n}\)
\(t [i:j:k]\)	if there exists a positive integer \(l\) such that \(i + l × j = 0.0\) then \(i, i+j, 0.0, k−j, k;\) otherwise \(i, i+j, k−j,k\)
\(t\) `delta` \([i,j]\)	boundary values of \(t\)
`boolean`	\(true, false\)
`int8`	\(-128, -127, 0, 126, 127\)
`int16`	\(-32768, -32767, 0, 32766, 32767\)
`int32`	\(-2147483648, -2147483647, 0 2147483646, 2147483647\)
`uint8`	\(0, 1, 254, 255\)
`uint16`	\(0, 1, 65534, 65535\)
`uint32`	\(0, 1, 4294967294, 4294967295\)
`double`	\(0.0\)
`single`	\(0.0\)

7.9. Assertion Coverage#

See the Assertions section.

7.10. User-Defined Target Coverage#

See the User Defined Coverage Targets section.

7.11. Coverage Highlighting#

When Coverage > Show Details in Simulator is selected, uncovered coverage targets in C code are reported visually as shown in Figure 7.1. Any C statement which has not been executed is rendered in red. If a decision has not evaluated to true it has a thin red overline. If a decision has not evaluated to false it has a thin red underline. If a condition has not evaluated to true it has a thick red overline. If a condition has not evaluated to false it has a thick red underline. If a decision has not met the MC/DC or MCC criteria, then the text of the decision is displayed in red. The MC/DC-related and MCC coverage details associated with a decision may be displayed by right clicking on the decision and selecting View Coverage Details.

To aid the user in finding uncovered targets, a thin red bar is drawn just to the right of the line number for any line that contains at least one uncovered coverage target.

_images/cvgHighlighting.png — Fig. 7.1 Coverage highlighting in C code.#

7.12. Excluding Coverage Targets#

Reactis for C supports several different ways to disable coverage tracking for a subset of targets. When a target is excluded using any of these mechanisms, Reactis Tester will not try to generate a test to exercise the target and Reactis Simulator will not report the target as uncovered.

7.12.1. Disabling a Coverage Metric#

In some cases, tracking a certain metric may not be desirable. For example, if accomplishing 100% MCC coverage requires an excessive number of tests, you may wish to not track that metric. To help you focus on the set of metrics of highest interest to you, Reactis lets you disable a whole coverage metric.

_images/disableStmtCvg.png — Fig. 7.2 Disabling all statement coverage targets via the Reactis Harness Editor (**Edit > Coverage Metrics**)#

To disable a coverage metric, do the following:

Make sure Reactis Simulator is disabled.
Open the Coverage Metrics tab in the Reactis Harness Editor (select Edit > Coverage Metrics…). This tab is shown in Figure 7.2.
Switch the desired metric to Disabled. For example, in Figure 7.2, statement coverage (item 1) has been disabled.

Disabling a coverage metric will cause it to not be shown in any of the various places where Reactis lists coverage metrics, including the Tester launch and run dialogs, the Coverage Summary dialog, the Coverage Report Browser, and exported coverage reports. Also, Reactis Tester will not target disabled metrics when generating tests.

7.12.2. Excluding Files#

Reactis for C can disable coverage for all targets within a selected file or library. This can be useful in many cases. For example, the code in a particular file might be already be covered by a different unit test, making it undesirable to include that code when measuring the coverage of subsequent tests.

_images/cvgFileDisable.png — Fig. 7.3 Disabling all statement coverage targets via the Reactis Harness Editor (**Edit > Coverage Metrics**)#

To disable coverage for a particular file or library, right-click on the name of the file in the hierarchy tree and select Coverage Tracking. This will open a submenu with four choices, as shown in item 1 of Figure 7.3:

Enable: Turn coverage tracking for the selected item on.
Disable: Turn coverage tracking for the selected item off.
Inherit: Inherit the coverage tracking setting for the selected item from its parent in the hierarchy tree.
Reset to Inherited: Change the coverage tracking setting for the selected item and all its descendants within the hierarchy tree to Inherit.

Files and libraries for which coverage is disabled are indicated by a ghosted appearance, as shown in item 2 of Figure 7.3. In addition, items whose coverage is explicitly set to Disable will be annotated with an \(x\), and items whose coverage is explicitly set to Enable will be annotated with a \(√\).

7.12.3. Excluding Individual Targets#

Excluding individual targets from coverage tracking allows fine-grained control if the code under test contains a few specific targets that cannot be exercised. Excluding such targets from coverage tracking (after confirming their unreachable status) can help you achieve the goal of 100% coverage of reachable targets.

Reactis offers two alternative ways to exclude a coverage target. If you simply exclude a target, Reactis will not attempt to exercise the target when generating tests and it will not report it as covered or uncovered. The second way to exclude a target is to exclude and monitor a target. This method lets you assert that a target is unreachable and therefore should not be included in the coverage reports; but, Reactis will also monitor the target and alert you if the target is ever exercised – that is, you will be notified if your claim that the target is unreachable is incorrect. Reactis accomplishes this by automatically creating a Validator assertion for each excluded and monitored target to flag if the target is ever exercised. If such an assertion is violated (either while generating tests in Tester or running the code under test in Simulator), Reactis will report the violation and Simulator can be employed to investigate how the presumably unreachable target got covered.

All target types tracked by Reactis for C except for boundary targets can be excluded. The method to exclude a target differs slightly by target type.

The process for excluding Decision, Condition, MC/DC, or MCC targets from coverage requires three steps (labeled 1-3 in Figure 7.4):

Right-click on the code region containing the target and select View Coverage Details.
In the resulting dialog, right-click on the test/step information of the specific target.
Select the Track Coverage menu item.

_images/excludeCondTarget.png — Fig. 7.4 Changing the exclusion status of a condition target.#

For all other targets right-click on a statement, name of a function at the point where the function is defined, or name of a function within a function-call expression and select Track Coverage and select the appropriate target from the submenu (see Item 1 of Figure 7.5). A check-mark in the submenu listing coverage targets indicates that coverage is currently being tracked for that target.

_images/excludeStmtTarget.png — Fig. 7.5 Changing the exclusion status of a statement target.#

Selecting Track Coverage brings up the dialog shown in Item 2 of Figure 7.5. This dialog offers three choices for the coverage exclusion status:

Track coverage for this target

If this option is selected, coverage is tracked as usual for the target.

Exclude target from coverage tracking

If selected, coverage is not tracked for the target:

The color of the target in the source code will change to blue (instead of black or red).
The target will not be counted in the coverage summary.
The target will be added to the list of targets in the Excluded Coverage Targets pane of the Reactis Harness Editor (invoked by Edit > Excluded Coverage Targets).
Whenever the coverage status is displayed for the target (e.g., in coverage reports and when hovering), the status will be excluded.

Exclude target from coverage tracking and monitor via assertion

The same rules apply as for the case above, with the following additions:

An assertion will be automatically created for the target. The assertion is included in the Assertion target count in the coverage summary.
The status of the excluded target will be shown as excluded in the coverage report, list of excluded targets and when hovering.

If an excluded and monitored target gets covered, this is recorded as follows:

The associated assertion changes to violated status.
The region of code containing the target is highlighted in yellow in the main panel
In the coverage report, the list of excluded targets, and when hovering in the main panel, the status will be listed as [t/s] where t/s is the test/step number in which the target was covered.
The associated assertion will be listed as violated in the test execution report.

7.12.4. Excluding Functions#

It is also possible to exclude all targets within a selected function from coverage tracking. When you right-click on the name of a function at the point where it is defined, and select Track Coverage, the submenu will contain an additional choice Track All Targets In Function… (see item 1 of Figure 7.6).

_images/excludeFunction.png — Fig. 7.6 Changing the exclusion status for all targets in a function.#

Selecting Track Coverage brings up the dialog shown in Item 2 of Figure 7.6. This dialog offers two choices for the coverage exclusion status: