Chapter 7 Reactis Coverage Metrics
Reactis for C uses a number of different coverage metrics to measure how thoroughly a test or set of tests exercises a program. In general, coverage metrics record how many of a given class of syntactic constructs, or coverage targets, that appear in a program have been executed at least once. The metrics discussed in this chapter may be visualized using Simulator and are central to test generation and C code validation using Tester and Validator.
7.1 Statement Coverage
Statement coverage records for each statement in a C program, whether or not the statement has been executed at least once.
7.2 Function Coverage
The Function metric records whether or not each function in a program has been called at least once. One target per function is created and a target is covered the first time the function is called.
7.3 Function Call Coverage
The Function Call metric records whether or not each particular call to a function has been executed at least once. One target per call to any function is created (i.e., if a function is called from multiple places, then a separate target is created for each call). A target is covered the first time the function is called from the location associated with the target. Targets are not created for calls via a pointer to a function.
Note that a single statement or condition may contain calls to more than one function, so function call coverage is not a subset of statement coverage.
7.4 Decision Coverage
Decision coverage tracks whether each decision in a program has evaluated to both true and false. The program elements that Reactis for C identifies as decisions can be configured via the Decision metric setting in the C Code tab of the Reactis for C Global Settings dialog. There are three possible Decision metric settings:
Note that the argument of a
7.5 Condition Coverage
Condition coverage tracks whether each condition in a program has
evaluated to both true and false. A condition is a Boolean-valued subexpression
of a decision which does not contain any Boolean operators. For example,
7.6 MC/DC Coverage
The MC/DC coverage metric is somewhat more complex to define. It was introduced by John J. Chilenski of Boeing in the early 90s; the definitive research paper was published by Chilenski and Steve Miller, of Rockwell-Collins, in 1994. MC/DC is the level of testing mandated by the Federal Aviation Administration (FAA) in its DO-178/B guidelines for the “most safety-critical” components of aviation software. The MC/DC targets in a program are the conditions; a condition C in decision D is covered by a test suite if there are two test steps X and Y (not necessarily consecutive) in the suite such that:
In other words, each condition must be shown to independently affect the outcome of its enclosing decision.
7.7 MCC Coverage
Multiple Condition Coverage (MCC, also known as Condition
Combination Coverage) tracks one target for each combination of
values to which the conditions in a decision evaluate. For a decision
with N conditions this will create 2N MCC targets. A condition
However, in virtually all programming languages (including C),
Boolean expressions are always evaluated using short-circuiting
for reasons of efficiency. A short-circuited
operator avoids evaluating all conditions of a decision if the outcome
is determined after evaluating only a subset of the conditions. For
example, a short-circuited “and” operator would not evaluate its
second argument if the first evaluates to false.
Without going into the technical details, it can be said that
achieving full MC/DC coverage is easier if short-circuited Boolean
operators are used. For MCC coverage, short-circuited expressions
generate a reduced number of coverage targets compared to
non-short-circuited expressions. The
7.8 Boundary Value Coverage
Boundary value coverage tracks whether a data item assumes values of interest for a particular variable. The boundary values tracked for an input or configuration variable are determined by its associated type 1 as shown in Table 7.1. If an input or configuration variable has a type not shown in the table, then it has no boundary value targets.
7.9 Assertion Coverage
See Section 10.1.1.
7.10 Boundary Value Coverage
See Section 10.1.2.
7.11 Coverage Highlighting
When Coverage -> Show Details in Simulator is selected, uncovered coverage targets in C code are reported visually as shown in Figure 7.1. Any C statement which has not been executed is rendered in red. If a decision has not evaluated to true it has a thin red overline. If a decision has not evaluated to false it has a thin red underline. If a condition has not evaluated to true it has a thick red overline. If a condition has not evaluated to false it has a thick red underline. If a decision has not met the MC/DC or MCC criteria, then the text of the decision is displayed in red. The MC/DC-related and MCC coverage details associated with a decision may be displayed by right clicking on the decision and selecting View Coverage Details.
To aid the user in finding uncovered targets, a thin red bar is drawn just to the right of the line number for any line that contains at least one uncovered coverage target.
7.12 Excluding Coverage Targets
Reactis for C supports several different ways to disable coverage tracking for a subset of targets. When a target is excluded using any of these mechanisms, Reactis Tester will not try to generate a test to exercise the target and Reactis Simulator will not report the target as uncovered.
7.12.1 Disabling a Coverage Metric
In some cases, tracking a certain metric may not be desirable. For example, if accomplishing 100% MCC coverage requires an excessive number of tests, you may wish to not track that metric. To help you focus on the set of metrics of highest interest to you, Reactis lets you disable a whole coverage metric.
To disable a coverage metric, do the following:
Disabling a coverage metric will cause it to not be shown in any of the various places where Reactis lists coverage metrics, including the Tester launch and run dialogs, the Coverage Summary dialog, the Coverage Report Browser, and exported coverage reports. Also, Reactis Tester will not target disabled metrics when generating tests.
7.12.2 Excluding Files
Reactis for C can disable coverage for all targets within a selected file or library. This can be useful in many cases. For example, the code in a particular file might be already be covered by a different unit test, making it undesirable to include that code when measuring the coverage of subsequent tests.
To disable coverage for a particular file or library, right-click on the name of the file in the hierarchy tree and select Coverage Tracking. This will open a submenu with four choices, as shown in item 1 of Figure 7.3:
Files and libraries for which coverage is disabled are indicated by a ghosted appearance, as shown in item 2 of Figure 7.3. In addition, items whose coverage is explicitly set to Disable will be annotated with an x, and items whose coverage is explicitly set to Enable will be annotated with a √.
7.12.3 Excluding Individual Targets
Excluding individual targets from coverage tracking allows fine-grained control if the code under test contains a few specific targets that cannot be exercised. Excluding such targets from coverage tracking (after confirming their unreachable status) can help you achieve the goal of 100% coverage of reachable targets.
Reactis offers two alternative ways to exclude a coverage target. If you simply exclude a target, Reactis will not attempt to exercise the target when generating tests and it will not report it as covered or uncovered. The second way to exclude a target is to exclude and monitor a target. This method lets you assert that a target is unreachable and therefore should not be included in the coverage reports; but, Reactis will also monitor the target and alert you if the target is ever exercised – that is, you will be notified if your claim that the target is unreachable is incorrect. Reactis accomplishes this by automatically creating a Validator assertion for each excluded and monitored target to flag if the target is ever exercised. If such an assertion is violated (either while generating tests in Tester or running the code under test in Simulator), Reactis will report the violation and Simulator can be employed to investigate how the presumably unreachable target got covered.
All target types tracked by Reactis for C except for boundary targets can be excluded. The method to exclude a target differs slightly by target type.
The process for excluding Decision, Condition, MC/DC, or MCC targets from coverage requires three steps (labeled 1-3 in Figure 7.4):
For all other targets right-click on a statement, name of a function at the point where the function is defined, or name of a function within a function-call expression and select Track Coverage and select the appropriate target from the submenu (see Item 1 of Figure 7.5). A check-mark in the submenu listing coverage targets indicates that coverage is currently being tracked for that target.
Selecting Track Coverage brings up the dialog shown in Item 2 of Figure 7.5. This dialog offers three choices for the coverage exclusion status:
7.12.4 Excluding Functions
It is also possible to exclude all targets within a selected function from coverage tracking. When you right-click on the name of a function at the point where it is defined, and select Track Coverage, the submenu will contain an additional choice Track All Targets In Function… (see item 1 of Figure 7.6).
Selecting Track Coverage brings up the dialog shown in Item 2 of Figure 7.6. This dialog offers two choices for the coverage exclusion status: