DO-178C Structural Coverage and MC/DC: Questions and Answers

What structural coverage does DO-178C require?

It depends on the software level: level A requires statement, decision and MC/DC coverage, level B requires statement and decision coverage, level C requires statement coverage, and level D has no structural coverage objective.

The definitions of the metrics are the usual ones: statement coverage means every statement has been invoked at least once, decision coverage means every decision has taken both of its outcomes, and MC/DC means that, in addition, each condition in a decision has been shown to independently affect that decision’s outcome. MC/DC coverage explained gives the exact definition with a worked example.

Is structural coverage a substitute for requirements-based testing?

No: DO-178C asks first for tests derived from the requirements, and then uses structural coverage to measure how much of the code those tests exercised.

The order matters. Test cases are developed from the software requirements and executed; structural coverage analysis then shows which parts of the code structure those requirements-based tests did not exercise. Coverage is evidence about the completeness of the requirements-based tests, not a goal that tests can be written to satisfy on their own. A test suite that reaches 100% MC/DC without checking the outputs against the requirements would meet the coverage number without showing that the software does what it is meant to do.

What if the tests do not reach all of the code?

Find out why each uncovered item is uncovered: it may be a gap in the tests, a gap in the requirements, or code that should not be there, and each needs a different response.

• A shortcoming in the requirements-based tests. Add or extend tests until the code is exercised.
• An inadequacy in the requirements. The code does something that the requirements do not describe, so the requirements may need to change and be tested.
• Extraneous code, including dead code. Remove it, or justify keeping it.
• Deactivated code. Code that is intentionally not executed in this configuration needs its own analysis.

In Reactis, after test generation, examine the coverage targets that are flagged as unreachable: they point to dead or redundant logic in the model or code. A second common cause is a pair of conditions that depend on the same underlying signal, which makes it impossible to change one without changing the other, so independence cannot be shown; restructuring the logic is often the right fix. Another is input constraints that are tighter than the values that can really occur. See what if I cannot reach 100% MC/DC for the details.

What is the difference between dead code and deactivated code?

Dead code is code that is not traceable to any requirement and can never execute, usually the result of a development error, while deactivated code is traceable to requirements but is not meant to execute in a particular configuration.

A feature that is switched off in one variant of a product, for example by a configuration setting, is deactivated code: it has a purpose, but not here. Dead code has no purpose, and its presence points to an error in the development process, so it is normally removed or explained. Structural coverage analysis is how both are found: they appear as targets that no test can reach. The Reactis configuration variable mechanism can be used to test all variants and see the cumulative coverage.

How many tests does MC/DC need?

For a decision with N conditions, MC/DC needs at least N+1 test steps, far fewer than the 2^N combinations of multiple condition coverage.

For the decision A && B, the three tests (true, true), (true, false) and (false, true) give full MC/DC, where exhaustive testing of the combinations would need four. For a decision with ten conditions, MC/DC needs at least eleven test steps where exhaustive testing needs 1,024. In practice the tests must also be reachable through the system’s inputs, not just the decision in isolation, and that is the part that is hard to do by hand. The MC/DC guide shows the reasoning in more detail.

How do I measure structural coverage with Reactis?

Run a test suite in Reactis Simulator: Reactis highlights the model or code where targets are uncovered and can produce a coverage report from the test execution.

1. Open the model in Reactis for Simulink (or the code in Reactis for C) and load or generate a test suite. Tests that you wrote yourself, or that were created in another tool, can be imported from a CSV file.
2. Execute the tests in Simulator. Uncovered parts of the model are marked, and you can look at the details of each remaining target.
3. Create a coverage report from the test execution and keep it as part of your verification evidence.

The metrics Reactis tracks include statement, decision, condition, MC/DC and multiple condition coverage, together with model-specific metrics for Simulink and Stateflow. See Reactis coverage metrics.

Can Reactis generate tests that achieve the coverage?

Yes: Reactis Tester generates a test suite automatically and uses the targets that are still uncovered, including MC/DC targets, to steer the creation of each new test step.

Tester uses a patented technique called guided simulation. As it generates test data it tracks a collection of coverage metrics and uses the uncovered targets to influence the construction of subsequent test steps, and the resulting suites are minimized to remove redundant steps. If targets remain, the user-guided simulation feature of Simulator lets you extend a test or create a new one by hand.

These generated tests are best used for the second part of the DO-178C process, closing the gaps that structural coverage analysis finds. They do not replace the requirements-based tests, as explained above.

Where do the requirements-based tests come from?

From you: they come from your requirements; Reactis helps by letting you write each requirement as a Validator assertion or user-defined target and by importing the tests you already have.

In Validator you express requirements as assertions, properties that should always be true if the model is working properly, and as user-defined targets, which are behaviors that tests should exercise. Tester then searches for scenarios that violate the assertions or cover the targets. A test suite that you have written by hand can be loaded into Simulator to measure how much coverage it achieves, so the structural coverage analysis can be done on the tests you already trust. For code that was produced from a model, running the model’s tests on the code (back-to-back testing) checks that the code behaves like the model.

How does this apply to models, and what is DO-331?

DO-331 is the supplement to DO-178C for model-based development and verification; it carries the structural coverage idea over to models, where the coverage items are model elements such as blocks and state transitions.

When requirements or design are expressed as Simulink and Stateflow models, DO-331 includes model coverage analysis alongside the code-level analysis. In Reactis, coverage is measured on the model itself: for example, the decision of a Logical Operator block, an If block, or the event and guard of a Stateflow transition, together with Simulink-specific measures such as branch and subsystem coverage. How model coverage and code coverage are combined, and which one is credited for which objective, is something to agree with your certification authority in your plan for software aspects of certification.

Does Reactis measure these metrics on C code?

Yes: Reactis for C measures statement, decision, condition, MC/DC and multiple condition coverage, plus function and function-call coverage, for standalone C code.

For C code that is part of a Simulink model (S-Functions, C Caller blocks and Stateflow custom code), the Reactis for C Plugin tracks coverage in that code together with the rest of the model, so one test suite exercises both. Reactis measures coverage of the source code as the tests execute in Reactis. If your certification plan calls for coverage evidence at another level, such as object code on the target, plan for that separately.

Do I need to qualify a coverage tool?

Possibly: DO-178C requires tools to be qualified when their output is used to eliminate, reduce or automate a process and is not itself verified, and DO-330 describes how qualification is done.

A tool that measures coverage is normally a verification tool, one that cannot introduce an error into the software but could fail to detect one, and the standard sets a correspondingly lower qualification level for such a tool than for a tool that generates code. Whether you need to qualify a particular tool depends on how you use its results in your certification plan, so settle it with your certification authority early. The ISO 26262 equivalent is described in ISO 26262 software testing: questions and answers and in the Reactis Safety Manual.