ISO 26262 Software Testing: Questions and Answers

Which parts of ISO 26262 cover software testing?

Part 6 (product development at the software level) covers software testing, and its clauses 9 (software unit verification) and 10 (software integration and verification) are the core testing activities, backed by clause 7 (architectural design) and clause 11 (verification of the software safety requirements).

• Clause 7, software architectural design: verification of the architecture, including walkthroughs, inspections and simulation of the design’s dynamic behavior (requirement 7.4.14).
• Clause 8, software unit design and implementation: design principles, including methods to detect and eliminate run-time errors (sub-clause 8.4.5).
• Clause 9, software unit verification: unit testing, including back-to-back comparison of model and implementation (9.4.2), deriving test cases (9.4.3) and measuring structural coverage (9.4.4).
• Clause 10, software integration and verification: integration testing, including requirements-based testing and back-to-back comparison (10.4.2) and coverage at the integration level (10.4.4).
• Clause 11, verification of the software safety requirements: verification on the target, such as hardware-in-the-loop testing (11.4.1).

What structural coverage does ISO 26262 call for?

ISO 26262-6 requires that the structural coverage achieved by unit tests be measured, and recommends statement, branch and MC/DC coverage at the unit level and function and call coverage at the integration level.

Requirement 9.4.4 says the structural coverage achieved when executing tests must be measured at the software unit level, together with an analysis that reveals shortcomings, including the discovery of dead code. If the level of coverage is judged to be insufficient, additional tests can be written, or a rationale based on other methods can be provided. At the integration level, requirement 10.4.4 recommends measuring function coverage and call coverage.

Reactis for Simulink tracks decision, condition, MC/DC and MCC coverage in Simulink and Stateflow models, and, with the C and EML plugins, statement coverage and the same metrics in the C and Embedded MATLAB code inside a model. Reactis for C tracks statement, decision, condition, MC/DC, MCC, function and function-call coverage in standalone C code. See MC/DC coverage explained and Reactis coverage metrics.

Is MC/DC required for ASIL D?

ISO 26262-6 does not make one coverage metric mandatory for every project: it recommends structural coverage metrics at the unit level and grades how strongly each is recommended by ASIL, with MC/DC carrying its strongest recommendation at ASIL D.

Check the table in ISO 26262-6 for your ASIL and apply the recommendation as written for your project. Where you do not follow a recommendation, the standard expects a rationale. See MC/DC coverage explained for what the metric means and how many tests it takes.

What is back-to-back testing, and when is it recommended?

Back-to-back testing checks that an implementation and its model produce the same outputs for the same inputs, and ISO 26262-6 (sub-clauses 9.4.2 and 10.4.2) recommends comparing a model with the implementation derived from it.

Four things make a back-to-back test meaningful. There should be high confidence in the model from earlier testing against its requirements; the implementation’s outputs should be reasonably close to the model’s (small differences can come from rounding in numerical calculations); and the tests should achieve a high degree of coverage of both the model and the implementation.

With Reactis, back-to-back testing of a model and its C code takes three steps:

1. Generate a test suite from the model with Reactis Tester.
2. Execute that test suite on the C code in Reactis for C.
3. Reactis flags any differences in behavior; the test suite and reports serve as evidence of the verification activity.

The C code can be hand-written or generated by a tool such as Embedded Coder or TargetLink. The same approach works for two other comparisons: a new version of a model against the previous one (regression testing, which ISO 26262-6 sub-clause 5.2 supports through methods such as nightly regression testing), and a model against a compiled executable by exporting the tests to CSV and running them in your own environment, which can be part of the hardware-in-the-loop testing recommended in 11.4.1. See back-to-back testing with Reactis.

How do I show that tests verify the requirements?

Express each requirement as an assertion or a user-defined target, let Reactis search for scenarios that violate the assertions or cover the targets, and keep the final test suite and report as evidence.

ISO 26262-6 requirement 10.4.2 calls for software integration to be verified by a variety of methods, including requirements-based testing. In Reactis Validator:

1. Analyze the system requirements and express them as Validator assertions and user-defined targets.
2. Run Reactis Tester to search for scenarios that cover the user-defined targets or detect assertion failures.
3. Use Reactis Simulator to diagnose and debug any failures.
4. Generate a report from the final test suite, which covers all user-defined targets and reports no assertion failures, to demonstrate that the requirements are satisfied.

How do I find run-time errors?

Generate a test suite for the model or C code with Reactis Tester, which flags run-time errors as it explores the behavior, and then replay the failing tests in Simulator to diagnose them.

ISO 26262-6 requirement 7.4.14 specifies simulation of the design’s dynamic behavior as a verification method, and sub-clause 8.4.5 describes design principles, including methods to detect and eliminate run-time errors, that apply at the unit level. A test suite that achieves a high level of coverage gives the best chance of finding a run-time error, because every part of the logic is exercised. For C code, Reactis detects errors such as out-of-bounds array indexes, buffer overruns and dangling pointers. See C code analysis with Reactis, and Finding Run-Time Errors in Simulink Models and C Code for the full list of errors and how they are found.

How do I find dead code?

Generate and run a test suite, then examine the coverage targets that Reactis flags as unreachable; these point to dead code in the model or C code.

Requirement 9.4.4 asks for an analysis that reveals shortcomings in structural coverage, including the discovery of dead code. Examining the unreachable targets after test generation is the Reactis way to do this for both Simulink models and C code.

Can I measure coverage of tests created outside Reactis?

Yes: import the tests from a CSV file, execute them in Reactis Simulator, and generate a coverage report.

If coverage does not reach 100%, Reactis highlights the model or code so that uncovered targets are easy to find. You can then add tests, including user-constructed tests created with the user-guided simulation feature, or provide a rationale as allowed by requirement 9.4.4. Imported tests can also be checked for run-time errors.

Which ISO 26262-6 activities does Reactis support?

Reactis supports verification activities in clauses 7 to 11 of ISO 26262-6, from architecture walkthroughs to exporting tests for target testing.

Reactis also supports walkthroughs and inspections with hierarchical browsing of Simulink, Stateflow, C and Embedded MATLAB, signal highlighting and text search. The Reactis Safety Manual describes each use case in detail.

Do I need to qualify Reactis as a software tool?

ISO 26262-8 clause 11 requires you to determine the confidence you need in each software tool, and Reactis has been prequalified by TÜV SÜD for use in an ISO 26262 process, but you remain responsible for confirming that the prequalification suits your project.

• Tool classification. ISO 26262-8 rates a tool by its tool impact (TI1 or TI2) and tool error detection (TD1 to TD3), which combine into a tool confidence level (TCL1 to TCL3). TCL1 needs no qualification; TCL2 and TCL3 do. The Reactis Safety Manual summarizes a failure modes and effects analysis (FMEA) of Reactis in each use case.
• Certified prequalification. Reactive Systems worked with TÜV SÜD, evaluating the Reactis tool development process under sub-clause 11.4.8 of Part 8 and validating Reactis under 11.4.9. TÜV SÜD certifies that Reactis is qualified for safety-related development according to ISO 26262, up to ASIL D. The certificate is available.
• Assumptions of use. The certification assumes, among other things, that Reactis is used inside an ISO 26262-compliant process, that you use a prequalified version, that you follow one of the documented use cases, and that you investigate any run-time errors and output differences that Reactis flags. The Safety Manual lists the prequalified versions and the supported MATLAB versions.

What evidence does Reactis produce?

Reactis produces test suites, coverage reports and test execution reports that can serve as evidence of verification activities.

The generated test suite is an artifact you can keep and re-run for regression testing. Coverage reports show which targets were covered and which were not, and test execution reports record the results of running a test suite, including the error-detection settings used, which acts as a safeguard against running with the wrong configuration. Because test suites can be exported to CSV, the same tests can be run in other environments, such as on the target.

Does using Reactis make my project ISO 26262 compliant?

No: compliance is a property of your whole development process, and Reactis supports specific verification activities within it; its prequalified status means you do not need to qualify the tool from scratch.

Reactis must be used within an ISO 26262-compliant development process and tool workflow, with the prequalified version and one of the documented use cases. The Safety Manual states that it is the end user’s responsibility to confirm that the certified prequalification is suitable for a particular project.