Finding Run-Time Errors in Simulink Models and C Code

What is a run-time error, and why are they hard to find?

A run-time error is an operation that has no valid result, such as an overflow or a division by zero, and in most environments it does not stop the program: execution carries on with a wrong value and the symptom appears much later.

In a typical C environment most of these errors do not interrupt execution. They produce an unintended result, which is then used in later calculations and may not cause an observable malfunction, such as an incorrect output, until much later. By then the place where the program went wrong is far behind you, which is what makes the cause hard to track down. Simulink models have the same problem: a block can overflow or divide by zero during a simulation without anything looking wrong at that point.

Which run-time errors does Reactis find in Simulink models?

Reactis for Simulink identifies potential run-time errors such as bad indexing, integer overflow, integer division by zero, and floating-point operations that result in an infinity (Inf) or not-a-number (NaN) value.

For each potential run-time error Reactis constructs a run-time error assertion (an RTE assertion): a logical expression that evaluates to false if the run-time error occurs and to true otherwise. A circled check mark is drawn on a Simulink block, or on a Stateflow state or transition, anywhere a potential run-time error has been identified. The Reactis User’s Guide describes the RTE assertions in detail.

How does Reactis find them?

Reactis combines a static check, which proves some errors impossible, with dynamic test generation, which searches for the rest using its guided simulation technique.

1. Static check. Reactis first identifies the potential failures that can never occur. For an error that is shown to be impossible the circled check mark is drawn in green.
2. Dynamic search. For the errors that are not proved impossible, Reactis Tester searches for a way to make them happen. If it finds one, the check mark becomes red and yellow and Reactis returns a test that can be executed to diagnose and debug the problem.

Reactis Tester performs both steps. Its test generation uses a patented technique called guided simulation: it simulates the model and, at each step, selects inputs that either exercise new coverage targets or cause a run-time error. This is a white-box approach, somewhat like the fuzzers used in the security community but aimed at covering the structure of the model and at finding errors, not at random inputs. Because Tester also tries to maximize coverage of metrics such as decision, condition and MC/DC coverage, it explores more of the model, which increases the probability of finding run-time errors.

Does Reactis report false positives?

No: every run-time error that Reactis reports comes with a concrete test that you can execute to reproduce it.

Because the error was found by running the model with specific inputs, there is nothing to argue about: the test shows the inputs, and replaying it in Reactis Simulator shows the failure. The other direction needs care. A green check mark means that the error was proved impossible. An error that was neither proved impossible nor found by the generated tests is not claimed to be absent, so more test generation time, or a closer look at the input ranges, may still turn something up. The higher the level of coverage, the higher the level of assurance that a passing check will continue to pass.

How do I check a Simulink model for run-time errors?

Load the model in Reactis for Simulink and choose Test Suite > Create; Tester then performs the static and dynamic analysis and generates a test suite.

1. Open the model in Reactis for Simulink with File > Open Model….
2. If ranges for the top-level inports are specified in the Simulink Data Dictionary, Reactis imports them. Otherwise specify them with Edit > Inports. Realistic ranges matter: ranges that are wider than your system’s real inputs can produce errors that your system would never see, and ranges that are too narrow can hide errors that it can.
3. Choose Test Suite > Create… and run Tester.
4. Review the circled check marks on the model and the test suite that was generated. The run produces a set of tests that can be saved, replayed and used again as a regression suite.

The model needs no special instrumentation for this: load it and run Tester. See automatic test generation for Simulink.

What do I do when Reactis finds an error?

Execute the test that Reactis returned in Reactis Simulator, which replays the exact sequence of inputs that leads to the error, and use Simulator's debug functions to find the cause.

Simulator is an advanced debug environment for Simulink models. You can execute a block, a step or several steps at a time, run in reverse to see what led to the failing step, inspect values by hovering, in a scope or in the watched variables panel, and set breakpoints. It also highlights the parts of the model that the test has not exercised. Once the cause is fixed, run Tester again to confirm that the error has gone and to keep the new test suite. See debugging Simulink models with Reactis.

Which run-time errors does Reactis find in C code?

Reactis for C detects overflow, divide by zero, invalid shifts, memory errors, and reads of uninitialized data, and stops at the point where the error occurs.

Overflow

A numeric calculation that produces a result too large to represent.

Divide by zero

Dividing a numeric value by zero.

Invalid shift

Shifting an integer value by an amount that gives an undefined result according to the C standard.

Memory errors

Accessing an invalid memory region in a way that gives an undefined result, such as accessing an array outside its bounds or accessing heap-allocated memory after it has been freed.

Uninitialized data access

Accessing memory before it has been initialized, so that the result of the access is undefined under C semantics.

These checks run whenever Reactis for C simulates the code in Simulator or generates tests in Tester. All of these errors are detected immediately, so the source of the error can be found quickly; the inputs that lead to the error are recorded, so that you can replay the execution up to the point of failure and look at the earlier calculations that may be the real root cause. The Reactis for C User’s Guide lists the error types in detail.

Can you show an example of an overflow that goes unnoticed?

Yes: a C program that squares 1000 using 16-bit arithmetic compiles cleanly, runs normally, and prints 16960 instead of one million.

When an integer result is too large for its type, the bits that do not fit are discarded, so the value wraps around from a very large number to a much smaller one. One million is 1,000,000, and 16-bit arithmetic keeps only the remainder after dividing by 65,536, which is 16,960. Nothing in the program stops or complains, and the wrong value flows into whatever uses it next. Reactis for C can be configured to interrupt execution whenever wrapping would occur, which makes this kind of bug easy to find and fix.

How are memory errors in C different, and how are they found?

A memory error is a read from or a write to an invalid address; Reactis for C detects them as they happen, which a conventional debugger usually cannot, because the damage often shows up much later.

C gives the programmer direct access to the program’s memory, which can improve performance but also lets a defect touch arbitrary memory locations. Reactis groups memory errors into two kinds:

• Spatial errors access an address outside the bounds of the intended target: an invalid array index, a buffer overrun, or an invalid pointer (for example a pointer overwritten with a non-pointer value through a union).
• Temporal errors use memory that has been recycled: a heap error (using memory after free()) or a stack error (using a pointer to a local variable after its function has returned). These are usually more complex and harder to diagnose.

When a memory error occurs, Reactis suspends execution and shows an error dialog. The Highlight button flashes the source line, and the report gives the kind of error, the address that was being accessed, and the allowed numeric and symbolic access ranges. The symbolic range is especially useful: when a variable is reached through a pointer, it gives the name and source location of the variable the pointer was supposed to refer to, which a traditional debugger, that shows only a numeric address, does not. The same machinery catches reads of uninitialized heap memory and of uninitialized local variables. See Finding and Fixing Runtime Errors in the white paper Finding Bugs in C Code with Reactis for C.

Does this work for C code inside a Simulink model?

Yes: the Reactis for C Plugin applies the run-time error checks to the C code in S-Functions, C Caller blocks and Stateflow custom code, and the Reactis for EML Plugin does the same for Embedded MATLAB code.

Without a plugin, Reactis for Simulink treats this code as a black box. With the plugins, one test suite exercises the Simulink, Stateflow and C or MATLAB parts of the model together, and the errors found in the code come with tests that you can replay and step through in the same Simulator. See the Reactis for C Plugin and the Reactis for EML Plugin.

Is this relevant to security?

Yes: every potential run-time error is a potential vulnerability, so removing them from Simulink models and C code is a step towards a secure application.

Memory errors and overflows are among the defects that attackers exploit, and the checks are easy to apply throughout development because they are a push-button operation on a model or program. Removing run-time errors does not by itself make software secure and is not a substitute for a security review, but it takes out defects that such a review would otherwise have to look for by hand.

Can I run the checks automatically in continuous integration?

Yes: Reactis can be driven through its API from a continuous integration system such as Jenkins or GitLab, so run-time error checking can run on every change.

See the white papers Automated Testing with Reactis and Jenkins and Using Reactis with GitLab CI/CD. Both approaches use the Reactis for Simulink API.

Is a model or program free of run-time errors also correct?

No: eliminating run-time errors is an important step toward high-quality software, but a program that does not crash and does the wrong thing is still unacceptable.

Reactis therefore pairs the error checks with ways to check behavior against your requirements. In Validator you express requirements as assertions and user-defined targets, and Tester searches for scenarios that violate the assertions or exercise the targets. For C code that is derived from a model, you can also run the model’s test suite on the code and have Reactis flag any output that differs (back-to-back testing). See also MC/DC coverage explained and ISO 26262 software testing.