Software for embedded devices requires extensive testing before a product reaches market maturity. However, monitoring the testing progress, as required by many safety standards, is no trivial task. Mostly, the targets’ resources are limited; code coverage faces the constraints of low storage capacity and CPU performance. Choosing the right approach facilitates monitoring code coverage analysis on embedded systems.
Software testing is a vital element in creating secure and reliable embedded devices. Standards, such as IEC 61508, specify strict requirements for testing methods and code coverage regarding safety-critical software. Furthermore, proof of testing needs to be supplied for many embedded systems, in order to obtain mandatory certification prior to their market launch. As a general rule, the higher the safety requirements applied to a device, the higher the code coverage level entailed. However, not all code coverage levels are appropriate in any scenario. In addition, each of them will yield a different set of detailed insights:
· Function coverage only covers what functions have been tested, and ignores the internal routines of the software. It delivers a relatively low benefit.
· Statement coverage will determine which statements were executed by the tests. This helps in finding dead code, for instance.
· Branch coverage will confirm whether all programme branches were tested. Branch coverage is more extensive than statement coverage and a minimum testing requirement completed/obtained with reasonable effort.
· MC/DC (Modified Condition/Decision Coverage) is the highest code coverage level required by the standards. In simple terms: all atomic conditions of a composite decision are tested.
Instrumenting the code
Specific analysers are used to determine the code coverage, enhancing the code with counters for the desired test levels. Then it is passed to the compiler, a process referred to as code instrumentation. Of course, the instrumentation will not leave the code unaffected, since the counters are usually saved as global arrays. This becomes obvious from the following, while condition coded in C:
while (! b == 0 )
{
r = a % b;
a = b;
b = r;
}
result = a;
Following the instrumentation – in this case with the Testwell CTC++ code coverage tool – the structure significantly changes:
while ( (( ! b == 0 ) ? (ctc_t[23]++, 1) : (ctc_f[23]++, 0)) )
{
r = a % b ;
a = b ;
b = r ;
}
result = a ;
The instrumentation inflates the code. The necessary arrays are in the data storage: both RAM and ROM require additional capacity. This may cause problems in embedded systems since hardware resources are generally kept to a minimum, in order to cap costs. In this case, make sure you use a code coverage analyser with a relatively low instrumentation overhead, otherwise the counters may easily exceed the limits of the available storage capacity. This is particularly valid for the most demanding code coverage levels, including MC/DC.
In contrast, code coverage analysers for embedded systems, such as Testwell CTC++, will expand the code only slightly. In some cases, however, this may already exceed the hardware capabilities of a given target. Storage capacity issues may be worked around using partial instrumentation. That means, only small code segments of the programme will be instrumented and tested. The test will be successively repeated for all code segments; the data obtained this way may be put together to give an overall picture.
Smaller counters
Another approach for small targets is to restrict the size of counters. Code coverage tools usually work with 32-bit counters. They may be reduced to 8 or 16 bits, although the counters might overflow in this case. Great care should thus be exercised when interpreting the data collected in this manner. Usually, it’s not relevant how often specific parts of a code have been tested. What matters is if these parts have been tested at all. In such cases, the counters can be reduced to single bits (Bit Coverage).
It has to be kept in mind, though, that the chosen coverage level will impact the storage requirements. Generally, a higher test level will make greater demands on the target’s hardware resources. However, powerful code coverage tools are capable of minimising the storage requirements for the instrumentation and changes in run-time behaviour.
Conclusion
Testing and the determination of code coverage will gain significant importance with regard to embedded systems, since software quality is a critical/crucial issue. While certainly not all standards will – in the long run – require a 100 per cent MC/DC coverage for all types of software, it is only a matter of time for standardisation organisations and industry associations to raise their requirements. This will most certainly include applications that are not safety-critical. Improved testing standards are in the very interest of the manufacturers themselves. After all, defective products entail high costs, not to mention the damage to a manufacturer’s reputation. Sticking to the banana principle will hardly generate acceptance among customers.
