State-Based Testing: Industrial Evaluation of the Cost-Effectiveness of Round-Trip Path and Sneak-Path Strategies

In the context of safety-critical software development, one important step in ensuring safe behavior is conformance testing, i.e., checking compliance between expected behavior and implementation. Round-trip path testing (RTP) is one example of conformance testing. Another essential step, however, is sneak-path testing, that is testing of how software reacts to unexpected events for a particular system state. Despite the importance of being systematic while testing, all testing activities take place, even for safety-critical software, under resource constraints. In this paper, we present an empirical evaluation of the cost-effectiveness of RTP when combined with sneak-path testing in the context of an industrial control system. Results highlight the importance of sneak-path testing since unexpected behavior is shown to be difficult to detect by other common, state-based test strategies. Results also suggest that sneak-path testing is a cost-effective supplement to RTP.