Thoroughness of specification-based testing of synchronous programs

Adequacy criteria for structural testing techniques are defined in terms of coverage (e.g. of statements or branches). For black-box testing, such criteria, making testers able to decide when to stop testing, are more difficult to define. In this paper, we propose such a definition for the particular case of specification-based testing techniques for synchronous software supplied by the Lutess testing environment. Lutess provides a framework for automatically building generators interacting with the software under test and feeding it with test input sequences. It requires a Lustre specification of both the software environment and the software safety properties. This specification is translated into an input-output automaton. A critical situation occurs when a safety property can be violated unless the software reacts adequately. Such situations correspond to particular states of the specification automaton, called suspect states. Suspect states definition can be used in two complementary ways: First, to design testing techniques able to reach several such states during testing. Second, to assess the thoroughness of a test input sequence in terms of covered suspect states. The above techniques are illustrated on a telephony software specification developed for the first Feature Interaction Detection Contest and involving 12 different telephone features. The thoroughness of the Lutess testing strategies is assessed as their ability to lead the software into suspect states.