Risk-Driven Software Reliability and Testing

Classical methods of accepting or rejecting products, which are not generally known by the software reliability community, are used as a baseline for developing test criteria. Testing criteria are designed to estimate the number of tests, and associated fault removal, that would result in reduced risk and increased reliability to the point where the software can be deployed. For the purpose of experimenting with test criteria, which are based on relationships between the producer and the consumer of software, we developed a template for test scenarios, using hypothetical distributions of faults and failures. The template was applied to a release of the Shuttle flight software to determine whether hypothetical and real-world results would be consistent. This was not the case; hence, we recommend that practitioners experiment with more than one test criterion to reduce the risk of deploying their software.