Are Fault Failure Rates Good Estimators of Adequate Test Set Size?

Test set size in terms of the number of test cases is an important consideration when testing software systems. Using too few test cases might result in poor fault detection and using too many might be very expensive and suffer from redundancy. For a given fault, the ratio of the number of failure causing inputs to the number of possible inputs is referred to as the failure rate. Assuming a test set represents the input domain uniformly, the failure rate can be re-defined as the fraction of failed test cases in the test set. This paper investigates the relationship between fault failure rates and the number of test cases required to detect the faults. Our experiments suggest that an accurate estimation of failure rates of potential fault(s) in a program can provide a reliable estimate of an adequate test set size with respect to fault detection (a test set of size sufficient to detect all of the faults) and therefore should be one of the factors kept in mind during test set generation.