An Analysis of Failure-Based Test Profiles for Random Testing

In random testing, the distribution of the generated test cases is known as the test profile. We consider the effects of different test profiles, taking advantage of probabilistic information about likely failure-revealing inputs, on the effectiveness of random testing for debugging. We examine a failure-proportional testing strategy, in which tests are randomly sampled with replacement, with probability proportional to a previously identified failure probability distribution, compared to a uniform testing strategy, in which tests are randomly sampled uniformly from the entire input domain. We show that neither strategy optimises failure-detection capabilities, and show an alternative strategy that does. We also consider selection without replacement, and examine the robustness of some strategies given a divergence between the estimated and actual failure probability distributions.