An Experimental Evaluation of the Reliability of Adaptive Random Testing Methods

Adaptive random testing (ART) techniques have been proposed in the literature to improve the effectiveness of random testing (RT) by evenly distributing test cases over the input space. Simulations and mutation analyses of various ART techniques have demonstrated their improvements on fault detecting ability when measured by the number of test cases required to detect the first fault. In this paper, we report an experiment with ART using mutants to evaluate ARTpsilas reliability in fault detecting ability. Our experiment discovered that ART is more reliable than RT in the sense that its degree of variation in fault detecting ability is significantly lower than RT. It is also recognized from the experiment data that the two main factors that affect ARTpsilas reliability are the failure rate of the system under test and the regularity of the failure domain measured by the standard deviation of random test results.