A Multistage Sequential Test Allocation for Software Reliability Estimation

We propose a method to determine how to sequentially allocate test cases among partitions of the software to minimize the expected loss incurred by the Bayes estimator of the overall reliability when the total number of software test cases is fixed. In contrast to fixed sampling schemes, where the proportion of test cases taken from each partition is determined before reliability testing begins, we make allocation decisions dynamically throughout the testing process. Using a fully Bayesian approach, we can take advantage of information from previous functional testing and insights from developers. We then refine these estimates in an iterative manner as we sample. We also compare the results from a multistage sampling scheme with the optimal fixed sampling scheme, and demonstrate its superiority in terms of the expected loss incurred when the overall reliability is estimated by its Bayes estimator both theoretically and through Monte Carlo simulations.