An empirical study of the correlation between code coverage and reliability estimation

Existing time-domain models for software reliability often result in an overestimation, of such reliability because they do not take the nature of testing techniques into account. Since every testing technique has a limit to its ability to reveal faults in a given system, as a technique approaches its saturation region fewer faults are discovered and reliability growth phenomena are predicted from the models. When the software is turned over to field operation, significant overestimates of reliability are observed. We present a technique to solve this problem by addressing both time and coverage measures for the prediction of software failures. Our technique uses coverage information collected during testing to extract only effective data from a given operational profile. Execution time between test cases which neither increase coverage nor cause a failure as reduced by a parameterized factor. Experiments using this technique were conducted on a program created in a simulation environment with simulated faults and on an industrial automatic flight control project which contained several natural faults. Results from both experiments indicate that overestimation of reliability is reduced significantly using our technique. This new approach not only helps reliability growth models make more accurate predictions, but also reveals the efficiency of a testing profile so that more effective testing techniques can be conducted