Estimating the failure rate of evolving software systems

When software is tested according to the customer´s operational profile, many techniques exist in the literature for using the resulting failure data to estimate the customer-perceived failure rate. Unfortunately, the overhead associated with defining and using the customer´s operational profile during system test can be significant. As a compromise, many projects test the software by executing it over a sustained period of time using simulated load scenarios that represent informed judgments of what the customer environment is likely to be. By itself, the load test data is suspect for estimating the customer-perceived failure rate since the load scenarios may not be accurate representations of the customer´s usage patterns. In addition, the load test data is limited by the relatively small amount of time that is set aside for load testing. In many cases the software being tested is an evolution of a previous release already in the field. The field experience with the previous release along with a characterization of the difference between the two releases are natural sources of information for building a prior distribution of the failure rate of the new release. The load test data can then be used to obtain a posterior distribution of the failure rate of the new release. The posterior distribution inherently reflects the customer´s operational profile through its dependence on the field data, and effectively enlarges the set of data from which inference about the failure rate of the new release is to be made. We develop a model for the prior distribution of the failure rate, and obtain the corresponding posterior distribution. We illustrate our modeling approach using real data collected from one of our software products