On the repeatability of metric models and metrics across software builds

We have developed various software metrics models over the years: Boolean discriminate functions (BDFs); the Kolmogorov-Smirnov distance; derivative calculations for assessing achievable quality; a stopping rule; point and confidence interval estimates of quality; relative critical value deviation metrics; and nonlinear regression functions. We would like these models and metrics to be repeatable across the n builds of a software system. The advantage of repeatability is that models and metrics only need to be developed and validated once on build, and then applied n-1 times without modification to subsequent builds, with considerable savings in analysis and computational effort. In practical terms, this approach involves using the same model parameters that were validated and applying them unchanged on subsequent builds. The disadvantage is that the quality and metrics data of builds 2, ..., n, which varies across builds, is not utilized. We make a comparison of this approach with one that involves validating models and metrics on each build i and applying them only on build i+1, and then repeating the process. The advantage of this approach is that all available data are used in the models and analysis but at considerable cost in effort. We report on experiments involving large sets of discrepancy reports and metrics data on the Space Shuttle flight software, where we compare the predictive accuracy and effort of the two approaches for BDFs, critical values, derivative quality and inspection calculations, and the stopping rule