Predicting Release Quality

Summary form only given. Identifying correlations between in-process development and test metrics is key in anticipating subsequent reliability performance in the field. For several years now at Cisco, our primary measure of field reliability has been Software Defects Per Million Hours (SWDPMH), and this metric has been goaled on a yearly basis for over 100 product families. A key reason SWDPMH is considered to be of critical importance is that we see a high correlation between SWDPMH and Software Customer Satisfaction (SW CSAT) over a wide spectrum of products and feature releases. Therefore it is important to try to anticipate SWDPMH for new releases before the software is released to customers, for several reasons: Early warning that a major feature release is likely to experience substantial quality problems in the field may allow for remediation of the release during, or even prior to, function and system testing Prediction of SWDPMH enables better planning for subsequent maintenance releases and rollout strategies Calculating the tradeoffs between SWDPMH and feature volume provides guidance concerning acceptable feature content, test effort, release cycle timing, and other key parameters affecting feature releases. Our efforts over the past two years have been to enhance our ability to predict SWDPMH in the field. Toward this end, we have developed predictive models, tested the models with a broad range of feature and maintenance releases, and have provided guidance to development, test, and release management teams on how to improve the chances of achieving best-in-class levels of SWDPMH. This work is ongoing, but several models are currently used in a production mode for more than 40 product families, with good results. In this paper we will show correlations with SWDPMH of feature release sequences for 16 product families. We will also show the models´ applicability to maintenance release sequences, features, and Business Unit contributions to feature rel- ases. We will also show the models´ performance characteristics with agile releases, hybrid agile/waterfall releases, and traditional waterfall releases.