Measurement and analysis: what can and does go wrong?

By now, one can point to many instances where measurement has been used effectively to inform management and technical decisions in support of the development and maintenance of software and software intensive systems. Yet, measurement is not very well integrated into software or systems engineering education or practice, and measurement remains challenging for all too many organizations. For this paper, we have analyzed 1350 findings drawn from 663 Software CMM® appraisals that were conducted between 1987 and 2002 inclusive. The results are augmented by questions from a survey of CIO´s from state and local governments and the private sector. Our analyses suggest several areas where both managers and engineers would benefit from better guidance about the proper use of measurement and analysis. Future work may include lexical analyses based on natural language processing as well as studies of appraiser understanding of the measurement content in CMMI® models.