Ranking software engineering measures related to reliability using expert opinion

The field of software engineering measurement appears to the unfamiliar eye as a chaotic environment lacking unifying principles and rigor. The number of software engineering measures developed over the years is stupefying and keeps increasing. Software engineering measures relate to multiple aspects of the software development process and product. Software development organizations typically select a small number of such software engineering measures to manage their development processes and products. The research presented in this paper is an attempt to help software development organizations identify the software engineering measures that are best predictors of software reliability. The research is based on the top 30 measures identified in an earlier study carried out by Lawrence Livermore National Laboratory (J.D. Lawrence et al., Technical Report UCRL-ID-136035, 1998). The set of ranking criteria was modified to fit the needs of the study. The score of each measure for each ranking criterion was elicited through expert opinion and then aggregated into a single score using multi-attribute utility theory. The basic aggregation scheme selected was a linear additive scheme. A comprehensive sensitivity analysis was carried out. The sensitivity analysis included variation of levels, variation of weights and variation of aggregation schemes