Benchmarking Technical Quality of Software Products

To enable systematic comparison of technical quality of (groups of) software products, we have collected measurement data of a wide range of systems into a benchmark repository. The measurements were taken over the course of several years of delivering software assessment services to corporations and public institutes. The granularity of the collected data follows the layered structure of a model for software product quality, based on the ISO/IEC 9126 international standard, which we developed previously. In this paper, we describe the design of our benchmark repository, and we explain how it can be used to perform comparisons of systems. To provide a concrete illustration of the concept without revealing confidential data, we use a selection of open source systems as example.