Large empirical case study of architecture-based software reliability

In this paper we present an empirical study of architecture-based software reliability based on a large open source application which consists of 350,000 lines of C code. The goals of our study are to analyze empirically the adequacy, applicability, and accuracy of architecture-based software reliability models. For this purpose we developed innovative approaches to efficiently extract and more accurately analyze a large amount of empirical data. Applying the theoretical results on a large scale field study allows us to test how and when they work, to understand their limitations, and outline the issues that need attention in the future research studies. Thus, our results show that for a subset of failures which can clearly be attributed to single components, both the composite and hierarchical models are very accurate when compared to the actual reliability. However, the assumptions made by the existing architecture-based software reliability models do not allow accounting for the remaining failures which led to fixing faults in multiple components. These results show that in order to progress further, software reliability engineering should go through cycles of building models, testing them empirically, learning from the experiments, and refining the models to capture the newly discovered phenomena