Adequacy, Accuracy, Scalability, and Uncertainty of Architecture-based Software Reliability: Lessons Learned from Large Empirical Case Studies

Our earlier research work on applying architecture-based software reliability models on a large scale case study allowed us to test how and when they work, to understand their limitations, and to outline the issues that need future research. In this paper we first present an additional case study which confirms our earlier findings. Then, we present uncertainty analysis of architecture-based software reliability for both case studies. The results show that Monte Carlo method scales better than the method of moments. The sensitivity analysis based on Monte Carlo method shows that (1) small number of parameters contribute to the most of the variation in system reliability and (2) given an operational profile, components´ reliabilities have more significant impact on system reliability than transition probabilities. Finally, we summarize the lessons learned from conducting large scale empirical case studies for the purpose of architecture-based reliability assessment and uncertainty analysis