Assessing the Impact of Using Fault Prediction in Industry

Software developers and testers need realistic ways to measure the practical effects of using fault prediction models to guide software quality improvement methods such as testing, code reviews, and refactoring. Will the availability of fault predictions lead to discovery of different faults, or to more efficient means of finding the same faults? Or do fault predictions have no practical impact at all? In this challenge paper we describe the difficulties of answering these questions, and the issues involved in devising meaningful ways to assess the impact of using prediction models. We present several experimental design options and discuss the pros and cons of each.