Estimating software fault content before coding

The standard software development process consists of multiple stages: requirements, design, coding, system test, first office, and finally delivery. An objective of this process is to minimize the number of faults in delivered code. Root cause analysis shows that many of the faults can be traced back to requirements or design faults. As part of the software development process, reviews are conducted to remove these faults before the requirements or design document is passed on to the next step. We have developed a method of instrumenting a review process to record document faults discovered by reviewers during their preparation. Then, using statistical techniques related to capture-recapture methods, we estimate the number of undiscovered faults remaining in the document. The key idea to our method is to look at how many common faults independent reviewers find and then extrapolate to die total number of faults. We do not seed the document with artificial faults - no additional faults are introduced. We have applied our methods to 13 review sessions (either feature requirements or feature design) and are in the process of a longitudinal study tracing these features. Our results to date estimate that about 20% of the faults are undetected by reviews. When the predicted number of undetected faults is greater than 20%, consideration should be given to reworking design and/or rereviewing the result. One surprising by-product of this study is a quantification of the number of faults found by group reviews. We find that only about 10% of the (discovered) document faults are found at the review (90% are found in preparation) and that the lead time to schedule a review is about ten working days.