On the Effectiveness of the Tarantula Fault Localization Technique for Different Fault Classes

Unlike test generation techniques, spectrum-based fault localization techniques have not been rigorously evaluated for their effectiveness in localizing different classes of faults. In this paper, we evaluate the effectiveness of the Tarantula fault localization technique. We state that the following three properties of a fault affect the effectiveness of localizing it: (1) accessibility, (2) original state failure condition, and (3) impact. Accessibility refers to how easy or hard it is to execute a faulty statement. It is measured by the size of the backward slice of the faulty statement. The original state failure condition is the condition that must be satisfied to create a local failure state upon executing the faulty statement. Impact refers to the fraction of the program that is affected by the execution of the faulty statement, measured by the size of the forward slice of the faulty statement. The results of our evaluation with the Siemens benchmark suite show that (1) original state failure condition based fault classes have no relationship with the effectiveness of localization, and (2) faults that are hard to access and have low impact are most effectively localized. These observations are consistent across random and branch coverage based test suites.