Study of the relationship of bug consistency with respect to performance of spectra metrics

In practice, manual debugging to locate bugs is a daunting and time-consuming task. By using software fault localization, we can reduce this time substantially. The technique of software fault localization can be performed using execution profiles of the software under several test inputs. Such profiles, known as program spectra, consist of the coverage of correct and incorrect executions statement from a given test suite. We have performed a systematic evaluation of several metrics that make use of measurement obtained from program spectra on Siemens test suite. In this paper, we discuss how the effectiveness of various metrics degrade in determining buggy statements as the bug consistency (error detection accuracy, q_e) of a statement approaches zero. Bug consistency of a statement refers to the ratio of the number of failed tests executing the statement over the total number of tests executing the statement. We proposed effect(M) as to measure the effectiveness of these metrics as q_e value varies. We also demonstrate that the q_e (previously not considered as a metric), is just as effective as some of the metrics proposed. We also formally prove that q_e is identical to the metric that Tarantula system uses for bug localization.