Empirical Evaluation of Factors Affecting Distinction between Failing and Passing Executions

Information captured in software execution profiles can benefit verification activities by supporting more cost-effective fault localization and execution classification. This paper proposes an experimental design which utilizes execution information to quantify the effect of factors such as different programs and fault inclusions on the distinction between passed and failed execution profiles. For this controlled experiment we use well-known, benchmark-like programs. In addition to experimentation, our empirical evaluation includes case studies of open source programs having more complex fault models. The results show that metrics reflecting distinction between failing and passing executions are affected more by program than by faults included.