Assessing Oracle Quality with Checked Coverage

A known problem of traditional coverage metrics is that they do not assess oracle quality - that is, whether the computation result is actually checked against expectations. In this paper, we introduce the concept of checked coverage - the dynamic slice of covered statements that actually influence an oracle. Our experiments on seven open-source projects show that checked coverage is a sure indicator for oracle quality - and even more sensitive than mutation testing, its much more demanding alternative.