Towards Cost-Effective Oracles

Oracles are the key of the success and, at the same time, the bottleneck of automated software testing. Today we can generate enormous amount of test cases and reach impressive code coverage, but the effectiveness of test cases is limited by the availability of test oracles that can distinguish failing executions. Simple implicit oracles come at negligible cost, but can reveal only few categories of often uninteresting failures. Semantically relevant test oracles can be extremely effective in revealing subtle failures, but come at high cost. In this paper we outline the dimensions of the problem, highlight the open research areas, and suggest how to produce a new generation of cost-effective oracles.