On Testing Effectiveness of Metamorphic Relations: A Case Study

One fundamental challenge for software testing is the oracle problem, which means that either there does not exist a mechanism (called oracle) to verify the test output given any possible program input, or it is very expensive, if not impossible, to apply the oracle. Metamorphic testing is an innovative approach to oracle problem. In metamorphic testing, metamorphic relations are derived from the innate characteristics of the software under test. These relations can help to generate test data and verify the correctness of the test result without the need of oracle. The effectiveness of metamorphic relations can play a significant role in the testing process. It has been argued that the metamorphic relations that cause different software execution behaviors should have high fault detection ability. In this paper, we conduct a case study to analyze the relationship between the execution behavior and the fault-detection effectiveness of metamorphic relations. Some code coverage criteria are used to reflect the execution behavior. It is shown that there is a certain degree of correlation between the code coverage achieved by a metamorphic relation and its fault-detection effectiveness.