Syntactic regression testing for tree-structured output

Regression testing is used by software developers to ensure that program modifications have not negatively impacted the correctness of code. While regression testing has been successfully applied in many domains, programs such as web applications, XML processors, and compilers remain expensive to test because harmless program evolutions make the tests appear to fail: in our experiments 82% of test case output differences are false positives. We present an automated tool that measures syntactic differences in the tree-structured output of such programs to reduce the number of false positives in, and thus the cost of, regression testing. We model test case outputs that merit human inspection through a set of structural and domain-specific features. We evaluate the performance of our technique on over 20,000 test case output comparisons, and find that we are three times as accurate as a naive comparator.