Intelligent Test Oracle Construction for Reactive Systems without Explicit Specifications

A test oracle is a mechanism that decides whether an SUT (software under test) fails or passes a test case. We investigate how to use machine learning techniques to automatically construct test oracles for reactive programs without reliance on explicit specifications. Firstly, we present a library, called In TOL (Intelligent Test Oracle Library), for the convenient and flexible collection of test traces. We can flexibly use either user guidance or program assertions to collect verdicts to test traces. Such verdicts are used as supervisory signals to the supervised learning algorithm (SLA) for a test oracle. Secondly, we present several sets of feature variables for the temporal relation among events in test traces of unbounded lengths. Then we present procedures that convert test traces into feature vectors, train an SLA with the feature vectors and their verdicts, and use the trained SLA as a test oracle. We report the implementation of In TOL on top of SVM (support vector machine). We experiment with two open-source benchmark SUTs from the internet to check the performance of our techniques. Our experiment data shows that high-accuracy test verdicts can be achieved with our test oracles for the benchmark SUTs.