Applicability and benefits of mutation analysis as an aid for unit testing

Unit testing is a highly popular and widely practiced measure for assuring software quality. Nevertheless, writing good unit tests requires experience in test design and in applying the testing frameworks. Hence, existing unit test suites often suffer from issues that limit their defect detection capability or that impact the understandability and maintainability of the implemented tests. Several methods and techniques have been proposed to aid the developer in creating good unit tests. Mutation analysis is one of the most promising techniques to assess the quality of a test suite. Over the last years it has caught increasing attention by researchers and practitioners and a variety of tools have been developed to support this technique. In this work, mutation analysis is studied for its practical applicability and the potential benefits in providing guidance for unit testing. Five mutation analysis tools are investigated on four test suites representing different levels of test quality. The results show that the applied tools have reached an acceptable level of maturity although practical application still uncovers technical limitations. Furthermore, the study results indicate that the implemented mutation operators allow a good approximation of the actual quality of a test suite and an advantage over conventional code coverage measures when a comprehensive set of mutation operators has been implemented.