Validation of Models and Tests for Constrained Combinatorial Interaction Testing

In Combinatorial Interaction Testing, models specify a set of parameters with associated domains, and some constraints over the parameters. Test generation tools produce, starting from these models, test suites for achieving some given coverage criteria. The validation of both the models and the produced test suites is a worthwhile activity. Validating the models permits to early discover possible defects in them, to feed the test generations tools with good inputs and, possibly, to improve the quality of the testing process. Validating the produced test suites, instead, permits to check if the test generation tools are correct and to judge their quality. This paper proposes to validate the models by checking that the constraints are consistent, that there is no constraint implied by the other constraints, and that the parameters and their values are really necessary. The proposed test suite validation, instead, consists in checking that the tests respect the type definitions and the constraints, that all the test requirements are covered, and that all the tests in the test suite are valid and necessary. For every error we propose a possible technique able to identify the potential causes and to suggest fixes for those problems. Experiments show that the targeted defects are widespread both in benchmark and real-life models.