Combinatorial interaction testing of tangled configuration options

Traditional t-way covering arrays have been shown to be highly effective at revealing option-related failures caused by the interactions of t or fewer configuration options. We however, argue that their effectiveness suffers in the presence of complex interactions among configuration options, which from now on will be referred to as tangled options. To overcome this shortcoming, we propose an approach in which 1) the source code of the system under test is analyzed to figure out how configuration options interact with each other, 2) the analysis results are used to determine the combinations of option settings as well as the conditions under which these combinations must be tested, and 3) a “minimal” interaction test suite covering all the required combinations, is created. To evaluate the proposed approach, we conducted a set of feasibility studies on two highly configurable software systems. The results we have obtained from these studies support our basic hypothesis that traditional covering arrays suffer in the presence of tangled options and that this shortcoming can be overcome by turning CIT from a black-box approach to a gray-box approach.