A method for analyzing system state-space coverage within a t-wise testing framework

Inadequate state-space coverage of complex configurable systems during test phases is an area of concern for systems engineers. Determining the state-space coverage of a proposed or executed test suite traditionally involves qualitative assessment, rendering meaningful comparative analysis between tests for a given system or across multiple systems difficult. We propose a method for assessing state-space coverage of a test suite utilizing t-wise testing, a combinatorial technique borrowed from the software testing community which generalizes pair-wise testing. We refine traditional notions of a t-wise test suite to analyze the configuration coverage of a test plan. This provides a methodology and a set of metrics to assess both the level and the distribution of state-space coverage. We detail a proof-of-concept experiment using this partial t-wise coverage framework to analyze Integration and Test (I&T) data from three separate NASA spacecraft.