Compacting regression-suites on-the-fly

Testing is one of the biggest problems of the software industry. Coverage is the main technique for showing that the testing has been thorough. Coverage can be used to find a good regression suite, i.e. a set of tests that is run on the application after sofware or data changes in order to check that no new bugs were introduced. This paper is about the experience gained in IBM Haifa Research Lab (HRL) in creating regression suites and minimizing their size, while maintaining high quality as measured in coverage. The problem we solve, while similar to the one addressed in the literature, has a key difference; the compaction algorithm is implemented online due to the large number of tests processed. We compare strategies for implementing online set-cover. The trade-offs are between the solution quality (as expressed by the size of the cover), the size of the intermediate sets and the computational resources. We show that it is possible to start discarding tests very early without getting a significantly larger final set