A Partial Replication with a Sample Size of One: A Smoke Test for Empirical Software Engineering

Performing a full external replication can be a hugely expensive undertaking. The aim of this study was to determine what could be reliably established by performing a partial replication with a single competent developer making good use of available tools. The developer was not constrained by artificial time limits and was directed to make good use of the Jumble tool for automated mutation testing and the Model Unit framework for automated state-based testing. The Java-based software under test (SUT) was Cruise Control, a car simulation system. The patterns of mutation kills and reasons for missed kills are compared with the patterns and reasons reported in the original experimental study which involved four groups of students who manually generated test cases. We conclude that it can be meaningful to have a single competent developer perform a partial replication as long as a checking team checks and, where necessary, revises the work of the developer. The approach can be viewed as a smoke test for empirical software engineering.