Identifying Experimental Incidents in Software Engineering Replications

The Empirical Software Engineering community is taking a growing interest in replicating experiments, but replications still pose a challenge for researchers. To be able to do better replications, we require detailed knowledge of what happens when a replication is performed. This article introduces a procedure for empirically evaluating replications designed to identify experimental incidents. The evaluation employs a qualitative method to analyze interviews with the principal investigator and other sources of information. We evaluated five replications of the same experiment run by different experimenters at different sites. We identified 49 incident types that occurred between 1 and 4 times across the evaluated replications. Although the replications were conducted within an experimentation network which was set up with collaborative instruments, we identified incidents in all the experimental activities. If experimenters know which incidents are likely to occur in replications, they will be able to focus on the identified problems and improve the replications that they perform. This report is a first step to create and improve experiment communication instruments such as laboratory packages.