Difficulties in Running Experiments in the Software Industry: Experiences from the Trenches

Controlled experiments in laboratory settings are relatively commonplace in software engineering, but experiments in industry are thin on the ground. Of the few existing cases, most are 1-1 (running one experiment at one company), just a few are n-1 (running n experiments at one company) and still fewer are 1-n (running one and the same experiment at n companies). In this paper we report the difficulties that we experienced running the same experiment at several companies. We ran the same experiment in five different settings at three companies, and the results were transferred to each company so that they could exploit the resulting evidence in their decision-making process. We have found that: 1) it was relatively easy to get companies involved; 2) they did not cooperate as much as they had agreed to in the project proposal; 3) our industrial environments imposed many more constraints on the experimental design than laboratory environments; 4) professionals were less motivated than students; 5) the reliability of the results could be compromised by subject characteristics and behaviour; and 6) experiment findings could not be transferred using just the standard reporting guidelines that are used for scientific articles.