Keynote

Why do we replicate experiments?- to verify the results from the first experiment, to expand our knowledge of the discipline, to build models that can be used to predict and to be challenged. So what does it mean to replicate an experiment? What are the criteria for a replication? Physicists and sociologists experiment and replicate experiments, but their expectations are different with respect to verifiability, expansion of knowledge, precision in prediction. The difference has to do with the nature of the domain they are studying. The domain affects their ability to generate relevant and testable hypotheses, identify, control, and manipulate the context variables, supply the appropriate documentation. In all cases it requires the support of a sufficiently large community of researchers who think empirically. Maybe we should have started with the question: What´s so different/hard about experimentation in software engineering? Over a decade ago, we (Filippo Lanubile, Forrest Shull, and I) proposed a framework for building a body of software engineering knowledge by identifying key dependent and independent variables and using them to integrate collections of experiments with like hypotheses, using context variables as a way to expand upon our knowledge and address the question of results generalization. We used it to integrate the results of some reading technique studies. In this talk I will discuss that framework, what we managed to do with it, where it falls short, and what issues are still hard to address. I would like to have an interactive discussion about what is so different and hard about experimentation and replication in software engineering.