Believing is Seeing: Confirmation Bias Studies in Software Engineering

Confirmation bias is the human tendency to search for, collect, interpret, analyse, or recall information in a way that confirms one´s prior beliefs or preferences. In this paper, we review previous research and demonstrate confirmation bias and its effect in two software engineering contexts. The first study documents that managers bias their interpretation of randomly generated project data towards confirmation of their preferred contract type. The second study reports from an analysis of the results of 35 published comparisons of regression and analogy-based cost estimation models. Twenty of these comparisons evaluate the performance of a self-developed analogy-based estimation model relative to a regression-based model and may be biased towards finding evidence confirming a better accuracy of their own model. A statistical meta-analysis of all 35 comparisons showed that the analogy-based models were more accurate than the regression-based models, whereas removing the 20 comparisons where the researchers evaluated their own models gave the opposite result. Our results support the idea that it is important to consider the presence and degree of confirmation bias when collecting information about topics where there are prior preferences, beliefs, or vested interests to avoid misleading conclusions.