Title :
A controlled experiment to assess the benefits of estimating with analogy and regression models
Author :
Myrtveit, Ingunn ; Stensrud, Erik
Author_Institution :
Norwegian Sch. of Manage., Sandvika, Norway
Abstract :
To have general validity, empirical results must converge. To be credible, an experimental science must understand the limitations and be able to explain the disagreements of empirical results. We describe an experiment to replicate previous studies which claim that estimation by analogy outperforms regression models. In the experiment, 68 experienced practitioners each estimated a project from a dataset of 48 industrial COTS projects. We applied two treatments, an analogy tool and a regression model, and we used the estimating performance when aided by the historical data as the control. We found that our results do not converge with previous results. The reason is that previous studies have used other datasets and partially different data analysis methods, and last but not least, the tools have been validated in isolation from the tool users. This implies that the results are sensitive to the experimental design: the characteristics of the dataset, the norms for removing outliers and other data points from the original dataset, the test metrics, significance levels, and the use of human subjects and their level of expertise. Thus, neither our results nor previous results are robust enough to claim any general validity.
Keywords :
project management; software cost estimation; statistical analysis; analogy tool; commercial off-the-shelf software projects; controlled experiment; data points; dataset; empirical results; enterprise resource planning; estimating performance; estimation by analogy; experienced practitioners; experimental design; experimental science; historical data; human performance; human subjects; industrial COTS projects; multivariate regression analysis; outliers; partially different data analysis methods; regression model; regression models; significance levels; software cost estimation; test metrics; tool users; Convergence; Costs; Data analysis; Design for experiments; Enterprise resource planning; Humans; Physics; Robustness; Software performance; Testing;
Journal_Title :
Software Engineering, IEEE Transactions on