A Further Empirical Investigation of the Relationship Between MRE and Project Size

The mean magnitude of relative error, MMRE, is the de facto standard evaluation criterion to assess the accuracy of software project prediction models. The fundamental metric of MMRE is MRE, a relative residual error. For MMRE to be a meaningful summary statistic, it is a necessary, but not sufficient, condition that MRE and project size are uncorrelated. Except for two previous conference studies done by the same authors, it has never been empirically validated that MRE and project size really are uncorrelated. In this paper, we extend the previous studies using the same data sets as before: Albrecht, Kemerer, Finnish, DMR and Accenture-ERP. Unlike the previous studies, we plot MRE against the predicted effort rather than against the actual effort and, in so doing, we obtain very different results from the previous studies. The results of this study suggest that MRE and project size are uncorrelated, which apparently is contradictory to the previous results where we found a negative correlation. The explanation for these seemingly contradictory results is presented in this study.

[1]  John E. Gaffney,et al.  Software Function, Source Lines of Code, and Development Effort Prediction: A Software Science Validation , 1983, IEEE Transactions on Software Engineering.

[2]  H. E. Dunsmore,et al.  Software engineering metrics and models , 1986 .

[3]  Chris F. Kemerer,et al.  An empirical validation of software cost estimation models , 1987, CACM.

[4]  Roderick J. A. Little,et al.  Statistical Analysis with Missing Data , 1988 .

[5]  Kari Känsälä,et al.  Inter-item correlations among function points , 1993, [1993] Proceedings First International Software Metrics Symposium.

[6]  Y. Miyazaki,et al.  Robust regression for developing software estimation models , 1994, J. Syst. Softw..

[7]  Jarrett Rosenberg,et al.  Some misconceptions about lines of code , 1997, Proceedings Fourth International Software Metrics Symposium.

[8]  Ingunn Myrtveit,et al.  Human performance estimating with analogy and regression models: an empirical validation , 1998, Proceedings Fifth International Software Metrics Symposium. Metrics (Cat. No.98TB100262).

[9]  Jarrett Rosenberg,et al.  Some misconceptions about lines of code , 1997, Proceedings Fourth International Software Metrics Symposium.

[10]  Ingunn Myrtveit,et al.  A Controlled Experiment to Assess the Benefits of Estimating with Analogy and Regression Models , 1999, IEEE Trans. Software Eng..

[11]  Stephen G. MacDonell,et al.  What accuracy statistics really measure , 2001, IEE Proc. Softw..

[12]  E. Stensrud,et al.  A Comparison of LAD and OLS Regression for Effort Prediction of Software Projects , 2001 .

[13]  E. Stensrud,et al.  MRE and Heteroscedasticity : An Empirical Validation of the Assumption of Homoscedasticity of the Magnitude of Relative Error , 2001 .

[14]  Ingunn Myrtveit,et al.  Analyzing Data Sets with Missing Data: An Empirical Evaluation of Imputation Methods and Likelihood-Based Methods , 2001, IEEE Trans. Software Eng..

[15]  Barbara A. Kitchenham,et al.  An empirical validation of the relationship between the magnitude of relative error and project size , 2002, Proceedings Eighth IEEE Symposium on Software Metrics.

[16]  Barbara A. Kitchenham,et al.  A Simulation Study of the Model Evaluation Criterion MMRE , 2003, IEEE Trans. Software Eng..

[17]  Erik Andersén.  Estimating with Enhanced Object Points vs . Function Points , 2022 .