Comparative studies of the model evaluation criterions mmre and pred in software cost estimation research

Software cost model research results depend on model accuracy criteria such as MMRE and PRED. Despite criticism, MMRE has emerged as the de facto standard criterion. Many alternatives have been proposed and studied, surprisingly however PRED, the second most popular criterion, has not been extensively studied. This work attempts to fill this gap in the literature and expand the understanding and use of evaluation criterion in general. The majority of this work is empirically based, applying MMRE and PRED to a number of COCOMO model variations with respect to a simulated data set and four publicly available cost estimation data sets. We replicate a number of results based on MMRE and extend them to PRED. We study qualities of MMRE and PRED as sample estimator statistics for parameters of a cost model error distribution. Standard error is used to ensure greater confidence in replicated and new results based on sample data.

[1]  Lionel C. Briand,et al.  A replicated assessment and comparison of common software cost modeling techniques , 2000, Proceedings of the 2000 International Conference on Software Engineering. ICSE 2000 the New Millennium.

[2]  Martin J. Shepperd,et al.  Estimating Software Project Effort Using Analogies , 1997, IEEE Trans. Software Eng..

[3]  Magne Jørgensen,et al.  Experience With the Accuracy of Software Maintenance Task Effort Prediction Models , 1995, IEEE Trans. Software Eng..

[4]  Jairus Hihn.  Studies in software cost model behavior: do we really understand cost model performance , 2005 .

[5]  Ingunn Myrtveit,et al.  Reliability and validity in comparative studies of software prediction models , 2005, IEEE Transactions on Software Engineering.

[6]  Martin J. Shepperd.  Evaluating Software Project Prediction Systems , 2005, IEEE METRICS.

[7]  Thomas J. Ostrand,et al.  \{PROMISE\} Repository of empirical software engineering data , 2007 .

[8]  Magne Jørgensen,et al.  How much does a vacation cost?: or what is a software cost estimate? , 2003, SOEN.

[9]  Kjetil Molkken,et al.  A Review of Surveys on Software Effort Estimation , 2003 .

[10]  Karen T. Lum,et al.  Selecting Best Practices for Effort Estimation , 2006, IEEE Transactions on Software Engineering.

[11]  Guilherme Horta Travassos,et al.  Cross versus Within-Company Cost Estimation Studies: A Systematic Review , 2007, IEEE Transactions on Software Engineering.

[12]  Barbara A. Kitchenham,et al.  A Simulation Study of the Model Evaluation Criterion MMRE , 2003, IEEE Trans. Software Eng..

[13]  Tim Menzies,et al.  Validation methods for calibrating software effort models , 2005, ICSE.

[14]  David Hinkley,et al.  Bootstrap Methods: Another Look at the Jackknife , 2008 .

[15]  R. Larsen,et al.  An introduction to mathematical statistics and its applications (2nd edition) , by R. J. Larsen and M. L. Marx. Pp 630. £17·95. 1987. ISBN 13-487166-9 (Prentice-Hall) , 1987, The Mathematical Gazette.

[16]  H. E. Dunsmore,et al.  Software engineering metrics and models , 1986 .

[17]  Magne Jørgensen,et al.  An analysis of data sets used to train and validate cost prediction systems , 2005, PROMISE '05.

[18]  Lionel C. Briand,et al.  An assessment and comparison of common software cost estimation modeling techniques , 1999, Proceedings of the 1999 International Conference on Software Engineering (IEEE Cat. No.99CB37002).

[19]  Barry W. Boehm,et al.  Software Engineering Economics , 1993, IEEE Transactions on Software Engineering.

[20]  Martin Shepperd,et al.  Using Simulation to Evaluate Prediction Techniques , 2001 .

[21]  E. Ziegel,et al.  Bootstrapping: A Nonparametric Approach to Statistical Inference , 1993 .

[22]  C. Land,et al.  Confidence Intervals for Linear Functions of the Normal Mean and Variance , 1971 .

[23]  Isabella Wieczorek,et al.  Resource Estimation in Software Engineering , 2002 .

[24]  Isabella Wieczorek,et al.  How valuable is company-specific data compared to multi-company data for software cost estimation? , 2002, Proceedings Eighth IEEE Symposium on Software Metrics.

[25]  Stephen G. MacDonell,et al.  What accuracy statistics really measure , 2001, IEE Proc. Softw..

[26]  Magne Jørgensen,et al.  A Systematic Review of Software Development Cost Estimation Studies , 2007, IEEE Transactions on Software Engineering.