On the Value of Ensemble Effort Estimation

Background: Despite decades of research, there is no consensus on which software effort estimation methods produce the most accurate models. Aim: Prior work has reported that, given M estimation methods, no single method consistently outperforms all others. Perhaps rather than recommending one estimation method as best, it is wiser to generate estimates from ensembles of multiple estimation methods. Method: Nine learners were combined with 10 preprocessing options to generate 9 × 10 = 90 solo methods. These were applied to 20 datasets and evaluated using seven error measures. This identified the best n (in our case n = 13) solo methods that showed stable performance across multiple datasets and error measures. The top 2, 4, 8, and 13 solo methods were then combined to generate 12 multimethods, which were then compared to the solo methods. Results: 1) The top 10 (out of 12) multimethods significantly outperformed all 90 solo methods. 2) The error rates of the multimethods were significantly less than the solo methods. 3) The ranking of the best multimethod was remarkably stable. Conclusion: While there is no best single effort estimation method, there exist best combinations of such effort estimation methods.

[1]  Daniel Ryan Baker,et al.  A Hybrid Approach to Expert and Model Based Effort Estimation , 2007 .

[2]  Emilia Mendes,et al.  A Comparative Study of Cost Estimation Models for Web Hypermedia Applications , 2003, Empirical Software Engineering.

[3]  John E. Gaffney,et al.  Software Function, Source Lines of Code, and Development Effort Prediction: A Software Science Validation , 1983, IEEE Transactions on Software Engineering.

[4]  Katrina D. Maxwell,et al.  Applied Statistics for Software Managers , 2002 .

[5]  Guilherme Horta Travassos,et al.  Cross versus Within-Company Cost Estimation Studies: A Systematic Review , 2007, IEEE Transactions on Software Engineering.

[6]  Emilia Mendes,et al.  How effective is Tabu search to configure support vector regression for effort estimation? , 2010, PROMISE '10.

[7]  Ron Kohavi,et al.  A Study of Cross-Validation and Bootstrap for Accuracy Estimation and Model Selection , 1995, IJCAI.

[8]  C. Kaynak,et al.  Techniques for Combining Multiple Learners , 1998 .

[9]  Klaus Nordhausen,et al.  The Elements of Statistical Learning: Data Mining, Inference, and Prediction, Second Edition by Trevor Hastie, Robert Tibshirani, Jerome Friedman , 2009 .

[10]  Emilia Mendes,et al.  Why comparative effort prediction studies may be invalid , 2009, PROMISE '09.

[11]  Karen T. Lum,et al.  Stable rankings for different effort models , 2010, Automated Software Engineering.

[12]  Y. Miyazaki,et al.  Robust regression for developing software estimation models , 1994, J. Syst. Softw..

[13]  Barbara A. Kitchenham,et al.  A Simulation Study of the Model Evaluation Criterion MMRE , 2003, IEEE Trans. Software Eng..

[14]  Suku Nair,et al.  A Model for Software Development Effort and Cost Estimation , 1997, IEEE Trans. Software Eng..

[15]  Giovanni Seni,et al.  Ensemble Methods in Data Mining: Improving Accuracy Through Combining Predictions , 2010, Ensemble Methods in Data Mining.

[16]  Sargur N. Srihari,et al.  Decision Combination in Multiple Classifier Systems , 1994, IEEE Trans. Pattern Anal. Mach. Intell..

[17]  Jiri Matas,et al.  On Combining Classifiers , 1998, IEEE Trans. Pattern Anal. Mach. Intell..

[18]  Ayse Basar Bener,et al.  A new perspective on data homogeneity in software cost estimation: a study in the embedded systems domain , 2010, Software Quality Journal.

[19]  Leo Breiman,et al.  Random Forests , 2001, Machine Learning.

[20]  Martin J. Shepperd,et al.  Comparing Software Prediction Techniques Using Simulation , 2001, IEEE Trans. Software Eng..

[21]  A. R. Venkatachalam.  Software cost estimation using artificial neural networks , 1993, Proceedings of 1993 International Conference on Neural Networks (IJCNN-93-Nagoya, Japan).

[22]  Emilia Mendes,et al.  Further investigation into the use of CBR and stepwise regression to predict development effort for Web hypermedia applications , 2002, Proceedings International Symposium on Empirical Software Engineering.

[23]  L. Breiman,et al.  Submodel selection and evaluation in regression. The X-random case , 1992 .

[24]  Ekrem Kocaguneli,et al.  Combining Multiple Learners Induced on Multiple Datasets for Software Effort Prediction , 2009 .

[25]  Eric Bauer,et al.  An Empirical Comparison of Voting Classification Algorithms: Bagging, Boosting, and Variants , 1999, Machine Learning.

[26]  Robert Tibshirani,et al.  The Elements of Statistical Learning: Data Mining, Inference, and Prediction, 2nd Edition , 2001, Springer Series in Statistics.

[27]  Barbara A. Kitchenham,et al.  Effort estimation using analogy , 1996, Proceedings of IEEE 18th International Conference on Software Engineering.

[28]  D. Ruppert.  The Elements of Statistical Learning: Data Mining, Inference, and Prediction , 2004 .

[29]  Kamal A. Ali,et al.  On the Link between Error Correlation and Error Reduction in Decision Tree Ensembles , 1995 .

[30]  Thong Ngee Goh,et al.  A study of project selection and feature weighting for analogy based software cost estimation , 2009, J. Syst. Softw..

[31]  Yoav Freund,et al.  A decision-theoretic generalization of on-line learning and an application to boosting , 1995, EuroCOLT.

[32]  Claes Wohlin,et al.  Distribution patterns of effort estimations , 2004, Proceedings. 30th Euromicro Conference, 2004..

[33]  Jacky W. Keung.  Theoretical Maximum Prediction Accuracy for Analogy-Based Software Cost Estimation , 2008, 2008 15th Asia-Pacific Software Engineering Conference.

[34]  Taghi M. Khoshgoftaar,et al.  Software quality analysis by combining multiple projects and learners , 2008, Software Quality Journal.

[35]  Martin J. Shepperd,et al.  Estimating Software Project Effort Using Analogies , 1997, IEEE Trans. Software Eng..

[36]  Ingunn Myrtveit,et al.  Reliability and validity in comparative studies of software prediction models , 2005, IEEE Transactions on Software Engineering.

[37]  Bora Caglayan,et al.  Prest: An Intelligent Software Metrics Extraction, Analysis and Defect Prediction Tool , 2009, SEKE.

[38]  Christian Bird,et al.  The inductive software engineering manifesto: principles for industrial data mining , 2011, MALETS '11.

[39]  Robin M. Hogarth,et al.  A note on aggregating opinions , 1978 .

[40]  Magne Jørgensen,et al.  A Systematic Review of Software Development Cost Estimation Studies , 2007, IEEE Transactions on Software Engineering.

[41]  Vadlamani Ravi,et al.  Software cost estimation using computational intelligence techniques , 2009, 2009 World Congress on Nature & Biologically Inspired Computing (NaBIC).

[42]  Chin-Liang Chang,et al.  Finding Prototypes For Nearest Neighbor Classifiers , 1974, IEEE Transactions on Computers.

[43]  T. Hughes,et al.  Signals and systems , 2006, Genome Biology.

[44]  Tim Menzies,et al.  2CEE, A TWENTY FIRST CENTURY EFFORT ESTIMATION METHODOLOGY , 2008 .

[45]  Ekrem Kocaguneli,et al.  Xiruxe: An intelligent fault tracking tool , 2009 .

[46]  Martin Shepperd,et al.  On configuring a case-based reasoning software project prediction system , 2000 .

[47]  Howard Kunreuther,et al.  Pricing Insurance and Warranties: Ambiguity and Correlated Risks , 1992 .

[48]  Magne Jørgensen,et al.  Practical Guidelines for Expert-Judgment-Based Software Effort Estimation , 2005, IEEE Softw..

[49]  Colin Robson,et al.  Real World Research: A Resource for Social Scientists and Practitioner-Researchers , 1993 .

[50]  K Atkinson,et al.  The use of function points to find cost analogies , 1994 .

[51]  Thomas G. Dietterich.  Multiple Classifier Systems , 2000, Lecture Notes in Computer Science.

[52]  Karen T. Lum,et al.  Selecting Best Practices for Effort Estimation , 2006, IEEE Transactions on Software Engineering.

[53]  Jack P. C. Kleijnen,et al.  Sensitivity analysis and related analyses: A review of some statistical techniques , 1997 .

[54]  Mahil Carr,et al.  Software Cost Estimation using Soft Computing Approaches , 2010 .

[55]  Jack P. C. Kleijnen.  Sensitivity analysis and related analysis : A survey of statistical techniques , 1995 .

[56]  Barry W. Boehm,et al.  Software Engineering Economics , 1993, IEEE Transactions on Software Engineering.

[57]  Tim Menzies,et al.  Sharing experiments using open‐source software , 2011, Softw. Pract. Exp..

[58]  Horst Bunke,et al.  Feature selection algorithms for the generation of multiple classifier systems and their application to handwritten word recognition , 2004 .

[59]  Michelle Cartwright,et al.  Predicting with Sparse Data , 2001, IEEE Trans. Software Eng..

[60]  Yue Jiang,et al.  Cost Curve Evaluation of Fault Prediction Models , 2008, 2008 19th International Symposium on Software Reliability Engineering (ISSRE).

[61]  Chris F. Kemerer,et al.  An empirical validation of software cost estimation models , 1987, CACM.

[62]  Heejun Park,et al.  An empirical validation of a neural network model for software effort estimation , 2008, Expert Syst. Appl..

[63]  Wei-Yin Loh,et al.  Classification and regression trees , 2011, WIREs Data Mining Knowl. Discov..

[64]  Tim Menzies,et al.  oftware effort models should be assessed via leave-one-out validation , 2013 .

[65]  J. Scott Armstrong,et al.  Principles of forecasting , 2001 .

[66]  Lionel C. Briand,et al.  Resource modeling in software engineering , 2002 .

[67]  Kurt Hornik,et al.  Multilayer feedforward networks are universal approximators , 1989, Neural Networks.

[68]  Joydeep Ghosh,et al.  Multiclassifier Systems: Back to the Future , 2002, Multiple Classifier Systems.

[69]  Leo Breiman,et al.  Bagging Predictors , 1996, Machine Learning.

[70]  George Wright,et al.  The Delphi technique as a forecasting tool: issues and analysis , 1999 .

[71]  Taghi M. Khoshgoftaar,et al.  Enhancing software quality estimation using ensemble-classifier based noise filtering , 2005, Intell. Data Anal..

[72]  J. Armstrong.  Significance Tests Harm Progress in Forecasting , 2007 .

[73]  Janez Demsar,et al.  Statistical Comparisons of Classifiers over Multiple Data Sets , 2006, J. Mach. Learn. Res..

[74]  Barry W. Boehm,et al.  Software development cost estimation approaches — A survey , 2000, Ann. Softw. Eng..

[75]  Bhekisipho Twala,et al.  Ensemble of missing data techniques to improve software prediction accuracy , 2006, ICSE.

[76]  Magne Jørgensen,et al.  A review of studies on expert estimation of software development effort , 2004, J. Syst. Softw..

[77]  Ayse Basar Bener,et al.  ENNA: software effort estimation using ensemble of neural networks with associative memory , 2008, SIGSOFT '08/FSE-16.

[78]  Sudha Ram,et al.  Constrained cascade generalization of decision trees , 2004, IEEE Transactions on Knowledge and Data Engineering.

[79]  Ekrem Kocaguneli,et al.  A Ranking Stability Indicator for Selecting the Best Effort Estimator in Software Cost Estimation , 2011 .

[80]  Ware Myers,et al.  Measures for Excellence: Reliable Software on Time, Within Budget , 1991 .

[81]  Kari Känsälä,et al.  Inter-item correlations among function points , 1993, [1993] Proceedings First International Software Metrics Symposium.

[82]  Ayse Basar Bener,et al.  BITS: Issue Tracking and Project Management Tool in Healthcare Software Development , 2009, SEKE.