Multi-objective Software Effort Estimation

We introduce a bi-objective effort estimation algorithm that combines Confidence Interval Analysis and assessment of Mean Absolute Error. We evaluate our proposed algorithm on three different alternative formulations, baseline comparators and current state-of-the-art effort estimators applied to five real-world datasets from the PROMISE repository, involving 724 different software projects in total. The results reveal that our algorithm outperforms the baseline, state-of-the-art and all three alternative formulations, statistically significantly (p

[1]  Building a software cost estimation model based on categorical data , 2001, Proceedings Seventh International Software Metrics Symposium.

[2]  Steve McConnell Software Estimation: Demystifying the Black Art , 2006 .

[3]  Mark Harman,et al.  Transformed Vargha-Delaney Effect Size , 2015, SSBSE.

[4]  Barbara A. Kitchenham,et al.  A Simulation Study of the Model Evaluation Criterion MMRE , 2003, IEEE Trans. Software Eng..

[5]  Lionel C. Briand,et al.  Identifying optimal trade-offs between CPU time usage and temporal constraints using search , 2014, ISSTA 2014.

[6]  Magne Jørgensen,et al.  A review of studies on expert estimation of software development effort , 2004, J. Syst. Softw..

[7]  Yuanyuan Zhang,et al.  Search-based software engineering: Trends, techniques and applications , 2012, CSUR.

[8]  Thomas J. Ostrand,et al.  \{PROMISE\} Repository of empirical software engineering data , 2007 .

[9]  Kalyanmoy Deb,et al.  A fast and elitist multiobjective genetic algorithm: NSGA-II , 2002, IEEE Trans. Evol. Comput..

[10]  Magne Jørgensen,et al.  A Systematic Review of Software Development Cost Estimation Studies , 2007, IEEE Transactions on Software Engineering.

[11]  Ioannis Stamelos,et al.  Multinomial Logistic Regression Applied on Software Productivity Prediction , 2003 .

[12]  Ayse Basar Bener,et al.  A comparative study for estimating software development effort intervals , 2011, Software Quality Journal.

[13]  Ioannis Stamelos,et al.  On the use of Bayesian belief networks for the prediction of software productivity , 2003, Inf. Softw. Technol..

[14]  Tim Menzies,et al.  On the Value of Ensemble Effort Estimation , 2012, IEEE Transactions on Software Engineering.

[15]  Xin Yao,et al.  Software Module Clustering as a Multi-Objective Search Problem , 2011, IEEE Transactions on Software Engineering.

[16]  M. Barros An analysis of the effects of composite objectives in multiobjective software module clustering , 2012, GECCO '12.

[17]  Barbara A. Kitchenham,et al.  A Further Empirical Investigation of the Relationship Between MRE and Project Size , 2003, Empirical Software Engineering.

[18]  Colin J Burgess,et al.  Can genetic programming improve software effort estimation? A comparative evaluation , 2001, Inf. Softw. Technol..

[19]  Ioannis Stamelos,et al.  A Simulation Tool for Efficient Analogy Based Cost Estimation , 2000, Empirical Software Engineering.

[20]  Mark Harman,et al.  Search Based Software Engineering: Techniques, Taxonomy, Tutorial , 2010, LASER Summer School.

[21]  Magne Jørgensen,et al.  The Ignorance of Confidence Levels in Minimum-Maximum Software Development Effort Intervals , 2014 .

[22]  Magne Jørgensen,et al.  Comments on ‘A Simulation Tool for Efficient Analogy Based Cost Estimation’, by L. Angelis and I. Stamelos, published in Empirical Software Engineering, 5, 35–68 (2000) , 2002, Empirical Software Engineering.

[23]  Karen T. Lum,et al.  Selecting Best Practices for Effort Estimation , 2006, IEEE Transactions on Software Engineering.

[24]  Michelle Cartwright,et al.  On Building Prediction Systems for Software Engineers , 2000, Empirical Software Engineering.

[25]  Michelle Cartwright,et al.  Issues on the Effective Use of CBR Technology for Software Project Prediction , 2001, ICCBR.

[26]  Silvio Romero de Lemos Meira,et al.  Software Effort Estimation using Machine Learning Techniques with Robust Confidence Intervals , 2007, 7th International Conference on Hybrid Intelligent Systems (HIS 2007).

[27]  Ioannis Stamelos,et al.  Managing uncertainty in project portfolio cost estimation , 2001, Inf. Softw. Technol..

[28]  Günther Ruhe,et al.  Search Based Software Engineering , 2013, Lecture Notes in Computer Science.

[29]  Barbara Kitchenham,et al.  A comparison of cross-company and within-company effort estimation models for Web applications , 2004, ICSE 2004.

[30]  Keith Phalp,et al.  An investigation of machine learning based prediction systems , 2000, J. Syst. Softw..

[31]  Lionel C. Briand,et al.  A Hitchhiker's guide to statistical tests for assessing randomized algorithms in software engineering , 2014, Softw. Test. Verification Reliab..

[32]  Emilia Mendes,et al.  Using tabu search to configure support vector regression for effort estimation , 2013, Empirical Software Engineering.

[33]  Adam Trendowicz,et al.  Software Project Effort Estimation , 2014, Springer International Publishing.

[34]  Daniel Port,et al.  Comparative studies of the model evaluation criterions mmre and pred in software cost estimation research , 2008, ESEM '08.

[35]  Sun-Jen Huang,et al.  Optimization of analogy weights by genetic algorithm for software effort estimation , 2006, Inf. Softw. Technol..

[36]  Stephen G. MacDonell,et al.  Evaluating prediction systems in software project estimation , 2012, Inf. Softw. Technol..

[37]  Emilia Mendes,et al.  A Comparative Study of Cost Estimation Models for Web Hypermedia Applications , 2003, Empirical Software Engineering.

[38]  Martin J. Shepperd,et al.  Estimating Software Project Effort Using Analogies , 1997, IEEE Trans. Software Eng..

[39]  John R. Koza,et al.  Genetic programming (videotape): the movie , 1992 .

[40]  Magne Jørgensen,et al.  Combination of software development effort prediction intervals: why, when and how? , 2002, SEKE '02.

[41]  Lothar Thiele,et al.  A Tutorial on the Performance Assessment of Stochastic Multiobjective Optimizers , 2006 .

[42]  DebK.,et al.  A fast and elitist multiobjective genetic algorithm , 2002 .

[43]  Christopher J. Lokan,et al.  What should you optimize when building an estimation model? , 2005, 11th IEEE International Software Metrics Symposium (METRICS'05).

[44]  Tim Menzies,et al.  Size doesn't matter?: on the value of software size features for effort estimation , 2012, PROMISE '12.

[45]  Adam Trendowicz,et al.  Software Project Effort Estimation: Foundations and Best Practice Guidelines for Success , 2014 .

[46]  Emilia Mendes,et al.  Further investigation into the use of CBR and stepwise regression to predict development effort for Web hypermedia applications , 2002, Proceedings International Symposium on Empirical Software Engineering.

[47]  Filomena Ferrucci,et al.  Genetic Programming for Effort Estimation: An Analysis of the Impact of Different Fitness Functions , 2010, 2nd International Symposium on Search Based Software Engineering.

[48]  Mark Harman,et al.  Not going to take this anymore: Multi-objective overtime planning for Software Engineering projects , 2013, 2013 35th International Conference on Software Engineering (ICSE).

[49]  Jacob Cohen Statistical Power Analysis for the Behavioral Sciences , 1969, The SAGE Encyclopedia of Research Design.

[50]  Ioannis Stamelos,et al.  Software productivity and effort prediction with ordinal regression , 2005, Inf. Softw. Technol..

[51]  Tim Menzies,et al.  Active learning and effort estimation: Finding the essential content of software effort estimation data , 2013, IEEE Transactions on Software Engineering.

[52]  Emilia Mendes,et al.  Investigating Tabu Search for Web Effort Estimation , 2010, 2010 36th EUROMICRO Conference on Software Engineering and Advanced Applications.

[53]  Mark Harman,et al.  Exact Mean Absolute Error of Baseline Predictor, MARP0 , 2016, Inf. Softw. Technol..

[54]  Filomena Ferrucci,et al.  How Multi-Objective Genetic Programming Is Effective for Software Development Effort Estimation? , 2011, SSBSE.

[55]  G. W. Hill,et al.  Algorithm 396: Student's t-quantiles , 1970, CACM.

[56]  Wei-Yin Loh,et al.  Classification and regression trees , 2011, WIREs Data Mining Knowl. Discov..

[57]  Martin Shepperd,et al.  Case-Based Reasoning and Software Engineering , 2003 .

[58]  H. E. Dunsmore,et al.  Software engineering metrics and models , 1986 .

[59]  Mark Harman,et al.  Search-Based Software Project Management , 2014, Software Project Management in a Changing World.

[60]  Tim Menzies,et al.  Special issue on repeatable results in software engineering prediction , 2012, Empirical Software Engineering.

[61]  Xin Yao,et al.  Software effort estimation as a multiobjective learning problem , 2013, TSEM.

[62]  Emilia Mendes,et al.  How effective is Tabu search to configure support vector regression for effort estimation? , 2010, PROMISE '10.

[63]  Durga L. Shrestha,et al.  Machine learning approaches for estimation of prediction interval for the model output , 2006, Neural Networks.

[64]  D. Ross Jeffery,et al.  A comparative study of two software development cost modeling techniques using multi-organizational and company-specific data , 2000, Inf. Softw. Technol..

[65]  Yuanyuan Zhang,et al.  Achievements, Open Problems and Challenges for Search Based Software Testing , 2015, 2015 IEEE 8th International Conference on Software Testing, Verification and Validation (ICST).

[66]  Derek Rayside,et al.  Comparison of exact and approximate multi-objective optimization for software product lines , 2014, SPLC.

[67]  José Javier Dolado,et al.  A Validation of the Component-Based Method for Software Size Estimation , 2000, IEEE Trans. Software Eng..

[68]  Lothar Thiele,et al.  Multiobjective evolutionary algorithms: a comparative case study and the strength Pareto approach , 1999, IEEE Trans. Evol. Comput..

[69]  Marco Laumanns,et al.  Performance assessment of multiobjective optimizers: an analysis and review , 2003, IEEE Trans. Evol. Comput..

[70]  Peter A. Whigham,et al.  A Baseline Model for Software Effort Estimation , 2015, TSEM.

[71]  Aurora Trinidad Ramirez Pozo,et al.  Search Based Software Engineering: Review and analysis of the field in Brazil , 2013, J. Syst. Softw..

[72]  Kjetil Moløkken-Østvold,et al.  A review of software surveys on software effort estimation , 2003, 2003 International Symposium on Empirical Software Engineering, 2003. ISESE 2003. Proceedings..

[73]  Bruce McMillin,et al.  Software engineering: What is it? , 2018, 2018 IEEE Aerospace Conference.

[74]  Vidroha Debroy,et al.  Genetic Programming , 1998, Lecture Notes in Computer Science.

[75]  Ayse Basar Bener,et al.  AI-Based Models for Software Effort Estimation , 2010, 2010 36th EUROMICRO Conference on Software Engineering and Advanced Applications.

[76]  Martin J. Shepperd,et al.  Using Genetic Programming to Improve Software Effort Estimation Based on General Data Sets , 2003, GECCO.

[77]  Ross Jeffery,et al.  A comparative Study of Cost Modelling Techniques using Public Domain multi-organisational and company-specific Data , 2000 .

[78]  Lionel C. Briand,et al.  Modeling Development Effort in Object-Oriented Systems Using Design Properties , 2001, IEEE Trans. Software Eng..

[79]  Ioannis Stamelos,et al.  Software Cost Prediction with Predefined Interval Estimates , 2004 .

[80]  B. Kitchenham,et al.  Case Studies for Method and Tool Evaluation , 1995, IEEE Softw..

[81]  Isabella Wieczorek,et al.  Resource Estimation in Software Engineering , 2002 .

[82]  Kjetil Molkken,et al.  A Review of Surveys on Software Effort Estimation , 2003 .

[83]  References , 1971 .

[84]  Marcel Korte,et al.  Confidence in software cost estimation results based on MMRE and PRED , 2008, PROMISE '08.

[85]  Ayse Basar Bener,et al.  Exploiting the Essential Assumptions of Analogy-Based Effort Estimation , 2012, IEEE Transactions on Software Engineering.

[86]  Stephen G. MacDonell,et al.  What accuracy statistics really measure , 2001, IEE Proc. Softw..

[87]  J. Royston An Extension of Shapiro and Wilk's W Test for Normality to Large Samples , 1982 .

[88]  Filomena Ferrucci,et al.  Single and Multi Objective Genetic Programming for software development effort estimation , 2012, SAC '12.

[89]  Daryl Essam,et al.  Software project effort estimation using genetic programming , 2002, IEEE 2002 International Conference on Communications, Circuits and Systems and West Sino Expositions.

[90]  Bora Caglayan,et al.  Experiences on Developer Participation and Effort Estimation , 2011, 2011 37th EUROMICRO Conference on Software Engineering and Advanced Applications.

[91]  Filomena Ferrucci,et al.  Using Tabu Search to Estimate Software Development Effort , 2009, IWSM/Mensura.

[92]  Magne Jørgensen,et al.  An effort prediction interval approach based on the empirical distribution of previous estimation accuracy , 2003, Inf. Softw. Technol..

[93]  Martin Shepperd,et al.  Using Simulation to Evaluate Prediction Techniques , 2001 .

[94]  Emilia Mendes,et al.  Further comparison of cross-company and within-company effort estimation models for Web applications , 2004 .

[95]  Magne Jørgensen,et al.  Better sure than safe? Over-confidence in judgement based software development effort prediction intervals , 2004, J. Syst. Softw..

[96]  Mark Harman,et al.  The Current State and Future of Search Based Software Engineering , 2007, Future of Software Engineering (FOSE '07).

[97]  Jerffeson Teixeira de Souza,et al.  Ten Years of Search Based Software Engineering: A Bibliometric Analysis , 2011, SSBSE.