Cost Prediction and Software Project Management

This chapter reviews the background and extent of the software project cost prediction problem. Given the importance of the topic, there has been a great deal of research activity over the past 40 years, most of which has focused on developing formal cost prediction systems. The problem is that presently there is limited evidence to suggest formal methods outperform experts, therefore detailed consideration is given to the available empirical evidence concerning expert performance. This shows that software professionals tend to be biased (optimistic) and over-confident, and there are a number of deep cognitive biases which help us understand why this is so. Finally, the chapter describes how this might best be tackled through a range of simple, practical and evidence-based methods.

[1]  Keith Phalp,et al.  An investigation of machine learning based prediction systems , 2000, J. Syst. Softw..

[2]  A. Tversky,et al.  Intuitive Prediction: Biases and Corrective Procedures , 1982 .

[3]  Magne Jørgensen,et al.  Group Processes in Software Effort Estimation , 2004, Empirical Software Engineering.

[4]  Stefano Tarantola,et al.  Sensitivity Analysis as an Ingredient of Modeling , 2000 .

[5]  Stephen G. MacDonell,et al.  What accuracy statistics really measure , 2001, IEE Proc. Softw..

[6]  Mary Shaw,et al.  Software Engineering Body of Knowledge (SWEBOK) , 2001, ICSE.

[7]  R. Buehler,et al.  Exploring the "planning fallacy": Why people underestimate their task completion times. , 1994 .

[8]  Carolyn Mair,et al.  The consistency of empirical comparisons of regression and analogy-based software project cost prediction , 2005, 2005 International Symposium on Empirical Software Engineering, 2005..

[9]  Michelle Cartwright,et al.  Predicting with Sparse Data , 2001, IEEE Trans. Software Eng..

[10]  Stephen G. MacDonell,et al.  Using prior-phase effort records for re-estimation during software projects , 2003, Proceedings. 5th International Workshop on Enterprise Networking and Computing in Healthcare Industry (IEEE Cat. No.03EX717).

[11]  Watts S. Humphrey,et al.  Introducing the personal software process , 1995, Ann. Softw. Eng..

[12]  Barbara A. Kitchenham,et al.  The question of scale economies in software - why cannot researchers agree? , 2002, Inf. Softw. Technol..

[13]  F. J. Heemstra,et al.  Software cost estimation , 1992, Inf. Softw. Technol..

[14]  A. Tversky,et al.  Judgment under Uncertainty: Heuristics and Biases , 1974, Science.

[15]  Magne Jørgensen,et al.  A review of studies on expert estimation of software development effort , 2004, J. Syst. Softw..

[16]  Qinbao Song,et al.  Missing Data Imputation Techniques , 2007, Int. J. Bus. Intell. Data Min..

[17]  Lefteris Angelis,et al.  Ranking and Clustering Software Cost Estimation Models through a Multiple Comparisons Algorithm , 2013, IEEE Transactions on Software Engineering.

[18]  Karen T. Lum,et al.  Stable rankings for different effort models , 2010, Automated Software Engineering.

[19]  Janet L. Kolodner,et al.  Case-Based Reasoning , 1989, IJCAI 1989.

[20]  Guilherme Horta Travassos,et al.  Cross versus Within-Company Cost Estimation Studies: A Systematic Review , 2007, IEEE Transactions on Software Engineering.

[21]  Qinbao Song,et al.  Predicting software project effort: A grey relational analysis based method , 2011, Expert Syst. Appl..

[22]  Khaled El Emam,et al.  Software Cost Estimation with Incomplete Data , 2001, IEEE Trans. Software Eng..

[23]  Martin Shepperd,et al.  Case-Based Reasoning and Software Engineering , 2003 .

[24]  Jae-Won Moon Reflection in Learning and Professional Development: Theory and Practice , 2005 .

[25]  Ronald L. Wasserstein,et al.  Monte Carlo: Concepts, Algorithms, and Applications , 1997 .

[26]  C. Weinstein,et al.  Self-regulated learning: The interactive influence of metacognitive awareness and goal-setting , 1992 .

[27]  Magne Jørgensen,et al.  Identification of more risks can lead to increased over-optimism of and over-confidence in software development effort estimates , 2010, Inf. Softw. Technol..

[28]  Shari Lawrence Pfleeger,et al.  An empirical study of maintenance and development estimation accuracy , 2002, J. Syst. Softw..

[29]  Paul D. Ellis,et al.  The essential guide to effect sizes : statistical power, meta-analysis, and the interpretation of research results , 2010 .

[30]  Tom DeMarco,et al.  Controlling software projects : management, measurement & estimation , 1982 .

[31]  Magne Jørgensen,et al.  The Impact of Lessons-Learned Sessions on Effort Estimation and Uncertainty Assessments , 2009, IEEE Transactions on Software Engineering.

[32]  E. GaffneyJ.,et al.  Software Function, Source Lines of Code, and Development Effort Prediction , 1983 .

[33]  Meliha Handzic,et al.  Managing Software Engineering Knowledge , 2010, Springer Berlin Heidelberg.

[34]  Carolyn Mair,et al.  A Literature Review of Expert Problem Solving using Analogy , 2009, EASE.

[35]  Tim Menzies,et al.  On the Value of Ensemble Effort Estimation , 2012, IEEE Transactions on Software Engineering.

[36]  Barbara A. Kitchenham,et al.  Estimates, Uncertainty, and Risk , 1997, IEEE Softw..

[37]  J. Armstrong Significance Tests Harm Progress in Forecasting , 2007 .

[38]  James W. Borchering,et al.  Estimeetings: Development Estimates and a Front-End Process For a Large Project , 1991, IEEE Trans. Software Eng..

[39]  Xin Yao,et al.  journal homepage: www.elsevier.com/locate/infsof Ensembles and locality: Insight on improving software effort estimation , 2022 .

[40]  Magne Jørgensen,et al.  Software effort estimation terminology: The tower of Babel , 2006, Inf. Softw. Technol..

[41]  Robert J Willis Critical path analysis and resource constrained project scheduling -- Theory and practice , 1985 .

[42]  Stephen G. MacDonell,et al.  Comparing Local and Global Software Effort Estimation Models -- Reflections on a Systematic Review , 2007, ESEM 2007.

[43]  Robert C. Mintram,et al.  Preliminary Data Analysis Methods in Software Estimation , 2005, Software Quality Journal.

[44]  Martin J. Shepperd,et al.  An experiment on software project size and effort estimation , 2003, 2003 International Symposium on Empirical Software Engineering, 2003. ISESE 2003. Proceedings..

[45]  Martin J. Shepperd,et al.  Estimating Software Project Effort Using Analogies , 1997, IEEE Trans. Software Eng..

[46]  D. Schoen The Reflective Practitioner , 1983 .

[47]  Ingunn Myrtveit,et al.  A Controlled Experiment to Assess the Benefits of Estimating with Analogy and Regression Models , 1999, IEEE Trans. Software Eng..

[48]  Mehwish Riaz,et al.  A systematic review of software maintainability prediction and metrics , 2009, ESEM 2009.

[49]  B. Kitchenham,et al.  Inter-item correlations among function points , 1993, Proceedings of 1993 15th International Conference on Software Engineering.

[50]  D. Ross Jeffery,et al.  Analogy-X: Providing Statistical Inference to Analogy-Based Software Cost Estimation , 2008, IEEE Transactions on Software Engineering.

[51]  J. Flavell Metacognition and Cognitive Monitoring: A New Area of Cognitive-Developmental Inquiry. , 1979 .

[52]  Ian H. Witten,et al.  Data mining: practical machine learning tools and techniques, 3rd Edition , 1999 .

[53]  Magne Jørgensen,et al.  How large are software cost overruns? A review of the 1994 CHAOS report , 2006, Inf. Softw. Technol..

[54]  Tim Menzies,et al.  Size doesn't matter?: on the value of software size features for effort estimation , 2012, PROMISE '12.

[55]  Barbara A. Kitchenham,et al.  A Simulation Study of the Model Evaluation Criterion MMRE , 2003, IEEE Trans. Software Eng..

[56]  M. van Genuchten,et al.  Calibrating a software cost estimation model: why and how , 1987 .

[57]  Muhammad Ali Babar,et al.  Evaluating the Usefulness and Ease of Use of a Groupware Tool for the Software Architecture Evaluation Process , 2007, ESEM 2007.

[58]  R. Buehler,et al.  Planning, personality, and prediction: The role of future focus in optimistic time predictions☆ , 2003 .

[59]  Khaled El Emam,et al.  A Replicated Survey of IT Software Project Failures , 2008, IEEE Software.

[60]  D. Kahneman,et al.  When More Pain Is Preferred to Less: Adding a Better End , 1993 .

[61]  Charles R. Symons,et al.  Function Point Analysis: Difficulties and Improvements , 1988, IEEE Trans. Software Eng..

[62]  Ronald Gulezian Reformulating and calibrating COCOMO , 1991, J. Syst. Softw..

[63]  H. D. Benington,et al.  Production of Large Computer Programs , 1983, Annals of the History of Computing.

[64]  M. Carr,et al.  “Spontaneous” strategy use: Perspectives from metacognitive theory☆ , 1987 .

[65]  Magne Jørgensen,et al.  A Systematic Review of Software Development Cost Estimation Studies , 2007 .

[66]  Donald A. Schön,et al.  Organizational Learning II: Theory, Method, and Practice , 1995 .

[67]  Stephen G. MacDonell,et al.  Combining techniques to optimize effort predictions in software project management , 2003, J. Syst. Softw..

[68]  D. Ross Jeffery,et al.  Calibrating estimation tools for software development , 1990, Softw. Eng. J..

[69]  Ayse Bener,et al.  Evaluation of Feature Extraction Methods on Software Cost Estimation , 2007, ESEM 2007.

[70]  T.C. Lethbridge,et al.  Guide to the Software Engineering Body of Knowledge (SWEBOK) and the Software Engineering Education Knowledge (SEEK) - a preliminary mapping , 2001, 10th International Workshop on Software Technology and Engineering Practice.

[71]  Barry W. Boehm,et al.  Software Engineering Economics , 1993, IEEE Transactions on Software Engineering.

[72]  Stephen G. MacDonell,et al.  Evaluating prediction systems in software project estimation , 2012, Inf. Softw. Technol..

[73]  Bent Flyvbjerg,et al.  Curbing Optimism Bias and Strategic Misrepresentation in Planning: Reference Class Forecasting in Practice , 2008 .

[74]  Forrest Shull,et al.  Local versus Global Lessons for Defect Prediction and Effort Estimation , 2013, IEEE Transactions on Software Engineering.

[75]  A. Saltelli,et al.  Sensitivity Anaysis as an Ingredient of Modeling , 2000 .

[76]  Albert L. Lederer,et al.  The Impact of the Environment on the Management of Information Systems , 1990, Inf. Syst. Res..

[77]  John E. Gaffney,et al.  Software Function, Source Lines of Code, and Development Effort Prediction: A Software Science Validation , 1983, IEEE Transactions on Software Engineering.

[78]  B. Flyvbjerg,et al.  Megaprojects and Risk: An Anatomy of Ambition , 2003 .

[79]  Magne Jørgensen,et al.  An effort prediction interval approach based on the empirical distribution of previous estimation accuracy , 2003, Inf. Softw. Technol..

[80]  Barry W. Boehm,et al.  Analyzing and handling local bias for calibrating parametric cost estimation models , 2013, Inf. Softw. Technol..

[81]  Robert T. Hughes,et al.  Expert judgement as an estimating method , 1996, Inf. Softw. Technol..

[82]  JørgensenMagne,et al.  Software effort estimation terminology , 2006 .

[83]  Stefan Wagner An Approach to Global Sensitivity Analysis: FAST on COCOMO , 2007, ESEM 2007.

[84]  S. A. Coutinho The relationship between goals, metacognition, and academic success , 2007 .

[85]  Chris F. Kemerer,et al.  An empirical validation of software cost estimation models , 1987, CACM.

[86]  Ellis Horowitz,et al.  Software Cost Estimation with COCOMO II , 2000 .

[87]  Magne Jørgensen,et al.  Better sure than safe? Over-confidence in judgement based software development effort prediction intervals , 2004, J. Syst. Softw..

[88]  Martin J. Shepperd,et al.  Comparing Software Prediction Techniques Using Simulation , 2001, IEEE Trans. Software Eng..

[89]  R. Buehler,et al.  Frequency, Probability, and Prediction: Easy Solutions to Cognitive Illusions? , 1999, Cognitive Psychology.

[90]  Magne Jørgensen,et al.  Software Development Estimation Biases: The Role of Interdependence , 2012, IEEE Transactions on Software Engineering.

[91]  Walker Royce Software Project Management , 1998 .