The Use of Simulation Techniques for Hybrid Software Cost Estimation and Risk Analysis

Abstract Cost estimation is a crucial field for companies developing software or software‐intensive systems. Besides point estimates, effective project management also requires information about cost‐related project risks, for example, a probability distribution of project costs. One possibility to provide such information is the application of Monte Carlo simulation. However, it is not clear whether other simulation techniques exist that are more accurate or efficient when applied in this context. We investigate this question with CoBRA®, 1 a cost estimation method that applies simulation, that is, random sampling, for cost estimation. This chapter presents an empirical study, which evaluates selected sampling techniques employed within the CoBRA® method. One result of this study is that the usage of Latin Hypercube sampling can improve average simulation accuracy by 60% and efficiency by 77%. Moreover, analytical solutions are compared with sampling methods, and related work, limitations of the study, and future research directions are described. In addition, the chapter presents a comprehensive overview and comparison of existing software effort estimation methods.

[1]  Claes Wohlin,et al.  Experimentation in Software Engineering , 2000, The Kluwer International Series in Software Engineering.

[2]  Claes Wohlin,et al.  Experimentation in software engineering: an introduction , 2000 .

[3]  D. Ross Jeffery,et al.  A comparative study of two software development cost modeling techniques using multi-organizational and company-specific data , 2000, Inf. Softw. Technol..

[4]  Frank Bomarius,et al.  COBRA: a hybrid method for software cost estimation, benchmarking, and risk assessment , 1998, Proceedings of the 20th International Conference on Software Engineering.

[5]  Barry W. Boehm,et al.  Bayesian Analysis of Empirical Software Engineering Cost Models , 1999, IEEE Trans. Software Eng..

[6]  Victor R. Basili,et al.  The TAME Project: Towards Improvement-Oriented Software Environments , 1988, IEEE Trans. Software Eng..

[7]  James R. Wilson,et al.  Correlation-induction techniques for estimating quantiles in simulation experiments , 1995, WSC '95.

[8]  D. Ross Jeffery,et al.  Cost Estimation : A Review of Models , Process , and Practice , 2010 .

[9]  M. D. McKay,et al.  A comparison of three methods for selecting values of input variables in the analysis of output from a computer code , 2000 .

[10]  Amrit L. Goel,et al.  Empirical Data Modeling in Software Engineering Using Radical Basis Functions , 2000, IEEE Trans. Software Eng..

[11]  Barry W. Boehm,et al.  Software Engineering Economics , 1993, IEEE Transactions on Software Engineering.

[12]  John J. Marciniak,et al.  Encyclopedia of Software Engineering , 1994, Encyclopedia of Software Engineering.

[13]  Zhihao Chen,et al.  Validation methods for calibrating software effort models , 2005, Proceedings. 27th International Conference on Software Engineering, 2005. ICSE 2005..

[14]  Eduardo Saliby Descriptive sampling: an improvement over Latin hypercube sampling , 1997, WSC '97.

[15]  Claes Wohlin,et al.  A subjective effort estimation experiment , 1997, Inf. Softw. Technol..

[16]  D. Ross Jeffery,et al.  An Empirical Study of Analogy-based Software Effort Estimation , 1999, Empirical Software Engineering.

[17]  Ellis Horowitz,et al.  Software Cost Estimation with COCOMO II , 2000 .

[18]  M.J.I.M. van Genuchten,et al.  Chapter 9 – Calibrating a software cost-estimation model: why and how , 1991 .

[19]  Akito Monden,et al.  Effort Estimation Based on Collaborative Filtering , 2004, PROFES.

[20]  Kaushal K. Shukla,et al.  Neuro-genetic prediction of software development effort , 2000, Inf. Softw. Technol..

[21]  Donald J. Reifer SoftCost-R: User experiences and lessons learned at the age of one , 1987, J. Syst. Softw..

[22]  Abigail Holley,et al.  Palisade Corporation , 2005, WSC '05.

[23]  M. Elkjaer,et al.  Stochastic budget simulation , 2000 .

[24]  Emilia Mendes,et al.  A Comparative Study of Cost Estimation Models for Web Hypermedia Applications , 2003, Empirical Software Engineering.

[25]  Stephen G. MacDonell,et al.  Applications of fuzzy logic to software metric models for development effort estimation , 1997, 1997 Annual Meeting of the North American Fuzzy Information Processing Society - NAFIPS (Cat. No.97TH8297).

[26]  Giuliano Antoniol,et al.  Maintenance and testing effort modeled by linear and nonlinear dynamic systems , 2001, Inf. Softw. Technol..

[27]  Carolyn Mair,et al.  An Investigation of Rule Induction Based Prediction Systems , 1999 .

[28]  Witold Pedrycz,et al.  Software cost estimation with fuzzy models , 2000, SIAP.

[29]  Barry W. Boehm,et al.  Software development cost estimation approaches — A survey , 2000, Ann. Softw. Eng..

[30]  Taghi M. Khoshgoftaar,et al.  Estimating software project effort by analogy based on linguistic values , 2002, Proceedings Eighth IEEE Symposium on Software Metrics.

[31]  Sun-Jen Huang,et al.  Fuzzy Decision Tree Approach for Embedding Risk Assessment Information into Software Cost Estimation Model , 2006, J. Inf. Sci. Eng..

[32]  K. Brown,et al.  Graduate Texts in Mathematics , 1982 .

[33]  D. C. Rapaport,et al.  Book review:Monte Carlo methods. Volume I: Basics , 1987 .

[34]  David Vose,et al.  Quantitative Risk Analysis: A Guide to Monte Carlo Simulation Modelling , 1996 .

[35]  Ioannis Stamelos,et al.  Software productivity and effort prediction with ordinal regression , 2005, Inf. Softw. Technol..

[36]  I. M. Sobolʹ The Monte Carlo method , 1974 .

[37]  Leo Breiman,et al.  Classification and Regression Trees , 1984 .

[38]  Magne Jørgensen Realism in assessment of effort estimation uncertainty: it matters how you ask , 2004, IEEE Transactions on Software Engineering.

[39]  Philip M. Johnson,et al.  Empirically Guided Software Effort Guesstimation , 2000, IEEE Softw..

[40]  Eric R. Ziegel,et al.  Quantitative Risk Analysis , 1999 .

[41]  Sankaran Mahadevan,et al.  Reliability analysis of automotive body-door subsystem , 2002, Reliab. Eng. Syst. Saf..

[42]  Chun Hung Cheng,et al.  Software development cost estimation: Integrating neural network with cluster analysis , 1998, Inf. Manag..

[43]  Janusz Zalewski,et al.  Rough sets: Theoretical aspects of reasoning about data , 1996 .

[44]  Magne Jørgensen,et al.  Group Processes in Software Effort Estimation , 2004, Empirical Software Engineering.

[45]  Douglas Fisher,et al.  Machine Learning Approaches to Estimating Software Development Effort , 1995, IEEE Trans. Software Eng..

[46]  A. Owen Monte Carlo Variance of Scrambled Net Quadrature , 1997 .

[47]  James W. Borchering,et al.  Estimeetings: Development Estimates and a Front-End Process For a Large Project , 1991, IEEE Trans. Software Eng..

[48]  David Ellison,et al.  Software cost estimation using an Albus perceptron (CMAC) , 1997, Inf. Softw. Technol..

[49]  J. T. Lochner The Journal of Defense Software Engineering , 1999 .

[50]  Pierre L'Ecuyer,et al.  Efficiency improvement and variance reduction , 1994, Proceedings of Winter Simulation Conference.

[51]  Da Yang,et al.  COCOMO-U: An Extension of COCOMO II for Cost Estimation with Uncertainty , 2006, SPW/ProSim.

[52]  Michael J. Prietula,et al.  Examining the Feasibility of a Case-Based Reasoning Model for Software Effort Estimation , 1992, MIS Q..

[53]  Martin J. Shepperd,et al.  Estimating Software Project Effort Using Analogies , 1997, IEEE Trans. Software Eng..

[54]  Barbara A. Kitchenham,et al.  Empirical studies of assumptions that underlie software cost-estimation models , 1992, Inf. Softw. Technol..

[55]  A. Stroud Approximate calculation of multiple integrals , 1973 .

[56]  Gerd Gigerenzer,et al.  What we have learned (so far) , 1999 .

[57]  P. W. Garratt,et al.  A Neurofuzzy cost estimator , 1999 .

[58]  Moataz A. Ahmed,et al.  Adaptive fuzzy logic-based framework for software development effort prediction , 2005, Inf. Softw. Technol..

[59]  Colin J Burgess,et al.  Can genetic programming improve software effort estimation? A comparative evaluation , 2001, Inf. Softw. Technol..

[60]  Lawrence H. Putnam,et al.  Software Estimating Models: Three Viewpoints , 2006 .

[61]  BRACE: Bootstrap Based Analogy Cost Estimation: Automated Support for An Enhanced Effort Prediction Method , 2001 .

[62]  Barbara A. Kitchenham,et al.  A framework for evaluating a software bidding model , 2005, Inf. Softw. Technol..

[63]  Eduardo Saliby,et al.  An empirical evaluation of sampling methods in risk analysis simulation: quasi-Monte Carlo, descriptive sampling, and latin hypercube sampling , 2002, Proceedings of the Winter Simulation Conference.

[64]  Alain Abran,et al.  COCOMO cost model using fuzzy logic , 2000 .

[65]  Stephen G. MacDonell,et al.  Alternatives to regression models for estimating software projects , 1996 .

[66]  Lionel C. Briand,et al.  A replicated assessment and comparison of common software cost modeling techniques , 2000, Proceedings of the 2000 International Conference on Software Engineering. ICSE 2000 the New Millennium.

[67]  Parag C. Pendharkar,et al.  A probabilistic model for predicting software development effort , 2003, IEEE Transactions on Software Engineering.

[68]  Barbara A. Kitchenham,et al.  Modeling Software Bidding Risks , 2003, IEEE Trans. Software Eng..

[69]  Martin Fowler,et al.  Planning Extreme Programming , 2000 .

[70]  Magne Jørgensen,et al.  Experience With the Accuracy of Software Maintenance Task Effort Prediction Models , 1995, IEEE Trans. Software Eng..

[71]  Frank Tsui,et al.  Essentials of software engineering , 2006 .

[72]  Isabella Wieczorek,et al.  Resource Estimation in Software Engineering , 2002 .

[73]  H. Kraemer,et al.  Summated Rating Scale Construction An Introduction , 2009 .

[74]  Günther Ruhe,et al.  A comparative study of attribute weighting heuristics for effort estimation by analogy , 2006, ISESE '06.

[75]  Allen S. Parrish,et al.  An Empirical Study Using Task Assignment Patterns to Improve the Accuracy of Software Effort Estimation , 2001, IEEE Trans. Software Eng..

[76]  David S. Christensen,et al.  Does Calibration Improve Prediction Accuracy , 2000 .

[77]  Taghi M. Khoshgoftaar,et al.  Identification of fuzzy models of software cost estimation , 2004, Fuzzy Sets Syst..

[78]  William Marsh,et al.  Making resource decisions for software projects , 2004, Proceedings. 26th International Conference on Software Engineering.

[79]  Michelle Cartwright,et al.  Issues on the Effective Use of CBR Technology for Software Project Prediction , 2001, ICCBR.

[80]  Isabel M. Ramos,et al.  An evolutionary approach to estimating software development projects , 2001, Inf. Softw. Technol..

[81]  Qinbao Song,et al.  Using grey relational analysis to predict software effort with small data sets , 2005, 11th IEEE International Software Metrics Symposium (METRICS'05).

[82]  Michel Loève,et al.  Probability Theory I , 1977 .

[83]  Abbas Heiat,et al.  Comparison of artificial neural network and regression models for estimating software development effort , 2002, Inf. Softw. Technol..

[84]  Magne Jørgensen,et al.  A Systematic Review of Software Development Cost Estimation Studies , 2007, IEEE Transactions on Software Engineering.

[85]  Victor R. Basili,et al.  A Pattern Recognition Approach for Software Engineering Data Analysis , 1992, IEEE Trans. Software Eng..

[86]  Andrea De Lucia,et al.  Assessing effort estimation models for corrective maintenance through empirical studies , 2005, Inf. Softw. Technol..

[87]  Magne Jørgensen,et al.  Practical Guidelines for Expert-Judgment-Based Software Effort Estimation , 2005, IEEE Softw..

[88]  Building a software cost estimation model based on categorical data , 2001, Proceedings Seventh International Software Metrics Symposium.

[89]  José Javier Dolado,et al.  On the problem of the software cost function , 2001, Inf. Softw. Technol..

[90]  Jon C. Helton,et al.  Latin Hypercube Sampling and the Propagation of Uncertainty in Analyses of Complex Systems , 2002 .

[91]  Daryl Essam,et al.  Software project effort estimation using genetic programming , 2002, IEEE 2002 International Conference on Communications, Circuits and Systems and West Sino Expositions.

[92]  Kenji Yokoyama,et al.  Development of a hybrid cost estimation model in an iterative manner , 2006, ICSE.

[93]  Liang Tian,et al.  Multistage software estimation , 2003, Proceedings of the 35th Southeastern Symposium on System Theory, 2003..

[94]  Y. Miyazaki,et al.  Robust regression for developing software estimation models , 1994, J. Syst. Softw..

[95]  T. Capers Jones,et al.  Estimating software costs , 1998 .

[96]  Martin J. Shepperd,et al.  Comparing Software Prediction Techniques Using Simulation , 2001, IEEE Trans. Software Eng..

[97]  W. Feller,et al.  An Introduction to Probability Theory and Its Application. , 1951 .

[98]  Ioannis Stamelos,et al.  Managing uncertainty in project portfolio cost estimation , 2001, Inf. Softw. Technol..

[99]  Stephen G. MacDonell Establishing relationships between specification size and software process effort in CASE environments , 1997, Inf. Softw. Technol..

[100]  A. Rollett,et al.  The Monte Carlo Method , 2004 .

[101]  Isabella Wieczorek Improved Software Cost Estimation – A Robust and Interpretable Modelling Method and a Comprehensive Empirical Investigation , 2004, Empirical Software Engineering.

[102]  Barbara A. Kitchenham,et al.  A Procedure for Analyzing Unbalanced Datasets , 1998, IEEE Trans. Software Eng..

[103]  Jing Ren,et al.  A neuro-fuzzy model for software cost estimation , 2003, Third International Conference on Quality Software, 2003. Proceedings..

[104]  Michelle Cartwright,et al.  Predicting with Sparse Data , 2001, IEEE Trans. Software Eng..