Software Effort Estimation Using a Neural Network Ensemble

Accurate software effort estimation is crucial for software consulting organizations to stay competitive in their software development costs and retain customers. Artificial Neural Network (ANN) is an effective tool to obtain accurate effort estimates. In this paper, software effort estimation models using Artificial Neural Network (ANN) ensembles and regression analysis are developed based on data collected from 163 software development projects. The main emphasis of the paper is in developing an effective experimental design to achieve superior effort estimation results. In addition, we compare the software effort estimation of ANNs and multiple regression analysis. We found two interesting results. First, variables other than size (function points) are not especially helpful in predicting software development effort. Second, a properly designed ANN ensemble significantly outperforms estimation using regression analysis and can achieve better effort estimate predictions.

[1]  Chun Hung Cheng,et al.  Software development cost estimation: Integrating neural network with cluster analysis , 1998, Inf. Manag..

[2]  Nancy L. Martin,et al.  IS Project Management: Size, Complexity, Practices and the Project Management Office , 2005, Proceedings of the 38th Annual Hawaii International Conference on System Sciences.

[3]  Mark N. Frolick,et al.  It Project Risk Factors: The Project Management Professionals Perspective , 2007, J. Comput. Inf. Syst..

[4]  Colin J Burgess,et al.  Can genetic programming improve software effort estimation? A comparative evaluation , 2001, Inf. Softw. Technol..

[5]  Lionel C. Briand,et al.  The impact of design properties on development cost in object-oriented systems , 2001, Proceedings Seventh International Software Metrics Symposium.

[6]  Kaushal K. Shukla,et al.  Neuro-genetic prediction of software development effort , 2000, Inf. Softw. Technol..

[7]  Sun-Jen Huang,et al.  Applying fuzzy neural network to estimate software development effort , 2009, Applied Intelligence.

[8]  Jean-Marc Desharnais,et al.  A comparison of software effort estimation techniques: Using function points with neural networks, case-based reasoning and regression models , 1997, J. Syst. Softw..

[9]  Oleksandr Makeyev,et al.  Neural network with ensembles , 2010, The 2010 International Joint Conference on Neural Networks (IJCNN).

[10]  Ralf Kneuper.  CMMI: Improving Software and Systems Development Processes Using Capability Maturity Model Integration , 2009 .

[11]  Yong Hu,et al.  Systematic literature review of machine learning based software development effort estimation models , 2012, Inf. Softw. Technol..

[12]  June M. Verner,et al.  State of the practice: An exploratory analysis of schedule estimation and software project success prediction , 2007, Inf. Softw. Technol..

[13]  Anders Krogh,et al.  Neural Network Ensembles, Cross Validation, and Active Learning , 1994, NIPS.

[14]  Lawrence H. Putnam,et al.  A General Empirical Solution to the Macro Software Sizing and Estimating Problem , 1978, IEEE Transactions on Software Engineering.

[15]  Kurt Hornik,et al.  Multilayer feedforward networks are universal approximators , 1989, Neural Networks.

[16]  Ioannis Stamelos,et al.  On the use of Bayesian belief networks for the prediction of software productivity , 2003, Inf. Softw. Technol..

[17]  Barry W. Boehm,et al.  Software development cost estimation approaches — A survey , 2000, Ann. Softw. Eng..

[18]  Manish Agrawal,et al.  Software Effort, Quality, and Cycle Time: A Study of CMM Level 5 Projects , 2007, IEEE Transactions on Software Engineering.

[19]  Berkman Sahiner,et al.  Classifier Performance Estimation Under the Constraint of a Finite Sample Size: Resampling Schemes Applied to Neural Network Classifiers , 2007, IJCNN.

[20]  Kevin L. Priddy,et al.  Artificial Neural Networks: An Introduction (SPIE Tutorial Texts in Optical Engineering, Vol. TT68) , 2005 .

[21]  A. S. M. Sajeev,et al.  A Vector-Based Approach to Software Size Measurement and Effort Estimation , 2001, IEEE Trans. Software Eng..

[22]  Barry W. Boehm,et al.  Bayesian Analysis of Empirical Software Engineering Cost Models , 1999, IEEE Trans. Software Eng..

[23]  Emilia Mendes,et al.  A comparison of development effort estimation techniques for Web hypermedia applications , 2002, Proceedings Eighth IEEE Symposium on Software Metrics.

[24]  V. Vemuri,et al.  Artificial neural networks: an introduction , 1988 .

[25]  Parag C. Pendharkar,et al.  A Probabilistic Model for Predicting Software Development Effort , 2003, ICCSA.

[26]  A.A. Minai,et al.  Effect of noise on the performance of the temporally-sequenced intelligent block-matching and motion-segmentation algorithm , 2004, 2004 IEEE International Joint Conference on Neural Networks (IEEE Cat. No.04CH37541).

[27]  C. Jones,et al.  By popular demand: Software estimating rules of thumb , 1996, Computer.

[28]  Guoqiang Peter Zhang,et al.  Avoiding Pitfalls in Neural Network Research , 2007, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[29]  Barry W. Boehm,et al.  Cost models for future software life cycle processes: COCOMO 2.0 , 1995, Ann. Softw. Eng..

[30]  Stephen G. MacDonell,et al.  A comparison of techniques for developing predictive models of software metrics , 1997, Inf. Softw. Technol..

[31]  Shari Lawrence Pfleeger,et al.  Software Metrics : A Rigorous and Practical Approach , 1998 .

[32]  Richard E. Fairley.  Recent advances in software estimation techniques , 1992, International Conference on Software Engineering.

[33]  K. Nakayama,et al.  Classification of multi-frequency signals with random noise using multilayer neural networks , 1993, Proceedings of 1993 International Conference on Neural Networks (IJCNN-93-Nagoya, Japan).

[34]  Stephen G. MacDonell,et al.  Factors systematically associated with errors in subjective estimates of software development effort: the stability of expert judgment , 1999, Proceedings Sixth International Software Metrics Symposium (Cat. No.PR00403).

[35]  Li-Wei Chen,et al.  Integration of the grey relational analysis with genetic algorithm for software effort estimation , 2008, Eur. J. Oper. Res..

[36]  Heejun Park,et al.  An empirical validation of a neural network model for software effort estimation , 2008, Expert Syst. Appl..

[37]  José Demisio Simões da Silva,et al.  An investigation of artificial neural networks based prediction systems in software project management , 2008, J. Syst. Softw..

[38]  Parag C. Pendharkar,et al.  Connectionist Models for Learning, Discovering, and Forecasting Software Effort: An Empirical Study , 2002, J. Comput. Inf. Syst..

[39]  Mohamed S. Kamel,et al.  Decision fusion in neural network ensembles , 2001, IJCNN'01. International Joint Conference on Neural Networks. Proceedings (Cat. No.01CH37222).

[40]  Douglas Fisher,et al.  Machine Learning Approaches to Estimating Software Development Effort , 1995, IEEE Trans. Software Eng..

[41]  Sheldon Gruber,et al.  Interpolation characteristics and noise sensitivity of neural network based inspection of machined surfaces , 1990, Proceedings., IEEE International Conference on Robotics and Automation.

[42]  Abbas Heiat,et al.  Comparison of artificial neural network and regression models for estimating software development effort , 2002, Inf. Softw. Technol..

[43]  Magne Jørgensen,et al.  A review of studies on expert estimation of software development effort , 2004, J. Syst. Softw..

[44]  Magne Jørgensen,et al.  Experience With the Accuracy of Software Maintenance Task Effort Prediction Models , 1995, IEEE Trans. Software Eng..

[45]  J. Michael Pearson,et al.  Is Project Management: Size, Practices and the Project Management Office12 , 2007, J. Comput. Inf. Syst..

[46]  Cuauhtémoc López Martín,et al.  Applying a Feedforward Neural Network for Predicting Software Development Effort of Short-Scale Projects , 2010, 2010 Eighth ACIS International Conference on Software Engineering Research, Management and Applications.

[47]  Steven A. Morris,et al.  Exploration of Information System Process Improvements and Firm Performance , 2008, J. Comput. Inf. Syst..

[48]  John E. Gaffney,et al.  Software Function, Source Lines of Code, and Development Effort Prediction: A Software Science Validation , 1983, IEEE Transactions on Software Engineering.

[49]  Norman E. Fenton,et al.  Software metrics: successes, failures and new directions , 1999, J. Syst. Softw..

[50]  Michael Perrone,et al.  Putting It All Together: Methods for Combining Neural Networks , 1993, NIPS.

[51]  Agostino Di Ciaccio,et al.  Computational Statistics and Data Analysis Measuring the Prediction Error. a Comparison of Cross-validation, Bootstrap and Covariance Penalty Methods , 2022 .

[52]  Gavin R. Finnie,et al.  Estimating software development effort with connectionist models , 1997, Inf. Softw. Technol..

[53]  G. R. Finnie,et al.  AI tools for software development effort estimation , 1996, Proceedings 1996 International Conference Software Engineering: Education and Practice.

[54]  Albert L. Lederer,et al.  An Agency Theory Explanation of Project Success , 2011, J. Comput. Inf. Syst..

[55]  N. B. Chang,et al.  Grey input-output analysis and its application for environmental cost allocation , 2003, Eur. J. Oper. Res..

[56]  Barry W. Boehm,et al.  Software Engineering Economics , 1993, IEEE Transactions on Software Engineering.

[57]  T. Capers Jones,et al.  Estimating software costs , 1998 .

[58]  Martin J. Shepperd,et al.  Comparing Software Prediction Techniques Using Simulation , 2001, IEEE Trans. Software Eng..

[59]  Michael J. Prietula,et al.  Software-Effort Estimation: An Exploratory Study of Expert Performance , 1991, Inf. Syst. Res..

[60]  Michael J. Prietula,et al.  Examining the Feasibility of a Case-Based Reasoning Model for Software Effort Estimation , 1992, MIS Q..