Fuzzy Analogy Based Effort Estimation: An Empirical Comparative Study

Software Development Effort Estimation (SDEE) plays a primary role in software project management. Among several techniques suggested for estimating software development effort, analogy-based software effort estimation approaches stand out as promising techniques.In this paper, the performance of Fuzzy Analogy is compared with that of six other SDEE techniques (Linear Regression, Support Vector Regression, Multi-Layer Perceptron, M5P and Classical Analogy). The evaluation of the SDEE techniques was performed over seven datasets with two evaluation techniques (All-in and Jackknife). The first step of the evaluation aimed to ensure that the SDEE techniques outperformed random guessing by using the Standardized Accuracy (SA). Then, we used a set of reliable performance measures (Pred(0.25), MAE, MBRE, MIBRE and LSD) and Borda count to rank them and identify which techniques are the most accurate.The results suggest that when using All-in evaluation, Fuzzy Analogy statistically outperformed the other SDEE techniques regardless of the dataset used. However, when using Jackknife evaluation, the results obtained depended on the dataset and the SDEE technique used. The results suggest that Fuzzy Analogy is a promising technique for software development effort estimation.

[1]  Martin Shepperd,et al.  Experiences Using Case-Based Reasoning to Predict Software Project Effort , 2000 .

[2]  Thomas J. Ostrand,et al.  \{PROMISE\} Repository of empirical software engineering data , 2007 .

[3]  Tim Menzies,et al.  oftware effort models should be assessed via leave-one-out validation , 2013 .

[4]  Mohammad Azzeh,et al.  Software effort estimation based on optimized model tree , 2011, Promise '11.

[5]  Francisco Herrera,et al.  A taxonomy for the crossover operator for real‐coded genetic algorithms: An experimental study , 2003, Int. J. Intell. Syst..

[6]  Tim Menzies,et al.  On the Value of Ensemble Effort Estimation , 2012, IEEE Transactions on Software Engineering.

[7]  Magne Jørgensen,et al.  A Systematic Review of Software Development Cost Estimation Studies , 2007, IEEE Transactions on Software Engineering.

[8]  Yong Hu,et al.  Systematic literature review of machine learning based software development effort estimation models , 2012, Inf. Softw. Technol..

[9]  Ali Idri,et al.  Software cost estimation by classical and Fuzzy Analogy for Web Hypermedia Applications: A replicated study , 2013, 2013 IEEE Symposium on Computational Intelligence and Data Mining (CIDM).

[10]  Tim Menzies,et al.  Transfer learning in effort estimation , 2015, Empirical Software Engineering.

[11]  Stephen G. MacDonell,et al.  What accuracy statistics really measure , 2001, IEE Proc. Softw..

[12]  Adriano Lorena Inácio de Oliveira,et al.  Estimation of software project effort with support vector regression , 2006, Neurocomputing.

[13]  Leandro L. Minku,et al.  An empirical evaluation of ensemble adjustment methods for analogy-based effort estimation , 2015, J. Syst. Softw..

[14]  Ioannis Stamelos,et al.  A Simulation Tool for Efficient Analogy Based Cost Estimation , 2000, Empirical Software Engineering.

[15]  Bernhard Schölkopf,et al.  A tutorial on support vector regression , 2004, Stat. Comput..

[16]  Gerardo Beni,et al.  A Validity Measure for Fuzzy Clustering , 1991, IEEE Trans. Pattern Anal. Mach. Intell..

[17]  Alain Abran,et al.  Investigating soft computing in case-based reasoning for software cost estimation , 2002 .

[18]  Alain Abran,et al.  Analogy-based software development effort estimation: A systematic mapping and review , 2015, Inf. Softw. Technol..

[19]  Martin J. Shepperd,et al.  Software project economics: a roadmap , 2007, Future of Software Engineering (FOSE '07).

[20]  Ali Idri,et al.  Software Cost Estimation Models Using Radial Basis Function Neural Networks , 2007, IWSM/Mensura.

[21]  James C. Bezdek,et al.  A Convergence Theorem for the Fuzzy ISODATA Clustering Algorithms , 1980, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[22]  Ali Idri,et al.  SOFTWARE COST ESTIMATION BY FUZZY ANALOGY FOR ISBSG REPOSITORY , 2012 .

[23]  Alain Abran,et al.  COCOMO cost model using fuzzy logic , 2000 .

[24]  Elena García Barriocanal,et al.  Software cost estimation with fuzzy inputs: Fuzzy modelling and aggregation of cost drivers , 2005, Kybernetika.

[25]  D. Lewis,et al.  Quantitative methods in psychology , 1950 .

[26]  Alain Abran,et al.  Missing data techniques in analogy-based software development effort estimation , 2016, J. Syst. Softw..

[27]  Ayse Basar Bener,et al.  A new perspective on data homogeneity in software cost estimation: a study in the embedded systems domain , 2010, Software Quality Journal.

[28]  Jacky W. Keung,et al.  Software Development Cost Estimation Using Analogy: A Review , 2009, 2009 Australian Software Engineering Conference.

[29]  Janez Demsar,et al.  Statistical Comparisons of Classifiers over Multiple Data Sets , 2006, J. Mach. Learn. Res..

[30]  Barbara A. Kitchenham,et al.  A Simulation Study of the Model Evaluation Criterion MMRE , 2003, IEEE Trans. Software Eng..

[31]  Mohammad Azzeh,et al.  A hybrid model for estimating software project effort from Use Case Points , 2016, Appl. Soft Comput..

[32]  Alain Abran,et al.  Software cost estimation by fuzzy analogy for web hypermedia applications , 2006 .

[33]  Alain Abran,et al.  Evaluating software project similarity by using linguistic quantifier guided aggregations , 2001, Proceedings Joint 9th IFSA World Congress and 20th NAFIPS International Conference (Cat. No. 01TH8569).

[34]  Heinz Mühlenbein,et al.  Predictive Models for the Breeder Genetic Algorithm I. Continuous Parameter Optimization , 1993, Evolutionary Computation.

[35]  Corinna Cortes,et al.  Support-Vector Networks , 1995, Machine Learning.

[36]  Alain Abran,et al.  Software Development Effort estimation using Classical and fuzzy Analogy: a Cross-Validation Comparative Study , 2014, Int. J. Comput. Intell. Appl..

[37]  Ricardo Massa Ferreira Lima,et al.  GA-based method for feature selection and parameters optimization for machine learning regression applied to software effort estimation , 2010, Inf. Softw. Technol..

[38]  Alain Abran,et al.  A fuzzy logic based set of measures for software project similarity: validation and possible improvements , 2001, Proceedings Seventh International Software Metrics Symposium.

[39]  Jacob Cohen,et al.  Quantitative Methods in Psychology , 1938, Nature.

[40]  T. Wright,et al.  Organizational Benchmarking Using the ISBSG Data Repository , 2001, IEEE Softw..

[41]  Martin J. Shepperd,et al.  Estimating Software Project Effort Using Analogies , 1997, IEEE Trans. Software Eng..

[42]  Magne Jørgensen,et al.  A Systematic Review of Software Development Cost Estimation Studies , 2007 .

[43]  Jacob Cohen,et al.  A power primer. , 1992, Psychological bulletin.

[44]  Xin Yao,et al.  Can cross-company data improve performance in software effort estimation? , 2012, PROMISE '12.

[45]  Stephen G. MacDonell,et al.  Evaluating prediction systems in software project estimation , 2012, Inf. Softw. Technol..