Investigating the Significance of the Bellwether Effect to Improve Software Effort Prediction: Further Empirical Study

<italic>Context:</italic> In addressing how best to estimate how much effort is required to develop software, a recent study found that using exemplary and recently completed projects [forming <italic>Bellwether moving windows </italic> (BMW)] in software effort prediction (SEP) models leads to relatively improved accuracy. More studies need to be conducted to determine whether the BMW yields improved accuracy in general, since different sizing and aging parameters of the BMW are known to affect accuracy. <italic>Objective:</italic> To investigate the existence of exemplary projects (<italic>Bellwethers</italic>) with defined window size and age parameters, and whether their use in SEP improves prediction accuracy. <italic>Method:</italic> We empirically investigate the moving window assumption based on the theory that the prediction outcome of a future event depends on the outcomes of prior events. Sampling of <italic>Bellwethers</italic> was undertaken using three introduced <italic>Bellwether methods</italic> (<italic>SSPM, SysSam,</italic> and <italic>RandSam</italic>). The ergodic Markov chain was used to determine the stationarity of the <italic>Bellwethers</italic>. <italic>Results:</italic> Empirical results show that 1) <italic>Bellwethers</italic> exist in SEP and 2) the BMW has an approximate size of 50 to 80 exemplary projects that should not be more than 2 years old relative to the <italic>new</italic> projects to be estimated. <italic>Conclusion:</italic> The study's results add further weight to the recommended use of <italic>Bellwethers</italic> for improved prediction accuracy in SEP.

[1]  Jinwu Gao,et al.  Law of Large Numbers for Uncertain Random Variables , 2016, IEEE Transactions on Fuzzy Systems.

[2]  Sousuke Amasaki,et al.  The Evaluation of Weighted Moving Windows for Software Effort Estimation , 2013, PROFES.

[3]  Sousuke Amasaki,et al.  A replication study on the effects of weighted moving windows for software effort estimation , 2016, EASE.

[4]  John W. Tukey,et al.  Exploratory data analysis , 1977, Addison-Wesley series in behavioral science : quantitative methods.

[5]  Raghu Ramakrishnan,et al.  Bellwether analysis: predicting global aggregates from local regions , 2006, VLDB.

[6]  Raghu Ramakrishnan,et al.  Bellwether analysis: Searching for cost-effective query-defined predictors in large databases , 2009, TKDD.

[7]  Bart Baesens,et al.  Data Mining Techniques for Software Effort Estimation: A Comparative Study , 2012, IEEE Transactions on Software Engineering.

[8]  Jacky W. Keung,et al.  Duplex output software effort estimation model with self-guided interpretation , 2018, Inf. Softw. Technol..

[9]  H. Keselman,et al.  Modern robust data analysis methods: measures of central tendency. , 2003, Psychological methods.

[10]  R. Dennis Cook,et al.  Detection of Influential Observation in Linear Regression , 2000, Technometrics.

[11]  Sousuke Amasaki,et al.  Towards Better Selection Between Moving Windows and Growing Portfolio , 2016, PROFES.

[12]  David J. Kupfer,et al.  Size of Treatment Effects and Their Importance to Clinical Research and Practice , 2006, Biological Psychiatry.

[13]  Tim Menzies,et al.  Transfer learning in effort estimation , 2015, Empirical Software Engineering.

[14]  J. Lachin.  Introduction to sample size determination and power analysis for clinical trials. , 1981, Controlled clinical trials.

[15]  Tim Menzies,et al.  Heterogeneous Defect Prediction , 2018, IEEE Trans. Software Eng..

[16]  Emilia Mendes,et al.  Investigating the Use of Duration-Based Moving Windows to Improve Software Effort Prediction , 2012, 2012 19th Asia-Pacific Software Engineering Conference.

[17]  Erhan Çinlar,et al.  Introduction to stochastic processes , 1974 .

[18]  M. Puri,et al.  The multivariate nonparametric Behrens–Fisher problem , 2002 .

[19]  Ying Zou,et al.  Data Transformation in Cross-project Defect Prediction , 2017, Empirical Software Engineering.

[20]  Barbara A. Kitchenham,et al.  A Simulation Study of the Model Evaluation Criterion MMRE , 2003, IEEE Trans. Software Eng..

[21]  Barry W. Boehm,et al.  Negative results for software effort estimation , 2016, Empirical Software Engineering.

[22]  Emilia Mendes,et al.  Investigating the use of moving windows to improve software effort prediction: a replicated study , 2017, Empirical Software Engineering.

[23]  K. Yuen,et al.  The two-sample trimmed t for unequal population variances , 1974 .

[24]  Xin Yao,et al.  Software effort estimation as a multiobjective learning problem , 2013, TSEM.

[25]  Jacky W. Keung,et al.  A Stratification and Sampling Model for Bellwether Moving Window , 2017, SEKE.

[26]  Jürgen Schmidhuber,et al.  Deep learning in neural networks: An overview , 2014, Neural Networks.

[27]  Akito Monden,et al.  An empirical evaluation of outlier deletion methods for analogy-based cost estimation , 2011, Promise '11.

[28]  Jeffrey C. Carver.  Towards Reporting Guidelines for Experimental Replications: A Proposal , 2010 .

[29]  Stefan Biffl,et al.  Optimal project feature weights in analogy-based cost estimation: improvement and limitations , 2006, IEEE Transactions on Software Engineering.

[30]  Andrew W. Moore,et al.  X-means: Extending K-means with Efficient Estimation of the Number of Clusters , 2000, ICML.

[31]  Elizabeth A. Peck,et al.  Introduction to Linear Regression Analysis , 2001 .

[32]  Tim Menzies,et al.  Too much automation? The bellwether effect and its implications for transfer learning , 2016, 2016 31st IEEE/ACM International Conference on Automated Software Engineering (ASE).

[33]  Stephen G. MacDonell,et al.  A Taxonomy of Data Quality Challenges in Empirical Software Engineering , 2013, 2013 22nd Australian Software Engineering Conference.

[34]  Xin Yao,et al.  journal homepage: www.elsevier.com/locate/infsof Ensembles and locality: Insight on improving software effort estimation , 2022 .

[35]  B. L. Welch.  THE SIGNIFICANCE OF THE DIFFERENCE BETWEEN TWO MEANS WHEN THE POPULATION VARIANCES ARE UNEQUAL , 1938 .

[36]  Stephen G. MacDonell,et al.  Investigating the Significance of Bellwether Effect to Improve Software Effort Estimation , 2017, 2017 IEEE International Conference on Software Quality, Reliability and Security (QRS).

[37]  Sousuke Amasaki,et al.  On the effectiveness of weighted moving windows: Experiment on linear regression based software effort estimation , 2015, J. Softw. Evol. Process..

[38]  Akito Monden,et al.  A stability assessment of solution adaptation techniques for analogy-based software effort estimation , 2017, Empirical Software Engineering.

[39]  A. O. Pittenger.  Introduction to Stochastic Processes (Erhan Çinlar) , 1977 .

[40]  N. Cliff.  Dominance statistics: Ordinal analyses to answer ordinal questions. , 1993 .

[41]  Peter A. Whigham,et al.  A Baseline Model for Software Effort Estimation , 2015, TSEM.

[42]  Mark Harman,et al.  Multi-objective Software Effort Estimation , 2016, 2016 IEEE/ACM 38th International Conference on Software Engineering (ICSE).

[43]  Lionel C. Briand,et al.  A Hitchhiker's guide to statistical tests for assessing randomized algorithms in software engineering , 2014, Softw. Test. Verification Reliab..

[44]  Mitra Mirhassani,et al.  Efficient VLSI Implementation of Neural Networks With Hyperbolic Tangent Activation Function , 2014, IEEE Transactions on Very Large Scale Integration (VLSI) Systems.

[45]  Tore Dybå,et al.  A systematic review of effect size in software engineering experiments , 2007, Inf. Softw. Technol..

[46]  Ian Goodfellow,et al.  Deep Learning with Differential Privacy , 2016, CCS.

[47]  Qiang Yang,et al.  A Survey on Transfer Learning , 2010, IEEE Transactions on Knowledge and Data Engineering.

[48]  Emilia Mendes,et al.  Further comparison of cross-company and within-company effort estimation models for Web applications , 2004 .

[49]  Elena García Barriocanal,et al.  Empirical findings on team size and productivity in software development , 2012, J. Syst. Softw..

[50]  Pearl Brereton,et al.  Robust Statistical Methods for Empirical Software Engineering , 2017, Empirical Software Engineering.

[51]  Min Xie,et al.  An empirical analysis of data preprocessing for machine learning-based software cost estimation , 2015, Inf. Softw. Technol..

[52]  Jammalamadaka.  Introduction to Linear Regression Analysis (3rd ed.) , 2003 .

[53]  Ronaldo Dias,et al.  A Review of Kernel Density Estimation with Applications to Econometrics , 2012, 1212.2812.

[54]  Tim Menzies,et al.  oftware effort models should be assessed via leave-one-out validation , 2013 .

[55]  Patrick Mair,et al.  Robust statistical methods in R using the WRS2 package , 2020, Behavior research methods.

[56]  Shari Lawrence Pfleeger,et al.  An empirical study of maintenance and development estimation accuracy , 2002, J. Syst. Softw..

[57]  Jacky W. Keung,et al.  Multi-Objective Optimization for Software Testing Effort Estimation , 2016, SEKE.

[58]  Thong Ngee Goh,et al.  A study of mutual information based feature selection for case based reasoning in software cost estimation , 2009, Expert Syst. Appl..

[59]  Emilia Mendes,et al.  Using tabu search to configure support vector regression for effort estimation , 2013, Empirical Software Engineering.

[60]  H. Fischer.  A History of the Central Limit Theorem: From Classical to Modern Probability Theory , 2010 .

[61]  David A. Belsley,et al.  Regression Analysis and its Application: A Data-Oriented Approach.@@@Applied Linear Regression.@@@Regression Diagnostics: Identifying Influential Data and Sources of Collinearity , 1981 .

[62]  Emilia Mendes,et al.  Applying moving windows to software effort estimation , 2009, 2009 3rd International Symposium on Empirical Software Engineering and Measurement.

[63]  Stephen G. MacDonell,et al.  Evaluating prediction systems in software project estimation , 2012, Inf. Softw. Technol..

[64]  Martin J. Shepperd,et al.  Estimating Software Project Effort Using Analogies , 1997, IEEE Trans. Software Eng..

[65]  Anh Tuan Nguyen,et al.  Combining Deep Learning with Information Retrieval to Localize Buggy Files for Bug Reports (N) , 2015, 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE).

[66]  Thong Ngee Goh,et al.  A study of the non-linear adjustment for analogy based software cost estimation , 2009, Empirical Software Engineering.

[67]  Rand R. Wilcox,et al.  A Comparison of Two Rank Tests for Repeated Measure Designs , 2007 .

[68]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[69]  Nian Zhang,et al.  An effective LS-SVM-based approach for surface roughness prediction in machined surfaces , 2016, Neurocomputing.

[70]  Emilia Mendes,et al.  Bayesian Network Models for Web Effort Prediction: A Comparative Study , 2008, IEEE Transactions on Software Engineering.

[71]  D. Marquardt.  An Algorithm for Least-Squares Estimation of Nonlinear Parameters , 1963 .

[72]  Geoffrey E. Hinton,et al.  Deep Learning , 2015, Nature.

[73]  Mohammad Bagher Menhaj,et al.  Training feedforward networks with the Marquardt algorithm , 1994, IEEE Trans. Neural Networks.

[74]  R. Wilcox.  Applying Contemporary Statistical Techniques , 2003 .

[75]  Thomas J. Ostrand,et al.  \{PROMISE\} Repository of empirical software engineering data , 2007 .

[76]  Song Wang,et al.  Automatically Learning Semantic Features for Defect Prediction , 2016, 2016 IEEE/ACM 38th International Conference on Software Engineering (ICSE).

[77]  T. Wright,et al.  Organizational Benchmarking Using the ISBSG Data Repository , 2001, IEEE Softw..

[78]  Emilia Mendes,et al.  Investigating the use of duration-based moving windows to improve software effort prediction: A replicated study , 2014, Inf. Softw. Technol..

[79]  Doo-Hwan Bae,et al.  On the value of outlier elimination on software effort estimation research , 2012, Empirical Software Engineering.

[80]  Tadashi Dohi,et al.  Enhancing Performance of Random Testing through Markov Chain Monte Carlo Methods , 2010, IEEE Transactions on Computers.

[81]  Yanning Zhang,et al.  Hybrid Deep Neural Network--Hidden Markov Model (DNN-HMM) Based Speech Emotion Recognition , 2013, 2013 Humaine Association Conference on Affective Computing and Intelligent Interaction.