Effort-Aware semi-Supervised just-in-Time defect prediction

Abstract Context Software defect prediction is an important technique that can help practitioners allocate their quality assurance efforts. In recent years, just-in-time (JIT) defect prediction has attracted considerable interest, as it enables developers to identify risky changes at check-in time. Objective Many studies have conducted research from supervised and unsupervised perspectives. A model that does not rely on label information would be preferred. However, the performance of unsupervised models proposed by previous studies in the classification scenario was unsatisfactory due to the lack of supervised information. Furthermore, most supervised models fail to outperform simple unsupervised models in the ranking scenario. To overcome this weakness, we conduct research from the semi-supervised perspective that only requires a small quantity of labeled data for training. Method In this paper, we propose a semi-supervised model for JIT defect prediction named Effort-Aware Tri-Training (EATT), which is an effort-aware method using a greedy strategy to rank changes. We compare EATT with the state-of-the-art supervised and unsupervised models with respect to different labeled rate. Results The experimental results on six open-source projects demonstrate that EATT outperforms existing supervised and unsupervised models for effort-aware JIT defect prediction, and has similar or superior performance in classifying defect-inducing changes. Conclusion The results show that EATT can not only achieve high classification accuracy as supervised models, but also offer more practical value than other compared models from the perspective of the effort needed to review changes.

[1]  Andreas Zeller,et al.  Mining metrics to predict component failures , 2006, ICSE.

[2]  Beijun Shen,et al.  Software Defect Prediction Using Semi-Supervised Learning with Change Burst Information , 2016, 2016 IEEE 40th Annual Computer Software and Applications Conference (COMPSAC).

[3]  Dewayne E. Perry,et al.  Toward understanding the rhetoric of small source code changes , 2005, IEEE Transactions on Software Engineering.

[4]  Sven Apel,et al.  Views on Internal and External Validity in Empirical Software Engineering , 2015, 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering.

[5]  Zhi-Hua Zhou,et al.  Semi-supervised learning by disagreement , 2010, Knowledge and Information Systems.

[6]  Ding Yuan,et al.  How do fixes become bugs? , 2011, ESEC/FSE '11.

[7]  N. Nagappan,et al.  Use of relative code churn measures to predict system defect density , 2005, Proceedings. 27th International Conference on Software Engineering, 2005. ICSE 2005..

[8]  David J. Miller,et al.  A Mixture of Experts Classifier with Learning Based on Both Labelled and Unlabelled Data , 1996, NIPS.

[9]  Yi Zhang,et al.  Classifying Software Changes: Clean or Buggy? , 2008, IEEE Transactions on Software Engineering.

[10]  Yan Zhou,et al.  Enhancing Supervised Learning with Unlabeled Data , 2000, ICML.

[11]  David A. Landgrebe,et al.  The effect of unlabeled samples in reducing the small sample size problem and mitigating the Hughes phenomenon , 1994, IEEE Trans. Geosci. Remote. Sens..

[12]  Tim Menzies,et al.  Revisiting unsupervised learning for defect prediction , 2017, ESEC/SIGSOFT FSE.

[13]  Michele Lanza,et al.  An extensive comparison of bug prediction approaches , 2010, 2010 7th IEEE Working Conference on Mining Software Repositories (MSR 2010).

[14]  Neil D. Lawrence,et al.  Semi-supervised Learning via Gaussian Processes , 2004, NIPS.

[15]  Zoubin Ghahramani,et al.  Combining active learning and semi-supervised learning using Gaussian fields and harmonic functions , 2003, ICML 2003.

[16]  Mikhail Belkin,et al.  Manifold Regularization: A Geometric Framework for Learning from Labeled and Unlabeled Examples , 2006, J. Mach. Learn. Res..

[17]  Cha Zhang,et al.  Ensemble Machine Learning: Methods and Applications , 2012 .

[18]  Akito Monden,et al.  An analysis of developer metrics for fault prediction , 2010, PROMISE '10.

[19]  Andreas Zeller,et al.  When do changes induce fixes? , 2005, ACM SIGSOFT Softw. Eng. Notes.

[20]  Licheng Jiao,et al.  Semi-Supervised Deep Fuzzy C-Mean Clustering for Software Fault Prediction , 2018, IEEE Access.

[21]  Bernhard Schölkopf,et al.  Learning with Local and Global Consistency , 2003, NIPS.

[22]  Yuming Zhou,et al.  Code Churn: A Neglected Metric in Effort-Aware Just-in-Time Defect Prediction , 2017, 2017 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM).

[23]  Naoyasu Ubayashi,et al.  Studying just-in-time defect prediction using cross-project models , 2015, Empirical Software Engineering.

[24]  Witold Pedrycz,et al.  A comparative analysis of the efficiency of change metrics and static code attributes for defect prediction , 2008, 2008 ACM/IEEE 30th International Conference on Software Engineering.

[25]  Xiaojin Zhu,et al.  Semi-Supervised Learning , 2010, Encyclopedia of Machine Learning.

[26]  Taghi M. Khoshgoftaar,et al.  Software Quality Analysis of Unlabeled Program Modules With Semisupervised Clustering , 2007, IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans.

[27]  D. Rubin,et al.  Maximum likelihood from incomplete data via the EM - algorithm plus discussions on the paper , 1977 .

[28]  Leo Breiman,et al.  Bagging Predictors , 1996, Machine Learning.

[29]  Xinli Yang,et al.  TLEL: A two-layer ensemble learning approach for just-in-time defect prediction , 2017, Inf. Softw. Technol..

[30]  Ayse Basar Bener,et al.  A Replication Study: Just-in-Time Defect Prediction with Ensemble Learning , 2018, 2018 IEEE/ACM 6th International Workshop on Realizing Artificial Intelligence Synergies in Software Engineering (RAISE).

[31]  Ahmed E. Hassan,et al.  Predicting faults using the complexity of code changes , 2009, 2009 IEEE 31st International Conference on Software Engineering.

[32]  Lin Tan,et al.  Do time of day and developer experience affect commit bugginess? , 2011, MSR '11.

[33]  Alexander Zien,et al.  Semi-Supervised Learning , 2006 .

[34]  Thorsten Joachims,et al.  Transductive Inference for Text Classification using Support Vector Machines , 1999, ICML.

[35]  Yan Wang,et al.  Effort-aware and just-in-time defect prediction with neural network , 2019, PloS one.

[36]  Shunzhi Zhu,et al.  An improved semi-supervised learning method for software defect prediction , 2014, J. Intell. Fuzzy Syst..

[37]  Zhi-Hua Zhou,et al.  Sample-based software defect prediction with active and semi-supervised learning , 2012, Automated Software Engineering.

[38]  F. Wilcoxon Individual Comparisons by Ranking Methods , 1945 .

[39]  Tracy Hall,et al.  A Systematic Literature Review on Fault Prediction Performance in Software Engineering , 2012, IEEE Transactions on Software Engineering.

[40]  Sebastian Thrun,et al.  Text Classification from Labeled and Unlabeled Documents using EM , 2000, Machine Learning.

[41]  D. Angluin,et al.  Learning From Noisy Examples , 1988, Machine Learning.

[42]  Xiang Chen,et al.  MULTI: Multi-objective effort-aware just-in-time software defect prediction , 2018, Inf. Softw. Technol..

[43]  David Lo,et al.  Supervised vs Unsupervised Models: A Holistic Look at Effort-Aware Just-in-Time Defect Prediction , 2017, 2017 IEEE International Conference on Software Maintenance and Evolution (ICSME).

[44]  Zhi-Hua Zhou,et al.  Software defect detection with rocus , 2011 .

[45]  Xinli Yang,et al.  Deep Learning for Just-in-Time Defect Prediction , 2015, 2015 IEEE International Conference on Software Quality, Reliability and Security.

[46]  Banu Diri,et al.  Unlabelled extra data do not always mean extra performance for semi‐supervised fault prediction , 2009, Expert Syst. J. Knowl. Eng..

[47]  Claes Wohlin,et al.  Experimentation in Software Engineering , 2012, Springer Berlin Heidelberg.

[48]  Mikhail Belkin,et al.  Semi-Supervised Learning on Riemannian Manifolds , 2004, Machine Learning.

[49]  Harvey P. Siy,et al.  Predicting Fault Incidence Using Software Change History , 2000, IEEE Trans. Software Eng..

[50]  Bojan Cukic,et al.  Software defect prediction using semi-supervised learning with dimension reduction , 2012, 2012 Proceedings of the 27th IEEE/ACM International Conference on Automated Software Engineering.

[51]  Bojan Cukic,et al.  A Semi-supervised Approach to Software Defect Prediction , 2014, 2014 IEEE 38th Annual Computer Software and Applications Conference.

[52]  Mary Shaw,et al.  Experiences and results from initiating field defect prediction and product test prioritization efforts at ABB Inc. , 2006, ICSE.

[53]  Hongfang Liu,et al.  An Investigation into the Functional Form of the Size-Defect Relationship for Software Modules , 2009, IEEE Transactions on Software Engineering.

[54]  Bojan Cukic,et al.  An iterative semi-supervised approach to software fault prediction , 2011, Promise '11.

[55]  Osamu Mizuno,et al.  The impact of context metrics on just-in-time defect prediction , 2019, Empirical Software Engineering.

[56]  David Lo,et al.  Revisiting supervised and unsupervised models for effort-aware just-in-time defect prediction , 2018, Empirical Software Engineering.

[57]  Alexander Zien,et al.  Semi-Supervised Classification by Low Density Separation , 2005, AISTATS.

[58]  Taghi M. Khoshgoftaar,et al.  Software quality estimation with limited fault data: a semi-supervised learning perspective , 2007, Software Quality Journal.

[59]  Avrim Blum,et al.  The Bottleneck , 2021, Monopsony Capitalism.

[60]  Yuming Zhou,et al.  Effort-aware just-in-time defect prediction: simple unsupervised models could be better than supervised models , 2016, SIGSOFT FSE.

[61]  Zhi-Hua Zhou,et al.  Tri-training: exploiting unlabeled data using three classifiers , 2005, IEEE Transactions on Knowledge and Data Engineering.

[62]  Audris Mockus,et al.  A large-scale empirical study of just-in-time quality assurance , 2013, IEEE Transactions on Software Engineering.

[63]  Yoshua Bengio,et al.  Semi-supervised Learning by Entropy Minimization , 2004, CAP.

[64]  David Lo,et al.  The Impact of Mislabeled Changes by SZZ on Just-in-Time Defect Prediction , 2019, IEEE Transactions on Software Engineering.

[65]  Xiao-Yuan Jing,et al.  Label propagation based semi-supervised learning for software defect prediction , 2016, Automated Software Engineering.

[66]  Taghi M. Khoshgoftaar,et al.  The Detection of Fault-Prone Programs , 1992, IEEE Trans. Software Eng..

[67]  Gemma Catolino,et al.  Cross-Project Just-in-Time Bug Prediction for Mobile Apps: An Empirical Assessment , 2019, 2019 IEEE/ACM 6th International Conference on Mobile Software Engineering and Systems (MOBILESoft).

[68]  Tibor Gyimóthy,et al.  Empirical validation of object-oriented metrics on open source software for fault prediction , 2005, IEEE Transactions on Software Engineering.

[69]  Y. Benjamini,et al.  Controlling the false discovery rate: a practical and powerful approach to multiple testing , 1995 .

[70]  Audris Mockus,et al.  Predicting risk of software changes , 2000, Bell Labs Technical Journal.

[71]  Philip J. Guo,et al.  Characterizing and predicting which bugs get fixed: an empirical study of Microsoft Windows , 2010, 2010 ACM/IEEE 32nd International Conference on Software Engineering.