Empirical Evaluation of Hunk Metrics as Bug Predictors

Reducing the number of bugs is a crucial issue during software development and maintenance. Software process and product metrics are good indicators of software complexity. These metrics have been used to build bug predictor models to help developers maintain the quality of software. In this paper we empirically evaluate the use of hunk metrics as predictor of bugs. We present a technique for bug prediction that works at smallest units of code change called hunks. We build bug prediction models using random forests, which is an efficient machine learning classifier. Hunk metrics are used to train the classifier and each hunk metric is evaluated for its bug prediction capabilities. Our classifier can classify individual hunks as buggy or bug-free with 86 % accuracy, 83 % buggy hunk precision and 77% buggy hunk recall. We find that history based and change level hunk metrics are better predictors of bugs than code level hunk metrics.

[1]  Elaine J. Weyuker,et al.  Predicting the location and number of faults in large software systems , 2005, IEEE Transactions on Software Engineering.

[2]  Alain Abran,et al.  Software Process and Product Measurement, International Conference, IWSM-Mensura 2007, Palma de Mallorca, Spain, November 5-8, 2007. Revised Papers , 2008, IWSM/Mensura.

[3]  Ian Witten,et al.  Data Mining , 2000 .

[4]  Sunghun Kim,et al.  Memories of bug fixes , 2006, SIGSOFT '06/FSE-14.

[5]  Sunghun Kim,et al.  Bug Classification Using Program Slicing Metrics , 2006, 2006 Sixth IEEE International Workshop on Source Code Analysis and Manipulation.

[6]  Taghi M. Khoshgoftaar,et al.  Predicting software errors, during development, using nonlinear regression models: a comparative study , 1992 .

[7]  Giuseppe Visaggio,et al.  Evaluating predictive quality models derived from software measures: Lessons learned , 1997, J. Syst. Softw..

[8]  Donald E. Neumann.  An Enhanced Neural Network Technique for Software Risk Analysis , 2002, IEEE Trans. Software Eng..

[9]  Hongfang Liu,et al.  Building effective defect-prediction models in practice , 2005, IEEE Software.

[10]  Tibor Gyimóthy,et al.  Empirical validation of object-oriented metrics on open source software for fault prediction , 2005, IEEE Transactions on Software Engineering.

[11]  Harald C. Gall,et al.  Populating a Release History Database from version control and bug tracking systems , 2003, International Conference on Software Maintenance, 2003. ICSM 2003. Proceedings..

[12]  Yi Zhang,et al.  Classifying Software Changes: Clean or Buggy? , 2008, IEEE Transactions on Software Engineering.

[13]  Andreas Zeller,et al.  Mining metrics to predict component failures , 2006, ICSE.

[14]  Yuriy Brun,et al.  Finding latent code errors via machine learning over program executions , 2004, Proceedings. 26th International Conference on Software Engineering.

[15]  Norman E. Fenton,et al.  A Critique of Software Defect Prediction Models , 1999, IEEE Trans. Software Eng..

[16]  Tim Menzies,et al.  Data Mining Static Code Attributes to Learn Defect Predictors , 2007 .

[17]  Harvey P. Siy,et al.  Predicting Fault Incidence Using Software Change History , 2000, IEEE Trans. Software Eng..

[18]  Audris Mockus,et al.  Identifying reasons for software changes using historic databases , 2000, Proceedings 2000 International Conference on Software Maintenance.

[19]  Perdita Stevens,et al.  Modelling Recursive Calls with UML State Diagrams , 2003, FASE.

[20]  David G. Kleinbaum,et al.  Logistic Regression. A Self- Learning Text , 1994 .

[21]  D. Wooff.  Logistic Regression: a Self-learning Text, 2nd edn , 2004 .

[22]  Franz Wotawa,et al.  Analysing Bug Prediction Capabilities of Static Code Metrics in Open Source Software , 2008, IWSM/Metrikon/Mensura.

[23]  Ian H. Witten,et al.  Data mining: practical machine learning tools and techniques, 3rd Edition , 1999 .

[24]  Audris Mockus,et al.  Predicting risk of software changes , 2000, Bell Labs Technical Journal.

[25]  Adam A. Porter,et al.  Empirically guided software development using metric-based classification trees , 1990, IEEE Software.

[26]  Harald C. Gall,et al.  EQ-Mine: Predicting Short-Term Defects for Software Evolution , 2007, FASE.

[27]  Martin J. Shepperd,et al.  Comparing Software Prediction Techniques Using Simulation , 2001, IEEE Trans. Software Eng..

[28]  Chadd C. Williams,et al.  Automatic mining of source code repositories to improve bug finding techniques , 2005, IEEE Transactions on Software Engineering.

[29]  Richard C. Holt,et al.  The top ten list: dynamic fault prediction , 2005, 21st IEEE International Conference on Software Maintenance (ICSM'05).

[30]  Andreas Zeller,et al.  When do changes induce fixes? , 2005, ACM SIGSOFT Softw. Eng. Notes.

[31]  J. Guilford.  Fundamental statistics in psychology and education , 1943 .