An Empirical Comparison of Machine Learning Techniques in Predicting the Bug Severity of Open and Closed Source Projects

Bug severity is the degree of impact that a defect has on the development or operation of a component or system, and can be classified into different levels based on their impact on the system. Identification of severity level can be useful for bug triager in allocating the bug to the concerned bug fixer. Various researchers have attempted text mining techniques in predicting the severity of bugs, detection of duplicate bug reports and assignment of bugs to suitable fixer for its fix. In this paper, an attempt has been made to compare the performance of different machine learning techniques namely Support vector machine SVM, probability based Naive Bayes NB, Decision Tree based J48 A Java implementation of C4.5, rule based Repeated Incremental Pruning to Produce Error Reduction RIPPER and Random Forests RF learners in predicting the severity level 1 to 5 of a reported bug by analyzing the summary or short description of the bug reports. The bug report data has been taken from NASA's PITS Projects and Issue Tracking System datasets as closed source and components of Eclipse, Mozilla & GNOME datasets as open source projects. The analysis has been carried out in RapidMiner and STATISTICA data mining tools. The authors measured the performance of different machine learning techniques by considering i the value of accuracy and F-Measure for all severity level and ii number of best cases at different threshold level of accuracy and F-Measure.

[1]  Bart Goethals,et al.  Predicting the severity of a reported bug , 2010, 2010 7th IEEE Working Conference on Mining Software Repositories (MSR 2010).

[2]  Foutse Khomh,et al.  Is it a bug or an enhancement?: a text-based approach to classify change requests , 2008, CASCON '08.

[3]  Sholom M. Weiss,et al.  Leightweight Document Clustering , 2000, PKDD.

[4]  Tim Menzies,et al.  Automated severity assessment of software defect reports , 2008, 2008 IEEE International Conference on Software Maintenance.

[5]  Gail E. Kaiser,et al.  BUGMINER: Software Reliability Analysis Via Data Mining of Bug Reports , 2011, SEKE.

[6]  Bojan Cukic,et al.  Detecting bug duplicate reports through local references , 2011, Promise '11.

[7]  Ronen Feldman,et al.  Book Reviews: The Text Mining Handbook: Advanced Approaches to Analyzing Unstructured Data by Ronen Feldman and James Sanger , 2008, CL.

[8]  K. K. Chaturvedi,et al.  Determining Bug severity using machine learning techniques , 2012, 2012 CSI Sixth International Conference on Software Engineering (CONSEG).

[9]  Vipin Kumar,et al.  Introduction to Data Mining, (First Edition) , 2005 .

[10]  Chih-Jen Lin,et al.  A comparison of methods for multiclass support vector machines , 2002, IEEE Trans. Neural Networks.

[11]  K. S. Sarma,et al.  Predictive Modeling With SAS Enterprise Miner: Practical Solutions for Business Applications , 2007 .

[12]  Ioannis Stamelos.  Teaching Software Engineering with Free/Libre Open Source Projects , 2009, Int. J. Open Source Softw. Process..

[13]  Per Runeson,et al.  Guidelines for conducting and reporting case study research in software engineering , 2009, Empirical Software Engineering.

[14]  Oscar Nierstrasz,et al.  Assigning bug reports using a vocabulary-based expertise model of developers , 2009, 2009 6th IEEE International Working Conference on Mining Software Repositories.

[15]  Fabrizio Sebastiani,et al.  Machine learning in automated text categorization , 2001, CSUR.

[16]  Gail C. Murphy,et al.  Automatic bug triage using text categorization , 2004, SEKE.

[17]  Serge Demeyer,et al.  Comparing Mining Algorithms for Predicting the Severity of a Reported Bug , 2011, 2011 15th European Conference on Software Maintenance and Reengineering.

[18]  Ahmed Tamrawi,et al.  Fuzzy set and cache-based approach for bug triaging , 2011, ESEC/FSE '11.

[19]  Westley Weimer,et al.  Modeling bug report quality , 2007, ASE '07.

[20]  Zippy Erlich,et al.  Open Source Software: Strengths and Weaknesses , 2009 .

[21]  Jiawei Han,et al.  Data Mining: Concepts and Techniques , 2000 .

[22]  Thomas Zimmermann,et al.  Improving bug triage with bug tossing graphs , 2009, ESEC/FSE '09.

[23]  Ingo Mierswa,et al.  YALE: rapid prototyping for complex data mining tasks , 2006, KDD '06.

[24]  Sholom M. Weiss,et al.  A system for real-time competitive market intelligence , 2002, KDD.

[25]  Onaiza Maqbool,et al.  Bug Prioritization to Facilitate Bug Report Triage , 2012, Journal of Computer Science and Technology.

[26]  Lillian Lee,et al.  Opinion Mining and Sentiment Analysis , 2008, Found. Trends Inf. Retr..

[27]  Tong Zhang,et al.  Text categorization for a comprehensive time-dependent benchmark , 2004, Inf. Process. Manag..

[28]  Daniel M. Germán,et al.  Towards a simplification of the bug report form in eclipse , 2008, MSR '08.

[29]  Gail C. Murphy,et al.  Who should fix this bug? , 2006, ICSE.

[30]  Tao Xie,et al.  Identifying security bug reports via text mining: An industrial case study , 2010, 2010 7th IEEE Working Conference on Mining Software Repositories (MSR 2010).

[31]  Tao Xie,et al.  An approach to detecting duplicate bug reports using natural language and execution information , 2008, 2008 ACM/IEEE 30th International Conference on Software Engineering.

[32]  Thomas J. Ostrand,et al.  \{PROMISE\} Repository of empirical software engineering data , 2007 .

[33]  Jean Hartley,et al.  Case study research , 2004 .

[34]  Iulian Neamtiu,et al.  Fine-grained incremental learning and multi-feature tossing graphs to improve bug triaging , 2010, 2010 IEEE International Conference on Software Maintenance.

[35]  Per Runeson,et al.  Detection of Duplicate Defect Reports Using Natural Language Processing , 2007, 29th International Conference on Software Engineering (ICSE'07).

[36]  Ashish Sureka,et al.  Detecting Duplicate Bug Report Using Character N-Gram-Based Features , 2010, 2010 Asia Pacific Software Engineering Conference.

[37]  Sholom M. Weiss,et al.  Lightweight Document Matching for Help-Desk Applications , 2000, IEEE Intell. Syst..

[38]  Philip J. Guo,et al.  Characterizing and predicting which bugs get fixed: an empirical study of Microsoft Windows , 2010, 2010 ACM/IEEE 32nd International Conference on Software Engineering.

[39]  Jesús M. González-Barahona,et al.  Repositories with Public Data about Software Development , 2010, Int. J. Open Source Softw. Process..

[40]  Tong Zhang,et al.  Text Mining: Predictive Methods for Analyzing Unstructured Information , 2004 .

[41]  Megan Squire,et al.  How the FLOSS Research Community Uses Email Archives , 2012, Int. J. Open Source Softw. Process..

[42]  Vladimir N. Vapnik,et al.  The Nature of Statistical Learning Theory , 2000, Statistics for Engineering and Information Science.

[43]  Christopher J. C. Burges,et al.  A Tutorial on Support Vector Machines for Pattern Recognition , 1998, Data Mining and Knowledge Discovery.

[44]  Pierre Baldi,et al.  Mining the coherence of GNOME bug reports with statistical topic models , 2009, 2009 6th IEEE International Working Conference on Mining Software Repositories.

[45]  Onaiza Maqbool,et al.  Managing Open Bug Repositories through Bug Report Prioritization Using SVMs , 2010 .

[46]  Nello Cristianini,et al.  Large Margin DAGs for Multiclass Classification , 1999, NIPS.

[47]  Patricia B. Cerrito.  Introduction to Data Mining Using SAS Enterprise Miner , 2006 .

[48]  Michael W. Godfrey,et al.  A tale of two browsers , 2011, MSR '11.

[49]  Leo Breiman,et al.  Random Forests , 2001, Machine Learning.

[50]  Vesna Injac-Malbaša.  Open E-Resources in Libraries , 2014 .

[51]  M. F. Porter,et al.  An algorithm for suffix stripping , 1997 .

[52]  Ian H. Witten,et al.  The WEKA data mining software: an update , 2009, SKDD.

[53]  Thomas Zimmermann,et al.  What Makes a Good Bug Report? , 2008, IEEE Transactions on Software Engineering.

[54]  J. Ross Quinlan,et al.  Induction of Decision Trees , 1986, Machine Learning.

[55]  Gerard Salton,et al.  Term-Weighting Approaches in Automatic Text Retrieval , 1988, Inf. Process. Manag..

[56]  Brad A. Myers,et al.  A Linguistic Analysis of How People Describe Software Problems , 2006, Visual Languages and Human-Centric Computing (VL/HCC'06).

[57]  William W. Cohen.  Fast Effective Rule Induction , 1995, ICML.

[58]  Fang Wu,et al.  Predicting Defect Priority Based on Neural Networks , 2010, ADMA.