Revisiting the Correlation Between Alerts and Software Defects: A Case Study on MyFaces, Camel, and CXF

Static analysis tools (e.g., FindBugs) are widely used to detect potential defects in software development. A recent study suggests that there is a moderate correlation between the alerts reported by static analysis tools and software defects [1]. However, despite the actionable alerts reported by static analysis tools, they may report too many meaningless unactionable alerts. Actionable alert refers to the alert which is meaningful and fixable. Unactionable alert (i.e., false positive alert) refers to the alert which is regarded as unimportant to developers, inessential to source code, or will not be fixed by developers. Are all alerts (including both actionable and unactionable alerts) suitable for indicating software defects? To address this question, we classify all the alerts into two categories, namely actionable alerts and unactionable alerts. By the following, we conduct an empirical study to evaluate the degree of correlation between defects and alerts on the evolution of three open source projects with totally 40 releases. The objective of the study is to explore two kinds of correlation analysis: one is the correlation between all the alerts reported by FindBugs and defects among the release history of a project, the other is the correlation between the actionable alerts and defects. As a result, we find that not all the alerts but the actionable alerts are suitable to be an early predictor of defects.

[1]  Dawson R. Engler,et al.  A few billion lines of code later , 2010, Commun. ACM.

[2]  David Hovemeyer,et al.  Using Static Analysis to Find Bugs , 2008, IEEE Software.

[3]  Will G. Hopkins,et al.  A new view of statistics , 2002 .

[4]  Sebastian G. Elbaum,et al.  Predicting accurate and actionable static analysis warnings , 2008, 2008 ACM/IEEE 30th International Conference on Software Engineering.

[5]  Sarah Smith Heckman,et al.  A systematic literature review of actionable alert identification techniques for automated static code analysis , 2011, Inf. Softw. Technol..

[6]  Atsushi Yamada,et al.  Experiences with program static analysis , 1998, Proceedings Fifth International Software Metrics Symposium. Metrics (Cat. No.98TB100262).

[7]  William Pugh,et al.  The Google FindBugs fixit , 2010, ISSTA '10.

[8]  Marco Tulio Valente,et al.  Static correspondence and correlation between field defects and warnings reported by a bug finding tool , 2011, Software Quality Journal.

[9]  Lin Tan,et al.  Finding patterns in static analysis alerts: improving actionable alert ranking , 2014, MSR 2014.

[10]  Yi Wang,et al.  IntFinder: Automatically Detecting Integer Bugs in x86 Binary Program , 2009, ICICS.

[11]  Robert W. Bowdidge,et al.  Why don't software developers use static analysis tools to find bugs? , 2013, 2013 35th International Conference on Software Engineering (ICSE).

[12]  Leon Moonen,et al.  Evaluating the relation between coding standard violations and faultswithin and across software versions , 2009, 2009 6th IEEE International Working Conference on Mining Software Repositories.

[13]  H. Abdi.  The Bonferonni and Šidák Corrections for Multiple Comparisons , 2006 .

[14]  Sarah Smith Heckman,et al.  On establishing a benchmark for evaluating static analysis alert prioritization and classification techniques , 2008, ESEM '08.

[15]  P. Sedgwick.  Spearman’s rank correlation coefficient , 2018, British Medical Journal.

[16]  Marco Tulio Valente,et al.  Study on the relevance of the warnings reported by Java bug-finding tools , 2011, IET Softw..

[17]  Andy Zaidman,et al.  Analyzing the State of Static Analysis: A Large-Scale Evaluation in Open Source Software , 2016, 2016 IEEE 23rd International Conference on Software Analysis, Evolution, and Reengineering (SANER).

[18]  Premkumar T. Devanbu,et al.  To what extent could we detect field defects? an empirical study of false negatives in static bug finding tools , 2012, 2012 Proceedings of the 27th IEEE/ACM International Conference on Automated Software Engineering.

[19]  Sarah Smith Heckman,et al.  A Model Building Process for Identifying Actionable Static Analysis Alerts , 2009, 2009 International Conference on Software Testing Verification and Validation.