Survey of Approaches for Handling Static Analysis Alarms

Static analysis tools have showcased their importance and usefulness in automated detection of code anomalies and defects. However, the large number of alarms reported and cost incurred in their manual inspections have been the major concerns with the usage of static analysis tools. Existing studies addressing these concerns differ greatly in their approaches to handle the alarms, varying from automatic postprocessing of alarms, supporting the tool-users during manual inspections of the alarms, to designing of light-weight static analysis tools. A comprehensive study of approaches for handling alarms is, however, not found. In this paper, we review 79 alarms handling studies collected through a systematic literature search and classify the approaches proposed into seven categories. The literature search is performed by combining the keywords-based database search and snowballing. Our review is intended to provide an overview of various alarms handling approaches, their merits and shortcomings, and different techniques used in their implementations. Our findings include that the categorized alarms handling approaches are complementary and they can be combined together in different ways. The categorized approaches and techniques employed in them can help the designers and developers of static analysis tools to make informed choices.

[1]  Hasan Sözer,et al.  Integrated static code analysis and runtime verification , 2015, Softw. Pract. Exp..

[2]  Michael D. Ernst,et al.  Prioritizing Warning Categories by Analyzing Software History , 2007, Fourth International Workshop on Mining Software Repositories (MSR'07:ICSE Workshops 2007).

[3]  Eric Mercer,et al.  A Meta Heuristic for Effectively Detecting Concurrency Errors , 2008, Haifa Verification Conference.

[4]  Sarah Smith Heckman,et al.  A systematic literature review of actionable alert identification techniques for automated static code analysis , 2011, Inf. Softw. Technol..

[5]  Robert W. Bowdidge,et al.  Why don't software developers use static analysis tools to find bugs? , 2013, 2013 35th International Conference on Software Engineering (ICSE).

[6]  David Hovemeyer,et al.  Tracking defect warnings across versions , 2006, MSR '06.

[7]  Chao Chen,et al.  ISA: a source code static vulnerability detection system based on data fusion , 2007 .

[8]  Qian Wu,et al.  An Approach to Merge Results of Multiple Static Analysis Tools (Short Paper) , 2008, 2008 The Eighth International Conference on Quality Software.

[9]  David Evans,et al.  Improving Security Using Extensible Lightweight Static Analysis , 2002, IEEE Softw..

[10]  Claes Wohlin,et al.  Systematic literature reviews in software engineering , 2013, Inf. Softw. Technol..

[11]  Paul Ralph,et al.  Grounded Theory in Software Engineering Research: A Critical Review and Guidelines , 2016, 2016 IEEE/ACM 38th International Conference on Software Engineering (ICSE).

[12]  Hosik Choi,et al.  An empirical study on classification methods for alarms from a bug-finding static C analyzer , 2007, Inf. Process. Lett..

[13]  Tukaram Muske.  Improving Review of Clustered-Code Analysis Warnings , 2014, 2014 IEEE International Conference on Software Maintenance and Evolution.

[14]  L. Moonen,et al.  Prioritizing Software Inspection Results using Static Profiling , 2006, 2006 Sixth IEEE International Workshop on Source Code Analysis and Manipulation.

[15]  Nuno Silva,et al.  Static Analysis Tools, a Practical Approach for Safety-Critical Software Verification , 2009 .

[16]  Claes Wohlin,et al.  Guidelines for snowballing in systematic literature studies and a replication in software engineering , 2014, EASE '14.

[17]  Patrick Cousot,et al.  The ASTREÉ Analyzer , 2005, ESOP.

[18]  Roberto Bagnara,et al.  Precise widening operators for convex polyhedra , 2003, Sci. Comput. Program..

[19]  Andy Zaidman,et al.  Analyzing the State of Static Analysis: A Large-Scale Evaluation in Open Source Software , 2016, 2016 IEEE 23rd International Conference on Software Analysis, Evolution, and Reengineering (SANER).

[20]  Padmanabhan Krishnan,et al.  A Method for Scalable and Precise Bug Finding Using Program Analysis and Model Checking , 2014, APLAS.

[21]  Ciera Jaspan,et al.  Tricorder: Building a Program Analysis Ecosystem , 2015, 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering.

[22]  Premkumar T. Devanbu,et al.  To what extent could we detect field defects? an empirical study of false negatives in static bug finding tools , 2012, 2012 Proceedings of the 27th IEEE/ACM International Conference on Automated Software Engineering.

[23]  Jianjun Zhao,et al.  EFindBugs: Effective Error Ranking for FindBugs , 2011, 2011 Fourth IEEE International Conference on Software Testing, Verification and Validation.

[24]  Hakjoo Oh,et al.  Sound Non-Statistical Clustering of Static Analysis Alarms , 2017, ACM Trans. Program. Lang. Syst..

[25]  Nikolai Kosmatov,et al.  Combining Static and Dynamic Analyses for Vulnerability Detection: Illustration on Heartbleed , 2015, Haifa Verification Conference.

[26]  Jun Zhou,et al.  A Hybrid Approach to Detecting Security Defects in Programs , 2009, 2009 Ninth International Conference on Quality Software.

[27]  Yannis Smaragdakis,et al.  Check 'n' crash: combining static checking and testing , 2005, Proceedings. 27th International Conference on Software Engineering, 2005. ICSE 2005..

[28]  Stefan Wagner,et al.  At Ease with Your Warnings: The Principles of the Salutogenesis Model Applied to Automatic Static Analysis , 2016, 2016 IEEE 23rd International Conference on Software Analysis, Evolution, and Reengineering (SANER).

[29]  Sam Blackshear,et al.  Almost-correct specifications: a modular semantic framework for assigning confidence to warnings , 2013, PLDI.

[30]  Emilia Mendes,et al.  How Reliable Are Systematic Reviews in Empirical Software Engineering? , 2010, IEEE Transactions on Software Engineering.

[31]  Sarah Smith Heckman.  Adaptively ranking alerts generated from automated static analysis , 2007, ACM Crossroads.

[32]  Tao Xie,et al.  Automatic construction of an effective training set for prioritizing static analysis warnings , 2010, ASE.

[33]  Yungbum Jung,et al.  Reducing False Alarms from an Industrial-Strength Static Analyzer by SVM , 2014, 2014 21st Asia-Pacific Software Engineering Conference.

[34]  J. Jenny Li,et al.  SoftWare IMmunization (SWIM) - A Combination of Static Analysis and Automatic Testing , 2011, 2011 IEEE 35th Annual Computer Software and Applications Conference.

[35]  Ying Xing,et al.  Automatically mining similar warnings and warning combinations , 2013, 2013 10th International Conference on Fuzzy Systems and Knowledge Discovery (FSKD).

[36]  Sarah Smith Heckman,et al.  Using groupings of static analysis alerts to identify files likely to contain field failures , 2007, ESEC-FSE '07.

[37]  Rahul Kumar,et al.  The economics of static analysis tools , 2013, ESEC/FSE 2013.

[38]  Michael D. Ernst,et al.  Which warnings should I fix first? , 2007, ESEC-FSE '07.

[39]  Chadd C. Williams,et al.  Automatic mining of source code repositories to improve bug finding techniques , 2005, IEEE Transactions on Software Engineering.

[40]  Auri Marcelo Rizzo Vincenzi,et al.  Static Analysis Techniques and Tools: A Systematic Mapping Study , 2013, ICSEA 2013.

[41]  Sarah Smith Heckman,et al.  A Model Building Process for Identifying Actionable Static Analysis Alerts , 2009, 2009 International Conference on Software Testing Verification and Validation.

[42]  Dawson R. Engler,et al.  A system and language for building system-specific, static analyses , 2002, PLDI '02.

[43]  Guoqing Xu,et al.  Dynamically validating static memory leak warnings , 2013, ISSTA.

[44]  Mary Lou Soffa,et al.  Generating analyses for detecting faults in path segments , 2011, ISSTA '11.

[45]  Maximilian Junker,et al.  SMT-Based False Positive Elimination in Static Program Analysis , 2012, ICFEM.

[46]  Kumar Madhukar,et al.  Efficient Elimination of False Positives Using Bounded Model Checking , 2013 .

[47]  Kwang-Moo Choe,et al.  Filtering false alarms of buffer overflow analysis using SMT solvers , 2010, Inf. Softw. Technol..

[48]  Qian Wu,et al.  An Effective Defect Detection and Warning Prioritization Approach for Resource Leaks , 2012, 2012 IEEE 36th Annual Computer Software and Applications Conference.

[49]  Tukaram B. Muske,et al.  Review efforts reduction by partitioning of static analysis warnings , 2013, 2013 IEEE 13th International Working Conference on Source Code Analysis and Manipulation (SCAM).

[50]  Hasan Sözer,et al.  Automated Classification of Static Code Analysis Alerts: A Case Study , 2013, 2013 IEEE International Conference on Software Maintenance.

[51]  Andreas Podelski,et al.  Classifying Bugs with Interpolants , 2016, TAP@STAF.

[52]  Kwangkeun Yi,et al.  Taming False Alarms from a Domain-Unaware C Analyzer by a Bayesian Statistical Post Analysis , 2005, SAS.

[53]  Pankaj Jalote,et al.  Integrating Static and Dynamic Analysis for Detecting Vulnerabilities , 2006, 30th Annual International Computer Software and Applications Conference (COMPSAC'06).

[54]  Alexander Serebrenik,et al.  Empirical analysis of the relationship between CC and SLOC in a large corpus of Java methods and C functions , 2016, J. Softw. Evol. Process..

[55]  Daniel Cordes,et al.  A Fast and Precise Static Loop Analysis Based on Abstract Interpretation, Program Slicing and Polytope Models , 2009, 2009 International Symposium on Code Generation and Optimization.

[56]  Pearl Brereton,et al.  Performing systematic literature reviews in software engineering , 2006, ICSE.

[57]  Claes Wohlin,et al.  Experiences from using snowballing and database searches in systematic literature studies , 2015, EASE.

[58]  William Pugh,et al.  Using checklists to review static analysis warnings , 2009, DEFECTS '09.

[59]  Dawson R. Engler,et al.  Z-Ranking: Using Statistical Analysis to Counter the Impact of Static Analysis Approximations , 2003, SAS.

[60]  Raoul Praful Jetley,et al.  Static analysis of medical device software using CodeSonar , 2008, SAW '08.

[61]  Yannis Smaragdakis,et al.  Residual Investigation , 2014, ACM Trans. Softw. Eng. Methodol..

[62]  Danfeng Zhang,et al.  Toward general diagnosis of static errors , 2014, POPL.

[63]  Laurie A. Williams,et al.  On the value of static analysis for fault detection in software , 2006, IEEE Transactions on Software Engineering.

[64]  Sebastian G. Elbaum,et al.  Predicting accurate and actionable static analysis warnings , 2008, 2008 ACM/IEEE 30th International Conference on Software Engineering.

[65]  Heejo Lee,et al.  Software Vulnerability Detection Using Backward Trace Analysis and Symbolic Execution , 2013, 2013 International Conference on Availability, Reliability and Security.

[66]  Qiang Zhang,et al.  Automated Detection of Code Vulnerabilities Based on Program Analysis and Model Checking , 2008, 2008 Eighth IEEE International Working Conference on Source Code Analysis and Manipulation.

[67]  Westley Weimer,et al.  Clustering static analysis defect reports to reduce maintenance costs , 2013, 2013 20th Working Conference on Reverse Engineering (WCRE).

[68]  Khoo Yit Phang,et al.  Triaging Checklists : a Substitute for a PhD in Static Analysis , 2009 .

[69]  William Pugh,et al.  The Google FindBugs fixit , 2010, ISSTA '10.

[70]  Priyanka Darke,et al.  Precise Analysis of Large Industry Code , 2012, 2012 19th Asia-Pacific Software Engineering Conference.

[71]  Lin Tan,et al.  Finding patterns in static analysis alerts: improving actionable alert ranking , 2014, MSR 2014.

[72]  Yi Wang,et al.  IntFinder: Automatically Detecting Integer Bugs in x86 Binary Program , 2009, ICICS.

[73]  Chadd C. Williams,et al.  Bug Driven Bug Finders , 2004, MSR.

[74]  Willem Visser,et al.  Combining static analysis and model checking for software analysis , 2001, Proceedings 16th Annual International Conference on Automated Software Engineering (ASE 2001).

[75]  Xu Zhou,et al.  Pruning False Positives of Static Data-Race Detection via Thread Specialization , 2013, APPT.

[76]  Nikolai Tillmann,et al.  DyTa: dynamic symbolic execution guided with static verification results , 2011, 2011 33rd International Conference on Software Engineering (ICSE).

[77]  J. David Morgenthaler,et al.  Evaluating static analysis defect warnings on production software , 2007, PASTE '07.

[78]  Lucas Layman,et al.  Toward Reducing Fault Fix Time: Understanding Developer Behavior for the Design of Automated Fault Detection Tools , 2007, First International Symposium on Empirical Software Engineering and Measurement (ESEM 2007).

[79]  Jeffrey S. Foster,et al.  A comparison of bug finding tools for Java , 2004, 15th International Symposium on Software Reliability Engineering.

[80]  Yunzhan Gong,et al.  Diagnosis-Oriented Alarm Correlations , 2013, 2013 20th Asia-Pacific Software Engineering Conference (APSEC).

[81]  Nicolas Anquetil,et al.  Identifying the exact fixing actions of static rule violation , 2015, 2015 IEEE 22nd International Conference on Software Analysis, Evolution, and Reengineering (SANER).

[82]  S. V. Subrahmanya,et al.  A Survey of Enterprise Software Development Risks in a Flat World , 2007, ESEM 2007.

[83]  Supratik Chakraborty,et al.  Automatically Refining Abstract Interpretations , 2008, TACAS.

[84]  Priyanka Darke,et al.  Eliminating Static Analysis False Positives Using Loop Abstraction and Bounded Model Checking , 2015, FM.

[85]  Tao Xie,et al.  DSD-Crasher: A hybrid analysis tool for bug finding , 2008 .

[86]  Mary Lou Soffa,et al.  Path-based fault correlations , 2010, FSE '10.

[87]  Isil Dillig,et al.  Automated error diagnosis using abductive inference , 2012, PLDI.

[88]  Shrawan Kumar,et al.  Effective false positive filtering for evolving software , 2011, ISEC.

[89]  Hironori Washizaki,et al.  A Gamified Tool for Motivating Developers to Remove Warnings of Bug Pattern Tools , 2014, 2014 6th International Workshop on Empirical Software Engineering in Practice.

[90]  Vibha Sazawal,et al.  Path projection for user-centered static analysis tools , 2008, PASTE '08.

[91]  Dawson R. Engler,et al.  A few billion lines of code later , 2010, Commun. ACM.

[92]  Xavier Rival,et al.  Understanding the Origin of Alarms in Astrée , 2005, SAS.

[93]  Chris Parnin,et al.  A catalogue of lightweight visualizations to support code smell inspection , 2008, SOFTVIS.

[94]  Sam Blackshear,et al.  Verification modulo versions: towards usable verification , 2014, PLDI.

[95]  Paul Anderson,et al.  Tool Support for Fine-Grained Software Inspection , 2003, IEEE Softw..

[96]  Shuvendu K. Lahiri,et al.  Angelic Verification: Precise Verification Modulo Unknowns , 2015, CAV.

[97]  Xin Zhang,et al.  A user-guided approach to program analysis , 2015, ESEC/SIGSOFT FSE.

[98]  Tukaram Muske,et al.  Efficient elimination of false positives using static analysis , 2015, 2015 IEEE 26th International Symposium on Software Reliability Engineering (ISSRE).

[99]  D. V. Radhika,et al.  An automated approach to detect violations with high confidence in incremental code using a learning system , 2014, ICSE Companion.

[100]  Ralf Huuck,et al.  Model checking driven static analysis for the real world: designing and tuning large scale bug detection , 2012, Innovations in Systems and Software Engineering.

[101]  Junfeng Yang,et al.  Correlation exploitation in error ranking , 2004, SIGSOFT '04/FSE-12.

[102]  Carsten Sinz,et al.  Reducing False Positives by Combining Abstract Interpretation and Bounded Model Checking , 2008, 2008 23rd IEEE/ACM International Conference on Automated Software Engineering.

[103]  Nikolai Kosmatov,et al.  Program slicing enhances a verification technique combining static and dynamic analysis , 2012, SAC '12.

[104]  Frank Elberzhager,et al.  A systematic mapping study on the combination of static and dynamic quality assurance techniques , 2012, Inf. Softw. Technol..

[105]  Xavier Rival,et al.  Abstract Dependences for Alarm Diagnosis , 2005, APLAS.

[106]  Ulf Nilsson,et al.  A Comparative Study of Industrial Static Analysis Tools , 2008, SSV.

[107]  Nicolas Anquetil,et al.  A Framework to Compare Alert Ranking Algorithms , 2012, 2012 19th Working Conference on Reverse Engineering.

[108]  Willem Visser,et al.  Variably interprocedural program analysis for runtime error detection , 2007, ISSTA '07.

[109]  Panagiotis Katsaros,et al.  Test-Driving Static Analysis Tools in Search of C Code Vulnerabilities , 2011, 2011 IEEE 35th Annual Computer Software and Applications Conference Workshops.