A Reflection on “An Exploratory Study on Exception Handling Bugs in Java Programs”

We study exception handling bugs from two real systems.We survey developers to understand their thoughts about exception handling bugs.Analysis of bug repositories shows small percentages of exception handling bugs.Exception handling bugs seems to be as hard to fix as other kind of bugs.We create an exception handling bug classification. Most mainstream programming languages provide constructs to throw and to handle exceptions. However, several studies argue that exception handling code is usually of poor quality and that it is commonly neglected by developers. Moreover, it is said to be the least understood, documented, and tested part of the implementation of a system. Nevertheless, there are very few studies that analyze the actual exception handling bugs that occur in real software systems or that attempt to understand developers' perceptions of these bugs. In this work we present an exploratory study on exception handling bugs that employs two complementary approaches: a survey of 154 developers and an analysis of 220 exception handling bugs from the repositories of Eclipse and Tomcat.Only 27% of the respondents claimed that policies and standards for the implementation of error handling are part of the culture of their organizations. Moreover, in 70% of the organizations there are no specific tests for the exception handling code. Also, 61% of the respondents stated that no to little importance is given to the documentation of exception handling in the design phase of the projects with which they are involved. In addition, about 40% of the respondents consider the quality of exception handling code to be either good or very good and only 14% of the respondents consider it to be bad or very bad. Furthermore, the repository analysis has shown (with statistical significance) that exception handling bugs are ignored by developers less often than other bugs. We have also observed that while overly general catch blocks are a well-known bad smell related to exceptions, bugs stemming from these catch blocks are rare, even though many overly general catch blocks occur in the code. Furthermore, while developers often mention empty catch blocks as causes of bugs they have fixed in the past, we found very few bug reports caused by them. On top of that, empty catch blocks are frequently used as part of bug fixes, including fixes for exception handling bugs.Based on our findings, we propose a classification of exception handling bugs and their causes. The proposed classification can be used to assist in the design and implementation of test suites, to guide code inspections, or as a basis for static analysis tools.

[1]  Martin P. Robillard,et al.  Enforcing Exception Handling Policies with a Domain-Specific Language , 2016, IEEE Transactions on Software Engineering.

[2]  Carl F. Schaefer,et al.  Static analysis of exception handling in Ada , 1993, Softw. Pract. Exp..

[3]  A. Agresti,et al.  Categorical Data Analysis , 1991, International Encyclopedia of Statistical Science.

[4]  John McCarthy,et al.  History of LISP , 1978, SIGP.

[5]  R. Fisher.  Statistical methods for research workers , 1927, Protoplasma.

[6]  Carlos José Pereira de Lucena,et al.  Unveiling and taming liabilities of aspects in the presence of exceptions: A static analysis based approach , 2011, Inf. Sci..

[7]  Arie van Deursen,et al.  Discovering faults in idiom-based exception handling , 2006, ICSE '06.

[8]  Nélio Cacho,et al.  An Exploratory Study of Exception Handling Behavior in Evolving Android and Java Applications , 2016, SBES '16.

[9]  Cheng Li,et al.  A study of the internal and external effects of concurrency bugs , 2010, 2010 IEEE/IFIP International Conference on Dependable Systems & Networks (DSN).

[10]  M. Hazelton.  Variable kernel density estimation , 2003 .

[11]  Yannis Smaragdakis,et al.  Exception analysis and points-to analysis: better together , 2009, ISSTA.

[12]  Weiyi Shang,et al.  Revisiting Exception Handling Practices with Exception Flow Analysis , 2017, 2017 IEEE 17th International Working Conference on Source Code Analysis and Manipulation (SCAM).

[13]  Per Runeson,et al.  Guidelines for conducting and reporting case study research in software engineering , 2009, Empirical Software Engineering.

[14]  Alessandro F. Garcia,et al.  Categorizing Faults in Exception Handling: A Study of Open Source Projects , 2014, 2014 Brazilian Symposium on Software Engineering.

[15]  Y. Benjamini,et al.  Controlling the false discovery rate: a practical and powerful approach to multiple testing , 1995 .

[16]  Felipe Ebert,et al.  A Study on Developers' Perceptions about Exception Handling Bugs , 2013, 2013 IEEE International Conference on Software Maintenance.

[17]  Carolyn B. Seaman,et al.  Qualitative Methods in Empirical Studies of Software Engineering , 1999, IEEE Trans. Software Eng..

[18]  Edward B. Allen,et al.  Exception Handling Defects: An Empirical Study , 2012, 2012 IEEE 14th International Symposium on High-Assurance Systems Engineering.

[19]  D. W. Scott,et al.  Variable Kernel Density Estimation , 1992 .

[20]  M. Kendall.  A NEW MEASURE OF RANK CORRELATION , 1938 .

[21]  Rossana M. de Castro Andrade,et al.  Preventing Erosion in Exception Handling Design Using Static-Architecture Conformance Checking , 2017, ECSA.

[22]  Eduardo Figueiredo,et al.  On the modularization and reuse of exception handling with aspects , 2009 .

[23]  Cecília M. F. Rubira,et al.  A comparative study of exception handling mechanisms for building dependable object-oriented software , 2001, J. Syst. Softw..

[24]  David Hovemeyer,et al.  Finding bugs is easy , 2004, SIGP.

[25]  F. Wilcoxon.  Individual Comparisons by Ranking Methods , 1945 .

[26]  Peter A. Buhr,et al.  Exception Handling , 2002, Advances in Computing.

[27]  Hina Shah,et al.  Understanding Exception Handling: Viewpoints of Novices and Experts , 2010, IEEE Transactions on Software Engineering.

[28]  Tom Cargill.  Exception handling: a false sense of security , 1996 .

[29]  Uirá Kulesza,et al.  The use of C++ exception handling constructs: A comprehensive study , 2015, 2015 IEEE 15th International Working Conference on Source Code Analysis and Manipulation (SCAM).

[30]  Shawn A. Ross,et al.  Survey Methodology , 2005, The SAGE Encyclopedia of the Sociology of Religion.

[31]  David Lorge Parnas,et al.  Response to undesired events in software systems , 1976, ICSE '76.

[32]  Uirá Kulesza,et al.  A catalogue of bug patterns for exception handling in aspect-oriented programs , 2008 .

[33]  Mira Mezini,et al.  A recommendation system for exception handling code , 2012, 2012 5th International Workshop on Exception Handling (WEH).

[34]  George C. Necula,et al.  Exceptional situations and program reliability , 2008, TOPL.

[35]  J. Hintze,et al.  Violin plots : A box plot-density trace synergism , 1998 .

[36]  Raed Shatnawi,et al.  An empirical study of the bad smells and class error probability in the post-release object-oriented system evolution , 2007, J. Syst. Softw..

[37]  Andrew P. Black,et al.  Exception handling : The case against , 1982 .

[38]  Paulo Marques,et al.  Exception Handling: A Field Study in Java and .NET , 2007, ECOOP.

[39]  Douglas C. Montgomery,et al.  Applied Statistics and Probability for Engineers, Third edition , 1994 .

[40]  P. M. Melliar-Smith,et al.  Software reliability: The role of programmed exception handling , 1977, Language Design for Reliable Software.

[41]  Cristina Marinescu.  Are the classes that use exceptions defect prone? , 2011, IWPSE-EVOL '11.

[42]  Uirá Kulesza,et al.  Understanding the Exception Handling Strategies of Java Libraries: An Empirical Study , 2016, 2016 IEEE/ACM 13th Working Conference on Mining Software Repositories (MSR).

[43]  Martin P. Robillard,et al.  Static analysis to support the evolution of exception structure in object-oriented systems , 2003, TSEM.

[44]  Shari Lawrence Pfleeger,et al.  Personal Opinion Surveys , 2008, Guide to Advanced Empirical Software Engineering.

[45]  S. Shapiro,et al.  An Analysis of Variance Test for Normality (Complete Samples) , 1965 .

[46]  Fernando Castor Filho,et al.  New exception interfaces for Java-like languages , 2013, SAC '13.

[47]  Brad A. Myers,et al.  Moonstone: Support for understanding and writing exception handling code , 2017, 2017 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC).

[48]  Janice Singer,et al.  Guide to Advanced Empirical Software Engineering , 2007 .

[49]  Carlos José Pereira de Lucena,et al.  Assessing the Impact of Aspects on Exception Flows: An Exploratory Study , 2008, ECOOP.

[50]  Premkumar T. Devanbu,et al.  The missing links: bugs and bug-fix commits , 2010, FSE '10.

[51]  A. Zeller,et al.  Predicting Defects for Eclipse , 2007, Third International Workshop on Predictor Models in Software Engineering (PROMISE'07: ICSE Workshops 2007).

[52]  Weiyi Shang,et al.  Studying the Prevalence of Exception Handling Anti-Patterns , 2017, 2017 IEEE/ACM 25th International Conference on Program Comprehension (ICPC).

[53]  Emery D. Berger,et al.  SurveyMan , 2014, OOPSLA.

[54]  Diomidis Spinellis,et al.  Undocumented and unchecked: exceptions that spell trouble , 2014, MSR 2014.

[55]  Cristina Marinescu.  Should We Beware the Exceptions? An Empirical Study on the Eclipse Project , 2013, 2013 15th International Symposium on Symbolic and Numeric Algorithms for Scientific Computing.

[56]  John B. Goodenough,et al.  Exception handling: issues and a proposed notation , 1975, CACM.

[57]  Dr. Andreas Müller,et al.  Exception Handling : Common Problems and Best Practice with Java 1 . 4 , .

[58]  Vikram S. Adve,et al.  An empirical study of reported bugs in server software with implications for automated bug diagnosis , 2010, 2010 ACM/IEEE 32nd International Conference on Software Engineering.

[59]  Windson Viana de Carvalho,et al.  Avaliando o Tratamento de Exceção em um Sistema Web Corporativo , 2017 .

[60]  Chen Fu,et al.  Exception-Chain Analysis: Revealing Exception Handling Architecture in Java Server Applications , 2007, 29th International Conference on Software Engineering (ICSE'07).

[61]  Ken Arnold,et al.  The Java Programming Language , 1996 .

[62]  Diomidis Spinellis,et al.  Towards a Context Dependent Java Exceptions Hierarchy , 2017, 2017 IEEE/ACM 39th International Conference on Software Engineering Companion (ICSE-C).

[63]  Diomidis Spinellis,et al.  Type Checking for Reliable APIs , 2017, 2017 IEEE/ACM 1st International Workshop on API Usage and Evolution (WAPI).

[64]  Hermann Kopetz,et al.  Fault tolerance, principles and practice , 1990 .

[65]  David Hovemeyer,et al.  Using Static Analysis to Find Bugs , 2008, IEEE Software.

[66]  Amey Karkare,et al.  Heap reference analysis using access graphs , 2006, ACM Trans. Program. Lang. Syst..

[67]  Sebastian G. Elbaum,et al.  Amplifying tests to validate exception handling code , 2012, 2012 34th International Conference on Software Engineering (ICSE).

[68]  Flaviu Cristian.  A Recovery Mechanism for Modular Software , 1979, ICSE.

[69]  Robin Milner,et al.  Definition of standard ML , 1990 .

[70]  José Carlos Bregieiro Ribeiro,et al.  Trends on empty exception handlers for Java open source libraries , 2017, 2017 IEEE 24th International Conference on Software Analysis, Evolution and Reengineering (SANER).