Learning Likely Invariants to Explain Why a Program Fails

Debugging is difficult. Recent studies show that automatic bug localization techniques have limited usefulness. One of the reasons is that programmers typically have to understand why the program fails before fixing it. In this work, we aim to help programmers understand a bug by automatically generating likely invariants which are violated in the failed tests. Given a program with an initial assertion and at least one test case failing the assertion, we first generate random test cases, identify potential bug locations through bug localization, and then generate program state mutation based on active learning techniques to identify a predicate "explaining" the cause of the bug. The predicate is a classifier for the passed test cases and failed test cases. Our main contribution is the application of invariant learning for bug explanation, as well as a novel approach to overcome the problem of lack of test cases in practice. We apply our method to real-world bugs and show the generated invariants are often correlated to the actual bug fixes.

[1]  Yannis Smaragdakis,et al.  Dynamically discovering likely interface invariants , 2006, ICSE '06.

[2]  Westley Weimer,et al.  Automatic documentation inference for exceptions , 2008, ISSTA '08.

[3]  Isabelle Guyon,et al.  An Introduction to Variable and Feature Selection , 2003, J. Mach. Learn. Res..

[4]  Michael D. Ernst,et al.  Invariant inference for static checking: , 2002, SIGSOFT '02/FSE-10.

[5]  Francesco Orabona,et al.  Better Algorithms for Selective Sampling , 2011, ICML.

[6]  Hal Berghel,et al.  An interactive source commenter for Prolog programs , 1990, SIGDOC '90.

[7]  Rupak Majumdar,et al.  Cause clue clauses: error localization using maximum satisfiability , 2010, PLDI '11.

[8]  Yuriy Brun,et al.  Repairing Programs with Semantic Code Search (T) , 2015, 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE).

[9]  Bertrand Meyer,et al.  Contracts in Practice , 2012, FM.

[10]  Andreas Zeller,et al.  Inferring Loop Invariants by Mutation, Dynamic Analysis, and Static Checking , 2014, IEEE Transactions on Software Engineering.

[11]  Zvonimir Rakamaric,et al.  Symbolic Learning of Component Interfaces , 2012, SAS.

[12]  Sebastián Uchitel,et al.  Automated Abstractions for Contract Validation , 2012, IEEE Transactions on Software Engineering.

[13]  Rui Abreu,et al.  A Survey on Software Fault Localization , 2016, IEEE Transactions on Software Engineering.

[14]  Alex Groce,et al.  What Went Wrong: Explaining Counterexamples , 2003, SPIN.

[15]  Girish H. Subramanian,et al.  Dimensionality reduction in software development effort estimation , 1993, J. Syst. Softw..

[16]  Shriram Krishnamurthi,et al.  Automated Fault Localization Using Potential Invariants , 2003, ArXiv.

[17]  Michael I. Jordan,et al.  Bug isolation via remote program sampling , 2003, PLDI.

[18]  Jun Sun,et al.  TLV: abstraction through testing, learning, and validation , 2015, ESEC/SIGSOFT FSE.

[19]  Michael D. Ernst,et al.  Efficient incremental algorithms for dynamic detection of likely invariants , 2004, SIGSOFT '04/FSE-12.

[20]  Sooyong Park,et al.  Which Crashes Should I Fix First?: Predicting Top Crashes at an Early Stage to Prioritize Debugging Efforts , 2011, IEEE Transactions on Software Engineering.

[21]  Rupak Majumdar,et al.  Bug-Assist: Assisting Fault Localization in ANSI-C Programs , 2011, CAV.

[22]  Westley Weimer,et al.  Automatically documenting program changes , 2010, ASE.

[23]  Alessandro Orso,et al.  Are automated debugging techniques actually helping programmers? , 2011, ISSTA '11.

[24]  Jun Sun,et al.  TzuYu: Learning stateful typestates , 2013, 2013 28th IEEE/ACM International Conference on Automated Software Engineering (ASE).

[25]  Shriram Krishnamurthi,et al.  Automated Fault Localization Using Potential Invariants , 2003, ArXiv.

[26]  Koushik Sen,et al.  Symbolic execution for software testing: three decades later , 2013, CACM.

[27]  Martin J. Shepperd,et al.  Estimating Software Project Effort Using Analogies , 1997, IEEE Trans. Software Eng..

[28]  Michael D. Ernst,et al.  Automated documentation inference to explain failed tests , 2011, 2011 26th IEEE/ACM International Conference on Automated Software Engineering (ASE 2011).

[29]  D. Ross Jeffery,et al.  Analogy-X: Providing Statistical Inference to Analogy-Based Software Cost Estimation , 2008, IEEE Transactions on Software Engineering.

[30]  Yves Le Traon,et al.  From testing to diagnosis: an automated approach , 2004, Proceedings. 19th International Conference on Automated Software Engineering, 2004..

[31]  Aws Albarghouthi,et al.  Beautiful Interpolants , 2013, CAV.

[32]  Sudheendra Hangal,et al.  Tracking down software bugs using automatic anomaly detection , 2002, ICSE '02.

[33]  Yiming Yang,et al.  A Comparative Study on Feature Selection in Text Categorization , 1997, ICML.

[34]  Tevfik Bultan,et al.  Interface Grammars for Modular Software Model Checking , 2007, IEEE Transactions on Software Engineering.

[35]  Andreas Zeller,et al.  Yesterday, my program worked. Today, it does not. Why? , 1999, ESEC/FSE-7.

[36]  Michael D. Ernst,et al.  Defects4J: a database of existing faults to enable controlled testing studies for Java programs , 2014, ISSTA 2014.

[37]  Alexander Aiken,et al.  Interpolants as Classifiers , 2012, CAV.

[38]  Michael D. Ernst,et al.  Feedback-Directed Random Test Generation , 2007, 29th International Conference on Software Engineering (ICSE'07).

[39]  Zhenkai Liang,et al.  DARWIN: An approach to debugging evolving programs , 2012, TSEM.

[40]  Pavol Cerný,et al.  Synthesis of interface specifications for Java classes , 2005, POPL '05.

[41]  Ilan Beer,et al.  Explaining counterexamples using causality , 2009, Formal Methods in System Design.

[42]  Daniel Kroening,et al.  Learning the Language of Error , 2015, ATVA.

[43]  Ayse Basar Bener,et al.  An industrial case study of classifier ensembles for locating software defects , 2011, Software Quality Journal.

[44]  A. Zeller.  Isolating cause-effect chains from computer programs , 2002, SIGSOFT '02/FSE-10.

[45]  Yuriy Brun,et al.  Leveraging existing instrumentation to automatically infer invariant-constrained models , 2011, ESEC/FSE '11.

[46]  Andreas Zeller,et al.  Generating parameterized unit tests , 2011, ISSTA '11.

[47]  Baowen Xu,et al.  Revisit of Automatic Debugging via Human Focus-Tracking Analysis , 2016, 2016 IEEE/ACM 38th International Conference on Software Engineering (ICSE).

[48]  Shin Yoo,et al.  Ask the Mutants: Mutating Faulty Programs for Fault Localization , 2014, 2014 IEEE Seventh International Conference on Software Testing, Verification and Validation.

[49]  Alessandro Orso,et al.  Isolating failure causes through test case generation , 2012, ISSTA 2012.

[50]  Domagoj Babic,et al.  Sigma*: symbolic learning of input-output specifications , 2013, POPL.

[51]  Sunghun Kim,et al.  How long did it take to fix bugs? , 2006, MSR '06.

[52]  William G. Griswold,et al.  Dynamically discovering likely program invariants to support program evolution , 1999, Proceedings of the 1999 International Conference on Software Engineering (IEEE Cat. No.99CB37002).

[53]  Michael D. Ernst,et al.  Automatic generation of program specifications , 2002, ISSTA '02.

[54]  Gordon Fraser,et al.  Evolutionary Generation of Whole Test Suites , 2011, 2011 11th International Conference on Quality Software.

[55]  Fan Long,et al.  Automatic patch generation by learning correct code , 2016, POPL.

[56]  Daphne Koller,et al.  Support Vector Machine Active Learning with Applications to Text Classification , 2000, J. Mach. Learn. Res..

[57]  Zhendong Su,et al.  Reusing debugging knowledge via trace-based bug search , 2012, OOPSLA '12.

[58]  Jun Sun,et al.  Assertion Generation through Active Learning , 2017, 2017 IEEE/ACM 39th International Conference on Software Engineering Companion (ICSE-C).

[59]  Edward Y. Chang,et al.  Support vector machine active learning for image retrieval , 2001, MULTIMEDIA '01.

[60]  Vikram S. Adve,et al.  Using likely invariants for automated software fault localization , 2013, ASPLOS '13.

[61]  Zvonimir Rakamaric,et al.  Hybrid learning: interface generation through static, dynamic, and symbolic analysis , 2013, ISSTA.

[62]  K. E. Kannammal,et al.  Software Fault Prediction: A Software Fault Prediction Model by Hybrid Feature Selection and Hybrid Classifier Approach , 2012 .

[63]  David Lo,et al.  Comprehensive evaluation of association measures for fault localization , 2010, 2010 IEEE International Conference on Software Maintenance.

[64]  Claire Le Goues,et al.  A systematic study of automated program repair: Fixing 55 out of 105 bugs for $8 each , 2012, 2012 34th International Conference on Software Engineering (ICSE).

[65]  Fabio Somenzi,et al.  Incremental, Inductive CTL Model Checking , 2012, CAV.

[66]  Peter Zoeteweij,et al.  Spectrum-Based Multiple Fault Localization , 2009, 2009 IEEE/ACM International Conference on Automated Software Engineering.

[67]  Andreas Zeller,et al.  Why Programs Fail: A Guide to Systematic Debugging , 2005 .

[68]  Earl T. Barr,et al.  Casper: Debugging Null Dereferences with Ghosts and Causality Traces , 2015 .

[69]  Greg Schohn,et al.  Less is More: Active Learning with Support Vector Machines , 2000, ICML.

[70]  Koushik Sen,et al.  DART: directed automated random testing , 2005, PLDI '05.

[71]  H. Cleve,et al.  Locating causes of program failures , 2005, Proceedings. 27th International Conference on Software Engineering, 2005. ICSE 2005..