On test repair using symbolic execution

When developers change a program, regression tests can fail not only due to faults in the program but also due to out-of-date test code that does not reflect the desired behavior of the program. When this occurs, it is necessary to repair test code such that the tests pass. Repairing tests manually is difficult and time consuming. We recently developed ReAssert, a tool that can automatically repair broken unit tests, but only if they lack complex control flow or operations on expected values. This paper introduces symbolic test repair, a technique based on symbolic execution, which can overcome some of ReAssert's limitations. We reproduce experiments from earlier work and find that symbolic test repair improves upon previously reported results both quantitatively and qualitatively. We also perform new experiments which confirm the benefits of symbolic test repair and also show surprising similarities in test failures for open-source Java and .NET programs. Our experiments use Pex, a powerful symbolic execution engine for .NET, and we find that Pex provides over half of the repairs possible from the theoretically ideal symbolic test repair.

[1]  Dawson R. Engler,et al.  KLEE: Unassisted and Automatic Generation of High-Coverage Tests for Complex Systems Programs , 2008, OSDI.

[2]  Atif M. Memon,et al.  Automatically repairing event sequence-based GUI test suites for regression testing , 2008, TSEM.

[3]  Patrice Godefroid,et al.  Compositional dynamic test generation , 2007, POPL '07.

[4]  Xiangyu Zhang,et al.  Avoiding Program Failures Through Safe Execution Perturbations , 2008, 2008 32nd Annual IEEE International Computer Software and Applications Conference.

[5]  David S. Rosenblum,et al.  Algorithms for efficient symbolic detection of faults in context-aware applications , 2008, 2008 23rd IEEE/ACM International Conference on Automated Software Engineering - Workshops.

[6]  Arie van Deursen,et al.  On the Interplay Between Software Testing and Evolution and its Effect on Program Comprehension , 2008, Software Evolution.

[7]  Corina S. Pasareanu,et al.  JPF-SE: A Symbolic Execution Extension to Java PathFinder , 2007, TACAS.

[8]  Marat Boshernitsan,et al.  From daikon to agitator: lessons and challenges in building a commercial tool for developer testing , 2006, ISSTA '06.

[9]  Nikolai Tillmann,et al.  DySy: dynamic symbolic execution for invariant inference , 2008, ICSE.

[10]  Andreas Zeller.  Automated Debugging: Are We Close , 2001, Computer.

[11]  Michael D. Ernst,et al.  Automatic creation of SQL Injection and cross-site scripting attacks , 2009, 2009 IEEE 31st International Conference on Software Engineering.

[12]  Koushik Sen,et al.  CUTE: a concolic unit testing engine for C , 2005, ESEC/FSE-13.

[13]  Zhendong Su,et al.  Context-aware statistical debugging: from bug predictors to faulty control flow paths , 2007, ASE.

[14]  Koushik Sen,et al.  Concolic testing , 2007, ASE.

[15]  Darko Marinov,et al.  ReAssert: Suggesting Repairs for Broken Unit Tests , 2009, 2009 IEEE/ACM International Conference on Automated Software Engineering.

[16]  Koushik Sen,et al.  DART: directed automated random testing , 2005, PLDI '05.

[17]  H. Cleve,et al.  Locating causes of program failures , 2005, Proceedings. 27th International Conference on Software Engineering, 2005. ICSE 2005..

[18]  Nikolai Tillmann,et al.  Pex-White Box Test Generation for .NET , 2008, TAP.

[19]  Mary Jean Harrold,et al.  An empirical study of the effects of test-suite reduction on fault localization , 2008, 2008 ACM/IEEE 30th International Conference on Software Engineering.

[20]  Xin Yao,et al.  A novel co-evolutionary approach to automatic software bug fixing , 2008, 2008 IEEE Congress on Evolutionary Computation (IEEE World Congress on Computational Intelligence).

[21]  Rajiv Gupta,et al.  A methodology for controlling the size of a test suite , 1990, Proceedings. Conference on Software Maintenance 1990.

[22]  Rajiv Gupta,et al.  BugFix: A learning-based tool to assist developers in fixing bugs , 2009, 2009 IEEE 17th International Conference on Program Comprehension.

[23]  Dawson R. Engler,et al.  EXE: automatically generating inputs of death , 2006, CCS '06.

[24]  Nikolai Tillmann,et al.  Parameterized Unit Testing with Pex , 2008, TAP.

[25]  Adam Kiezun,et al.  Grammar-based whitebox fuzzing , 2008, PLDI '08.

[26]  Mei-Hwa Chen,et al.  Maintaining Multi-Tier Web Applications , 2007, 2007 IEEE International Conference on Software Maintenance.

[27]  Leonardo Mariani,et al.  In-field healing of integration problems with COTS components , 2009, 2009 IEEE 31st International Conference on Software Engineering.

[28]  Frank Tip,et al.  A survey of program slicing techniques , 1994, J. Program. Lang..

[29]  Koushik Sen,et al.  Universal symbolic execution and its application to likely data structure invariant generation , 2008, ISSTA '08.

[30]  Shing-Chi Cheung,et al.  Taming coincidental correctness: Coverage refinement with context patterns to improve fault localization , 2009, 2009 IEEE 31st International Conference on Software Engineering.

[31]  Nikolaj Bjørner,et al.  Z3: An Efficient SMT Solver , 2008, TACAS.

[32]  James C. King,et al.  Symbolic execution and program testing , 1976, CACM.

[33]  Oscar H. Ibarra,et al.  Symbolic String Verification: Combining String Analysis and Size Analysis , 2009, TACAS.

[34]  Rupak Majumdar,et al.  Testing for buffer overflows with length abstraction , 2008, ISSTA '08.

[35]  Zhenkai Liang,et al.  Towards Automatic Discovery of Deviations in Binary Implementations with Applications to Error Detection and Fingerprint Generation , 2007, USENIX Security Symposium.

[36]  Leonardo Mariani,et al.  Self-healing strategies for component integration faults , 2008, 2008 23rd IEEE/ACM International Conference on Automated Software Engineering - Workshops.

[37]  Tao Xie,et al.  Time-aware test-case prioritization using integer linear programming , 2009, ISSTA.

[38]  Nikolai Tillmann,et al.  Parameterized unit tests , 2005, ESEC/FSE-13.

[39]  Rupak Majumdar,et al.  Hybrid Concolic Testing , 2007, 29th International Conference on Software Engineering (ICSE'07).

[40]  Wayne Zachary,et al.  A Tool for Automatically Generating Transcripts of Human-Computer Interaction , 1993 .

[41]  Claire Le Goues,et al.  Automatically finding patches using genetic programming , 2009, 2009 IEEE 31st International Conference on Software Engineering.

[42]  Rajiv Gupta,et al.  Fault localization using value replacement , 2008, ISSTA '08.

[43]  Frank Tip,et al.  JUnitMX - A change-aware unit testing tool , 2009, 2009 IEEE 31st International Conference on Software Engineering.

[44]  Neelam Gupta,et al.  Automated Debugging Using Path-Based Weakest Preconditions , 2004, FASE.

[45]  Chen Fu,et al.  Maintaining and evolving GUI-directed test scripts , 2009, 2009 IEEE 31st International Conference on Software Engineering.

[46]  Andreas Zeller,et al.  Efficient mutation testing by checking invariant violations , 2009, ISSTA.

[47]  Willem Visser,et al.  Variably interprocedural program analysis for runtime error detection , 2007, ISSTA '07.

[48]  Shujuan Jiang,et al.  Fault localization and repair for Java runtime exceptions , 2009, ISSTA.

[49]  Mark Harman,et al.  Automated Session Data Repair for Web Application Regression Testing , 2008, 2008 1st International Conference on Software Testing, Verification, and Validation.

[50]  Marat Boshernitsan,et al.  Predicting Effectiveness of Automatic Testing Tools , 2008, 2008 23rd IEEE/ACM International Conference on Automated Software Engineering.

[51]  Michael R. Lowry,et al.  Combining unit-level symbolic execution and system-level concrete execution for testing nasa software , 2008, ISSTA '08.