make test-zesti: A symbolic execution solution for improving regression testing

Software testing is an expensive and time consuming process, often involving the manual creation of comprehensive regression test suites. However, current testing methodologies do not take full advantage of these tests. In this paper, we present a technique for amplifying the effect of existing test suites using a lightweight symbolic execution mechanism, which thoroughly checks all sensitive operations (e.g., pointer dereferences) executed by the test suite for errors, and explores additional paths around sensitive operations. We implemented this technique in a prototype system called ZESTI (Zero-Effort Symbolic Test Improvement), and applied it to three open-source code bases - GNU Coreutils, libdwarf and readelf - where it found 52 previously unknown bugs, many of which are out of reach of standard symbolic execution. Our technique works transparently to the tester, requiring no additional human effort or changes to source code or tests.

[1]  Alessandro Orso,et al.  Test-Suite Augmentation for Evolving Software , 2008, 2008 23rd IEEE/ACM International Conference on Automated Software Engineering.

[2]  Andreas Zeller,et al.  Simplifying and Isolating Failure-Inducing Input , 2002, IEEE Trans. Software Eng..

[3]  Patrice Godefroid,et al.  Compositional dynamic test generation , 2007, POPL '07.

[4]  Robert O. Hastings,et al.  Fast detection of memory leaks and access errors , 1991 .

[5]  Corina S. Pasareanu,et al.  JPF-SE: A Symbolic Execution Extension to Java PathFinder , 2007, TACAS.

[6]  George Candea,et al.  Parallel symbolic execution for automated real-world software testing , 2011, EuroSys '11.

[7]  Zhenkai Liang,et al.  BitBlaze: A New Approach to Computer Security via Binary Analysis , 2008, ICISS.

[8]  Frank Tip,et al.  Directed test generation for effective fault localization , 2010, ISSTA '10.

[9]  Koushik Sen,et al.  Predictive testing: amplifying the effectiveness of software testing , 2007, ESEC-FSE '07.

[10]  Alexander Aiken,et al.  Flow-sensitive type qualifiers , 2002, PLDI '02.

[11]  A. W. Kemp,et al.  Univariate Discrete Distributions , 1993 .

[12]  Todd M. Austin,et al.  High Coverage Detection of Input-Related Security Faults , 2003, USENIX Security Symposium.

[13]  Koushik Sen.  DART: Directed Automated Random Testing , 2009, Haifa Verification Conference.

[14]  Junfeng Yang,et al.  Using model checking to find serious file system errors , 2004, TOCS.

[15]  Sarfraz Khurshid,et al.  Symbolic execution for software testing in practice: preliminary assessment , 2011, 2011 33rd International Conference on Software Engineering (ICSE).

[16]  Nicholas Nethercote,et al.  Valgrind: a framework for heavyweight dynamic binary instrumentation , 2007, PLDI '07.

[17]  William R. Bush,et al.  A static analyzer for finding dynamic programming errors , 2000, Softw. Pract. Exp..

[18]  James C. King,et al.  Symbolic execution and program testing , 1976, CACM.

[19]  Dawson R. Engler,et al.  KLEE: Unassisted and Automatic Generation of High-Coverage Tests for Complex Systems Programs , 2008, OSDI.

[20]  Vikram S. Adve,et al.  LLVM: a compilation framework for lifelong program analysis & transformation , 2004, International Symposium on Code Generation and Optimization, 2004. CGO 2004..

[21]  Patrice Godefroid,et al.  Automated Whitebox Fuzz Testing , 2008, NDSS.

[22]  Rupak Majumdar,et al.  Hybrid Concolic Testing , 2007, 29th International Conference on Software Engineering (ICSE'07).

[23]  Dawn Xiaodong Song,et al.  MACE: Model-inference-Assisted Concolic Exploration for Protocol and Vulnerability Discovery , 2011, USENIX Security Symposium.

[24]  Koushik Sen,et al.  CUTE: a concolic unit testing engine for C , 2005, ESEC/FSE-13.

[25]  Amitabh Srivastava,et al.  Effectively prioritizing tests in development environment , 2002, ISSTA '02.

[26]  Audris Mockus,et al.  Predicting risk of software changes , 2000, Bell Labs Technical Journal.

[27]  Mark Harman,et al.  Measuring and Improving Latency to Avoid Test Suite Wear Out , 2009, 2009 International Conference on Software Testing, Verification, and Validation Workshops.

[28]  Sorin Lerner,et al.  ESP: path-sensitive program verification in polynomial time , 2002, PLDI '02.

[29]  Gregg Rothermel,et al.  Analyzing Regression Test Selection Techniques , 1996, IEEE Trans. Software Eng..

[30]  Nikolai Tillmann,et al.  Pex-White Box Test Generation for .NET , 2008, TAP.

[31]  Rajiv Gupta,et al.  A methodology for controlling the size of a test suite , 1990, Proceedings. Conference on Software Maintenance 1990.

[32]  Gerard J. Holzmann,et al.  The Model Checker SPIN , 1997, IEEE Trans. Software Eng..

[33]  Nikolai Tillmann,et al.  Automating Software Testing Using Program Analysis , 2008, IEEE Software.

[34]  J. Davis.  Univariate Discrete Distributions , 2006 .

[35]  A. W. Kemp,et al.  Univariate Discrete Distributions: Johnson/Univariate Discrete Distributions , 2005 .

[36]  Sriram K. Rajamani,et al.  Automatically validating temporal safety properties of interfaces , 2001, SPIN '01.

[37]  Gregory Tassey,et al.  Prepared for what , 2007 .

[38]  David A. Wagner,et al.  A First Step Towards Automated Detection of Buffer Overrun Vulnerabilities , 2000, NDSS.

[39]  Patrice Godefroid,et al.  Model checking for programming languages using VeriSoft , 1997, POPL '97.

[40]  Stephen McCamant,et al.  Statically-directed dynamic automated test generation , 2011, ISSTA '11.

[41]  Nikolai Tillmann,et al.  eXpress: guided path exploration for efficient regression test generation , 2011, ISSTA '11.

[42]  Susan Horwitz,et al.  Incremental program testing using program dependence graphs , 1993, POPL '93.

[43]  Mary Lou Soffa,et al.  Program Slicing‐Based Regression Testing Techniques , 1996 .

[44]  Dawson R. Engler,et al.  RWset: Attacking Path Explosion in Constraint-Based Test Generation , 2008, TACAS.

[45]  Myra B. Cohen,et al.  Directed test suite augmentation: techniques and tradeoffs , 2010, FSE '10.