Solving complex path conditions through heuristic search on induced polytopes

Test input generators using symbolic and concolic execution must solve path conditions to systematically explore a program and generate high coverage tests. However, path conditions may contain complicated arithmetic constraints that are infeasible to solve: a solver may be unavailable, solving may be computationally intractable, or the constraints may be undecidable. Existing test generators either simplify such constraints with concrete values to make them decidable, or rely on strong but incomplete constraint solvers. Unfortunately, simplification yields coarse approximations whose solutions rarely satisfy the original constraint. Moreover, constraint solvers cannot handle calls to native library methods. We show how a simple combination of linear constraint solving and heuristic search can overcome these limitations. We call this technique Concolic Walk. On a corpus of 11 programs, an instance of our Concolic Walk algorithm using tabu search generates tests with two- to three-times higher coverage than simplification-based tools while being up to five-times as efficient. Furthermore, our algorithm improves the coverage of two state-of-the-art test generators by 21% and 32%. Other concolic and symbolic testing tools could integrate our algorithm to solve complex path conditions without having to sacrifice any of their own capabilities, leading to higher overall coverage.

[1]  Matthew B. Dwyer,et al.  Green: reducing, reusing and recycling constraints in program analysis , 2012, SIGSOFT FSE.

[2]  GodefroidPatrice Higher-order test generation , 2011 .

[3]  Michael D. Ernst,et al.  Randoop: feedback-directed random testing for Java , 2007, OOPSLA '07.

[4]  Marcelo d'Amorim,et al.  Symbolic Execution with Interval Solving and Meta-heuristic Search , 2012, 2012 IEEE Fifth International Conference on Software Testing, Verification and Validation.

[5]  Gordon Fraser,et al.  Combining search-based and constraint-based testing , 2011, 2011 26th IEEE/ACM International Conference on Automated Software Engineering (ASE 2011).

[6]  Robert J. Simmons,et al.  Proofs from Tests , 2008, IEEE Transactions on Software Engineering.

[7]  Nikolai Tillmann,et al.  Fitness-guided path exploration in dynamic symbolic execution , 2009, 2009 IEEE/IFIP International Conference on Dependable Systems & Networks.

[8]  Dawson R. Engler,et al.  EXE: Automatically Generating Inputs of Death , 2008, TSEC.

[9]  Sarfraz Khurshid,et al.  Test input generation with java PathFinder , 2004, ISSTA '04.

[10]  Alex Groce,et al.  Comparing non-adequate test suites using coverage criteria , 2013, ISSTA.

[11]  Martin D. Davis Hilbert's Tenth Problem is Unsolvable , 1973 .

[12]  Nikolai Tillmann,et al.  Precise identification of problems for structural test generation , 2011, 2011 33rd International Conference on Software Engineering (ICSE).

[13]  Tao Xie,et al.  Improving Structural Testing of Object-Oriented Programs via Integrating Evolutionary Testing and Symbolic Execution , 2008, 2008 23rd IEEE/ACM International Conference on Automated Software Engineering.

[14]  Corina S. Pasareanu,et al.  Symbolic PathFinder: integrating symbolic execution with model checking for Java bytecode analysis , 2013, Automated Software Engineering.

[15]  Gordon Fraser,et al.  EvoSuite: automatic test suite generation for object-oriented software , 2011, ESEC/FSE '11.

[16]  Andreas Zeller,et al.  Generating test cases for specification mining , 2010, ISSTA '10.

[17]  Philippe Codognet,et al.  Yet Another Local Search Method for Constraint Solving , 2001, SAGA.

[18]  Patrice Godefroid Higher-order test generation , 2011, PLDI '11.

[19]  Arnaud Gotlieb,et al.  A uniform random test data generator for path testing , 2010, J. Syst. Softw..

[20]  Koushik Sen DART: Directed Automated Random Testing , 2009, Haifa Verification Conference.

[21]  Gordon Fraser,et al.  Improving search-based test suite generation with dynamic symbolic execution , 2013, 2013 IEEE 24th International Symposium on Software Reliability Engineering (ISSRE).

[22]  Bogdan Korel,et al.  Automated Software Test Data Generation , 1990, IEEE Trans. Software Eng..

[23]  Dawson R. Engler,et al.  KLEE: Unassisted and Automatic Generation of High-Coverage Tests for Complex Systems Programs , 2008, OSDI.

[24]  Marcelo d'Amorim,et al.  CORAL: Solving Complex Constraints for Symbolic PathFinder , 2011, NASA Formal Methods.

[25]  Corina S. Pasareanu,et al.  Symbolic execution with mixed concrete-symbolic solving , 2011, ISSTA '11.

[26]  James Kennedy,et al.  Particle swarm optimization , 1995, Proceedings of ICNN'95 - International Conference on Neural Networks.

[27]  Phil McMinn,et al.  Search‐based software test data generation: a survey , 2004, Softw. Test. Verification Reliab..

[28]  Alessandro Orso,et al.  Automated Behavioral Regression Testing , 2010, 2010 Third International Conference on Software Testing, Verification and Validation.

[29]  Thomas A. Henzinger,et al.  SYNERGY: a new algorithm for property checking , 2006, SIGSOFT '06/FSE-14.

[30]  John Hatcliff,et al.  Kiasan: A Verification and Test-Case Generation Framework for Java Based on Symbolic Execution , 2006, Second International Symposium on Leveraging Applications of Formal Methods, Verification and Validation (isola 2006).

[31]  Fred W. Glover,et al.  Tabu Search - Part I , 1989, INFORMS J. Comput..

[32]  Fred Glover,et al.  Tabu Search - Part II , 1989, INFORMS J. Comput..

[33]  Nikolai Tillmann,et al.  eXpress: guided path exploration for efficient regression test generation , 2011, ISSTA '11.

[34]  Marcelo d'Amorim,et al.  A Comparative Study of Randomized Constraint Solvers for Random-Symbolic Testing , 2009, NASA Formal Methods.

[35]  Koushik Sen,et al.  CUTE and jCUTE: Concolic Unit Testing and Explicit Path Model-Checking Tools , 2006, CAV.

[36]  Nikolai Tillmann,et al.  Pex-White Box Test Generation for .NET , 2008, TAP.

[37]  Sarfraz Khurshid,et al.  Generalized Symbolic Execution for Model Checking and Testing , 2003, TACAS.

[38]  Arnaud Gotlieb,et al.  Path-oriented random testing , 2006, RT '06.

[39]  Mark Harman,et al.  FloPSy - Search-Based Floating Point Constraint Solving for Symbolic Execution , 2010, ICTSS.

[40]  Paolo Tonella,et al.  Symbolic search-based testing , 2011, 2011 26th IEEE/ACM International Conference on Automated Software Engineering (ASE 2011).

[41]  Marcelo d'Amorim,et al.  Randomized constraint solvers: a comparative study , 2010, Innovations in Systems and Software Engineering.

[42]  Moni Naor,et al.  NASA Formal Methods , 2016, Lecture Notes in Computer Science.

[43]  Gregory Tassey,et al.  Prepared for what , 2007 .

[44]  Sarfraz Khurshid,et al.  Memoise: A tool for memoized symbolic execution , 2013, 2013 35th International Conference on Software Engineering (ICSE).

[45]  Nikolaj Bjørner,et al.  Z3: An Efficient SMT Solver , 2008, TACAS.