Combining Search-Based Testing and Dynamic Symbolic Execution by Evolvability Metric

In the area of software testing, search-based software testing (SBST) and dynamic symbolic execution (DSE) are two efficient testing techniques for test cases generation. However, both of the two approaches have their own drawbacks: The efficiency of SBST depends on the guidance of the fitness landscape. When the fitness landscape has some plateaus with no gradient for directing search process, SBST may degenerate into random testing. DSE relies on the capability of constraint solvers. It may struggle to generate test cases with constraints that are difficult to be solved. In this paper, we combine the strengths of both techniques. SBST is used to help DSE for solving difficult constraints and DSE is used to improve the efficiency and capability of SBST. Evolvability metric is introduced for measuring when the software is not suitable for SBST. A novel switch mechanism based on the evolvability metric between SBST and DSE is proposed in this paper to help to choose the proper technique at the proper time. Experiments on several benchmarks reveal the promising results of our proposal.

[1]  James Miller,et al.  Automatic test data generation using genetic algorithm and program dependence graphs , 2006, Inf. Softw. Technol..

[2]  H. B. Mann,et al.  On a Test of Whether one of Two Random Variables is Stochastically Larger than the Other , 1947 .

[3]  J. L. Dalley.  The art of software testing , 1991, Proceedings of the IEEE 1991 National Aerospace and Electronics Conference NAECON 1991.

[4]  Xavier Lorca,et al.  Choco: an Open Source Java Constraint Programming Library , 2008 .

[5]  Patrice Godefroid,et al.  Precise pointer reasoning for dynamic test generation , 2009, ISSTA.

[6]  Xiong Xu,et al.  An adaptive fitness function based on branch hardness for search based testing , 2017, GECCO.

[7]  Paolo Tonella,et al.  Symbolic search-based testing , 2011, 2011 26th IEEE/ACM International Conference on Automated Software Engineering (ASE 2011).

[8]  Lionel C. Briand,et al.  A practical guide for using statistical tests to assess randomized algorithms in software engineering , 2011, 2011 33rd International Conference on Software Engineering (ICSE).

[9]  Corina S. Pasareanu,et al.  Test input generation for java containers using state matching , 2006, ISSTA '06.

[10]  Johannes Mayer,et al.  Towards the determination of typical failure patterns , 2007, SOQUA '07.

[11]  Gordon Fraser,et al.  Whole Test Suite Generation , 2013, IEEE Transactions on Software Engineering.

[12]  Gordon Fraser,et al.  EvoSuite: automatic test suite generation for object-oriented software , 2011, ESEC/FSE '11.

[13]  Dong Yan,et al.  The Floating-Point Extension of Symbolic Execution Engine for Bug Detection , 2016, 2016 23rd Asia-Pacific Software Engineering Conference (APSEC).

[14]  Lionel C. Briand,et al.  Search-Driven String Constraint Solving for Vulnerability Detection , 2017, 2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE).

[15]  Lori A. Clarke,et al.  A System to Generate Test Data and Symbolically Execute Programs , 1976, IEEE Transactions on Software Engineering.

[16]  Jun Yan,et al.  Test Data Generation for C Programs with String-Handling Functions , 2008, 2008 2nd IFIP/IEEE International Symposium on Theoretical Aspects of Software Engineering.

[17]  Mark Harman,et al.  FloPSy - Search-Based Floating Point Constraint Solving for Symbolic Execution , 2010, ICTSS.

[18]  Arnaud Gotlieb,et al.  Combining Genetic Algorithms and Constraint Programming to Support Stress Testing of Task Deadlines , 2015, ACM Trans. Softw. Eng. Methodol..

[19]  Giovanni Denaro,et al.  Combining symbolic execution and search-based testing for programs with complex heap inputs , 2017, ISSTA.

[20]  A. Vargha,et al.  A Critique and Improvement of the CL Common Language Effect Size Statistics of McGraw and Wong , 2000 .

[21]  Phil McMinn,et al.  Search-based software test data generation: a survey: Research Articles , 2004 .

[22]  Marcelo d'Amorim,et al.  CORAL: Solving Complex Constraints for Symbolic PathFinder , 2011, NASA Formal Methods.

[23]  Nikolai Tillmann,et al.  Pex-White Box Test Generation for .NET , 2008, TAP.

[24]  Gul A. Agha,et al.  Solving complex path conditions through heuristic search on induced polytopes , 2014, FSE 2014.

[25]  Mark Harman,et al.  Testability transformation , 2004, IEEE Transactions on Software Engineering.

[26]  Mark Harman,et al.  Improving Evolutionary Testing By Flag Removal , 2002, GECCO.

[27]  Nikolai Tillmann,et al.  Fitness-guided path exploration in dynamic symbolic execution , 2009, 2009 IEEE/IFIP International Conference on Dependable Systems & Networks.

[28]  Leonardo Bottaci.  Instrumenting Programs With Flag Variables For Test Data Search By Genetic Algorithms , 2002, GECCO.

[29]  Koushik Sen.  DART: Directed Automated Random Testing , 2009, Haifa Verification Conference.

[30]  Gordon Fraser,et al.  Improving search-based test suite generation with dynamic symbolic execution , 2013, 2013 IEEE 24th International Symposium on Software Reliability Engineering (ISSRE).

[31]  Gordon Fraser,et al.  Extending a search-based test generator with adaptive dynamic symbolic execution , 2014, ISSTA 2014.

[32]  Marcelo d'Amorim,et al.  Symbolic Execution with Interval Solving and Meta-heuristic Search , 2012, 2012 IEEE Fifth International Conference on Software Testing, Verification and Validation.

[33]  Darko Marinov,et al.  A Comparison of Constraint-Based and Sequence-Based Generation of Complex Input Data Structures , 2010, 2010 Third International Conference on Software Testing, Verification, and Validation Workshops.

[34]  Yann-Gaël Guéhéneuc,et al.  Constraint-Based Fitness Function for Search-Based Software Testing , 2013, CPAIOR.

[35]  Dawson R. Engler,et al.  EXE: Automatically Generating Inputs of Death , 2008, TSEC.

[36]  Koushik Sen,et al.  CUTE: a concolic unit testing engine for C , 2005, ESEC/FSE-13.

[37]  Gordon Fraser,et al.  Combining search-based and constraint-based testing , 2011, 2011 26th IEEE/ACM International Conference on Automated Software Engineering (ASE 2011).

[38]  Mark Harman,et al.  AUSTIN: An open source tool for search based software testing of C programs , 2013, Inf. Softw. Technol..

[39]  Corina S. Pasareanu,et al.  Parallel symbolic execution for structural test generation , 2010, ISSTA '10.

[40]  Paolo Tonella,et al.  Evolutionary testing of classes , 2004, ISSTA '04.

[41]  Yann-Gaël Guéhéneuc,et al.  Boosting Search Based Testing by Using Constraint Based Testing , 2012, SSBSE.

[42]  Marcelo F. Frias,et al.  Analysis of invariants for efficient bounded verification , 2010, ISSTA '10.