Efficient Leveraging of Symbolic Execution to Advanced Coverage Criteria

Automatic test data generation (ATG) is a major topic in software engineering. In this paper, we bridge the gap between the coverage criteria supported by state-of-the-art white-box ATG technologies, especially Dynamic Symbolic Execution, and advanced coverage criteria found in the literature. We define a new testing criterion, label coverage, and prove it to be both expressive and amenable to efficient automation. We propose several innovative techniques resulting in an effective blackbox support for label coverage, while a direct approach induces an exponential blow-up of the search space. Experiments show that our optimisations yield very significant savings allowing to leverage ATG to label coverage with only a slight overhead.

[1]  Franz Wotawa,et al.  Using Constraints for Equivalent Mutant Detection , 2012, WS-FMDS.

[2]  Koushik Sen,et al.  Heuristics for Scalable Dynamic Test Generation , 2008, 2008 23rd IEEE/ACM International Conference on Automated Software Engineering.

[3]  Mike Papadakis,et al.  Automatically performing weak mutation with the aid of symbolic execution, concolic testing and search-based testing , 2011, Software Quality Journal.

[4]  Patrice Godefroid,et al.  SAGE: Whitebox Fuzzing for Security Testing , 2012, ACM Queue.

[5]  A. Jefferson Offutt,et al.  MuJava: a mutation system for java , 2006, ICSE.

[6]  Mike Papadakis,et al.  An Effective Path Selection Strategy for Mutation Testing , 2009, 2009 16th Asia-Pacific Software Engineering Conference.

[7]  Philippe Herrmann,et al.  Pruning the Search Space in Path-Based Test Generation , 2009, 2009 International Conference on Software Testing Verification and Validation.

[8]  Nikolai Kosmatov,et al.  Program slicing enhances a verification technique combining static and dynamic analysis , 2012, SAC '12.

[9]  René Just,et al.  Do Redundant Mutants Affect the Effectiveness and Efficiency of Mutation Analysis? , 2012, 2012 IEEE Fifth International Conference on Software Testing, Verification and Validation.

[10]  Patrice Godefroid,et al.  Automated Whitebox Fuzz Testing , 2008, NDSS.

[11]  Bruno Marre,et al.  On-the-fly generation of k-path tests for C functions , 2004 .

[12]  Dawson R. Engler,et al.  EXE: automatically generating inputs of death , 2006, CCS '06.

[13]  William E. Howden,et al.  Weak Mutation Testing and Completeness of Test Sets , 1982, IEEE Transactions on Software Engineering.

[14]  Koushik Sen,et al.  CUTE: a concolic unit testing engine for C , 2005, ESEC/FSE-13.

[15]  A. Jefferson Offutt,et al.  Introduction to Software Testing , 2008 .

[16]  A. Jefferson Offutt,et al.  Mutation 2000: uniting the orthogonal , 2001 .

[17]  Koushik Sen,et al.  DART: directed automated random testing , 2005, PLDI '05.

[18]  A. Jefferson Offutt,et al.  Using compiler optimization techniques to detect equivalent mutants , 1994, Softw. Test. Verification Reliab..

[19]  Daniel Kroening,et al.  A Tool for Checking ANSI-C Programs , 2004, TACAS.

[20]  Bruno Marre,et al.  On-the-fly generation of k-path tests for C functions , 2004, Proceedings. 19th International Conference on Automated Software Engineering, 2004..

[21]  Dawson R. Engler,et al.  KLEE: Unassisted and Automatic Generation of High-Coverage Tests for Complex Systems Programs , 2008, OSDI.

[22]  Hong Zhu,et al.  Software unit test coverage and adequacy , 1997, ACM Comput. Surv..

[23]  A. Jefferson Offutt,et al.  Mutation analysis using mutant schemata , 1993, ISSTA '93.

[24]  Bogdan Korel,et al.  Assertion-oriented automated test data generation , 1996, Proceedings of IEEE 18th International Conference on Software Engineering.

[25]  George C. Necula,et al.  CIL: Intermediate Language and Tools for Analysis and Transformation of C Programs , 2002, CC.

[26]  A. Jefferson Offutt,et al.  An Empirical Evaluation of Weak Mutation , 1994, IEEE Trans. Software Eng..

[27]  W. Eric Wong,et al.  Reducing the cost of mutation testing: An empirical study , 1995, J. Syst. Softw..

[28]  Nikolai Kosmatov,et al.  Automating structural testing of C programs: Experience with PathCrawler , 2009, 2009 ICSE Workshop on Automation of Software Test.

[29]  Philippe Herrmann,et al.  OSMOSE: automatic structural testing of executables , 2011, Softw. Test. Verification Reliab..

[30]  A. Jefferson Offutt,et al.  Investigations of the software testing coupling effect , 1992, TSEM.

[31]  Dawson R. Engler,et al.  EXE: automatically generating inputs of death , 2006, CCS '06.

[32]  James C. King,et al.  Symbolic execution and program testing , 1976, CACM.

[33]  Nikolai Tillmann,et al.  Fitness-guided path exploration in dynamic symbolic execution , 2009, 2009 IEEE/IFIP International Conference on Dependable Systems & Networks.

[34]  Patrice Godefroid,et al.  Active property checking , 2008, EMSOFT '08.

[35]  Mike Papadakis,et al.  Towards automating the generation of mutation tests , 2010, AST '10.

[36]  A Jeeerson Ooutt,et al.  Subsumption of Condition Coverage Techniques by Mutation Testing , 1996 .

[37]  Richard J. Lipton,et al.  Hints on Test Data Selection: Help for the Practicing Programmer , 1978, Computer.

[38]  Morgan B Kaufmann,et al.  Mutation Testing for the New Century , 2002, Computer.

[39]  Nikolai Tillmann,et al.  Pex-White Box Test Generation for .NET , 2008, TAP.

[40]  Gregg Rothermel,et al.  An experimental evaluation of selective mutation , 1993, Proceedings of 1993 15th International Conference on Software Engineering.

[41]  Gordon Fraser,et al.  Generating Test Suites with Augmented Dynamic Symbolic Execution , 2013, TAP@STAF.

[42]  A. Jefferson Offutt,et al.  Constraint-Based Automatic Test Data Generation , 1991, IEEE Trans. Software Eng..

[43]  Lionel C. Briand,et al.  Is mutation an appropriate tool for testing experiments? , 2005, ICSE.

[44]  Nikolai Tillmann,et al.  Guided test generation for coverage criteria , 2010, 2010 IEEE International Conference on Software Maintenance.

[45]  Nikolai Tillmann,et al.  Test generation via Dynamic Symbolic Execution for mutation testing , 2010, 2010 IEEE International Conference on Software Maintenance.

[46]  Mike Papadakis,et al.  Automatic Mutation Test Case Generation via Dynamic Symbolic Execution , 2010, 2010 IEEE 21st International Symposium on Software Reliability Engineering.

[47]  Nikolai Tillmann,et al.  Automating Software Testing Using Program Analysis , 2008, IEEE Software.