FSX: fine-grained incremental unit test generation for C/C++ programs

Automated unit test generation bears the promise of significantly reducing test cost and hence improving software quality. However, the maintenance cost of the automatically generated tests presents a significant barrier to adoption of this technology. To address this challenge, we propose a novel technique for automated and fine-grained incremental generation of unit tests through minimal augmentation of an existing test suite. The technique uses iterative, incremental refinement of test-drivers and symbolic execution, guided by a diagnostics engine. The diagnostics engine works off a novel precise and efficient byte-level dynamic dependence analysis built using Reduced Ordered Binary Decision Diagrams (ROBDDs). We present a tool FSX implementing this technique and evaluate it under two practical use-cases of incremental unit test generation, on five revisions of the open-source software iPerf, as well as on 3 large subjects, comprising more than 60 thousand lines of code, from in-house commercial network products. The evaluation shows that FSX can generate high-quality unit tests on large industrial software while minimizing the maintenance cost of the overall test-suite.

[1]  Dawson R. Engler,et al.  KLEE: Unassisted and Automatic Generation of High-Coverage Tests for Complex Systems Programs , 2008, OSDI.

[2]  Rui Abreu,et al.  Continuous test generation: enhancing continuous integration with automated test generation , 2014, ASE.

[3]  Nikolai Tillmann,et al.  eXpress: guided path exploration for efficient regression test generation , 2011, ISSTA '11.

[4]  Moonzoo Kim,et al.  Automated unit testing of large industrial embedded software using concolic testing , 2013, 2013 28th IEEE/ACM International Conference on Automated Software Engineering (ASE).

[5]  Tsong Yueh Chen,et al.  Quasi-Random Testing , 2005, IEEE Transactions on Reliability.

[6]  Corina S. Pasareanu,et al.  A survey of new trends in symbolic execution for software testing and analysis , 2009, International Journal on Software Tools for Technology Transfer.

[7]  Guodong Li,et al.  JST: An automatic test generation tool for industrial Java applications with strings , 2013, 2013 35th International Conference on Software Engineering (ICSE).

[8]  Debra J. Richardson,et al.  Structural specification-based testing: automated support and experimental evaluation , 1999, ESEC/FSE-7.

[9]  Michael D. Ernst,et al.  Randoop: feedback-directed random testing for Java , 2007, OOPSLA '07.

[10]  Vikram S. Adve,et al.  LLVM: a compilation framework for lifelong program analysis & transformation , 2004, International Symposium on Code Generation and Optimization, 2004. CGO 2004..

[11]  Patrice Godefroid,et al.  Automated Whitebox Fuzz Testing , 2008, NDSS.

[12]  Myra B. Cohen,et al.  Hybrid Directed Test Suite Augmentation: An Interleaving Framework , 2014, 2014 IEEE Seventh International Conference on Software Testing, Verification and Validation.

[13]  Randal E. Bryant,et al.  Symbolic Boolean manipulation with ordered binary-decision diagrams , 1992, CSUR.

[14]  Sarfraz Khurshid,et al.  Directed incremental symbolic execution , 2011, PLDI '11.

[15]  Corina S. Pasareanu,et al.  Symbolic PathFinder: symbolic execution of Java bytecode , 2010, ASE.

[16]  Gregory Tassey,et al.  Prepared for what , 2007 .

[17]  Guodong Li,et al.  SymJS: automatic symbolic testing of JavaScript web applications , 2014, SIGSOFT FSE.

[18]  Koushik Sen,et al.  CUTE: a concolic unit testing engine for C , 2005, ESEC/FSE-13.

[19]  Andreas Zeller,et al.  Mutation-Driven Generation of Unit Tests and Oracles , 2012, IEEE Trans. Software Eng..

[20]  Phil McMinn,et al.  Search‐based software test data generation: a survey , 2004, Softw. Test. Verification Reliab..

[21]  Nikolai Tillmann,et al.  Parameterized unit tests , 2005, ESEC/FSE-13.

[22]  Koushik Sen,et al.  DART: directed automated random testing , 2005, PLDI '05.

[23]  Koushik Sen,et al.  CUTE and jCUTE: Concolic Unit Testing and Explicit Path Model-Checking Tools , 2006, CAV.

[24]  David L. Dill,et al.  A Decision Procedure for Bit-Vectors and Arrays , 2007, CAV.

[25]  Gordon Fraser,et al.  EvoSuite: automatic test suite generation for object-oriented software , 2011, ESEC/FSE '11.

[26]  Koushik Sen,et al.  Generating Succinct Test Cases Using Don't Care Analysis , 2015, 2015 IEEE 8th International Conference on Software Testing, Verification and Validation (ICST).

[27]  A. Zeller,et al.  Mutation-driven Generation of Oracles and Unit Tests , 2010 .

[28]  Phil McMinn,et al.  Search-based software test data generation: a survey: Research Articles , 2004 .

[29]  A. Jefferson Offutt,et al.  Constraint-Based Automatic Test Data Generation , 1991, IEEE Trans. Software Eng..

[30]  Myra B. Cohen,et al.  Factors affecting the use of genetic algorithms in test suite augmentation , 2010, GECCO '10.

[31]  Myra B. Cohen,et al.  Directed test suite augmentation: techniques and tradeoffs , 2010, FSE '10.

[32]  Nikolai Tillmann,et al.  Pex-White Box Test Generation for .NET , 2008, TAP.

[33]  Sarfraz Khurshid,et al.  Memoized symbolic execution , 2012, ISSTA 2012.

[34]  Zhihong Xu,et al.  Directed test suite augmentation , 2011, 2011 33rd International Conference on Software Engineering (ICSE).

[35]  Rupak Majumdar,et al.  Reducing Test Inputs Using Information Partitions , 2009, CAV.

[36]  S. L. Gerhart,et al.  Toward a theory of test data selection , 1975, IEEE Transactions on Software Engineering.

[37]  Alessandro Orso,et al.  Test-Suite Augmentation for Evolving Software , 2008, 2008 23rd IEEE/ACM International Conference on Automated Software Engineering.

[38]  Nikolaj Bjørner,et al.  Z3: An Efficient SMT Solver , 2008, TACAS.

[39]  Sigrid Eldh Software Testing Techniques , 2007 .

[40]  Michael F. P. O'Boyle,et al.  ACM SIGPLAN Conference on Programming Language Design and Implementation, PLDI '14, Edinburgh, United Kingdom - June 09 - 11, 2014 , 2014, ACM-SIGPLAN Symposium on Programming Language Design and Implementation.

[41]  Zhenkai Liang,et al.  BitBlaze: A New Approach to Computer Security via Binary Analysis , 2008, ICISS.

[42]  Guodong Li,et al.  KLOVER: A Symbolic Execution and Automatic Test Generation Tool for C++ Programs , 2011, CAV.

[43]  Sarfraz Khurshid,et al.  Symbolic execution for software testing in practice: preliminary assessment , 2011, 2011 33rd International Conference on Software Engineering (ICSE).