FSX: a tool for fine-grained incremental unit test generation for C/C++ programs

Automated unit test generation bears the promise of significantly reducing test cost and hence improving software quality. However, the maintenance cost of the automatically generated tests presents a significant barrier to adoption of this technology. To address this challenge, in previous work, we proposed a novel technique for automated and fine-grained incremental generation of unit tests through minimal augmentation of an existing test suite. In this paper we describe a tool FSX, implementing this technique. We describe the architecture, user-interface, and salient features of FSX, and specific practical use-cases of its technology. We also report on a real, large-scale deployment of FSX, as a practical validation of the underlying research contribution and of automated test generation research in general.

[1]  Alessandro Orso,et al.  Test-Suite Augmentation for Evolving Software , 2008, 2008 23rd IEEE/ACM International Conference on Automated Software Engineering.

[2]  Nikolaj Bjørner,et al.  Z3: An Efficient SMT Solver , 2008, TACAS.

[3]  S. L. Gerhart,et al.  Toward a theory of test data selection , 1975, IEEE Transactions on Software Engineering.

[4]  Sigrid Eldh Software Testing Techniques , 2007 .

[5]  Moonzoo Kim,et al.  Automated unit testing of large industrial embedded software using concolic testing , 2013, 2013 28th IEEE/ACM International Conference on Automated Software Engineering (ASE).

[6]  Nikolai Tillmann,et al.  eXpress: guided path exploration for efficient regression test generation , 2011, ISSTA '11.

[7]  Nikolai Tillmann,et al.  Pex-White Box Test Generation for .NET , 2008, TAP.

[8]  Patrice Godefroid,et al.  Automated Whitebox Fuzz Testing , 2008, NDSS.

[9]  Sarfraz Khurshid,et al.  Memoized symbolic execution , 2012, ISSTA 2012.

[10]  Zhihong Xu,et al.  Directed test suite augmentation , 2011, 2011 33rd International Conference on Software Engineering (ICSE).

[11]  Nikolai Tillmann,et al.  Parameterized unit tests , 2005, ESEC/FSE-13.

[12]  Randal E. Bryant,et al.  Symbolic Boolean manipulation with ordered binary-decision diagrams , 1992, CSUR.

[13]  Sarfraz Khurshid,et al.  Directed incremental symbolic execution , 2011, PLDI '11.

[14]  Hiroaki Yoshida,et al.  FSX: fine-grained incremental unit test generation for C/C++ programs , 2016, ISSTA.

[15]  Koushik Sen,et al.  CUTE: a concolic unit testing engine for C , 2005, ESEC/FSE-13.

[16]  Myra B. Cohen,et al.  Hybrid Directed Test Suite Augmentation: An Interleaving Framework , 2014, 2014 IEEE Seventh International Conference on Software Testing, Verification and Validation.

[17]  Corina S. Pasareanu,et al.  Symbolic PathFinder: symbolic execution of Java bytecode , 2010, ASE.

[18]  Corina S. Pasareanu,et al.  A survey of new trends in symbolic execution for software testing and analysis , 2009, International Journal on Software Tools for Technology Transfer.

[19]  Koushik Sen,et al.  DART: directed automated random testing , 2005, PLDI '05.

[20]  Zhenkai Liang,et al.  BitBlaze: A New Approach to Computer Security via Binary Analysis , 2008, ICISS.

[21]  Phil McMinn,et al.  Search‐based software test data generation: a survey , 2004, Softw. Test. Verification Reliab..

[22]  Sarfraz Khurshid,et al.  Symbolic execution for software testing in practice: preliminary assessment , 2011, 2011 33rd International Conference on Software Engineering (ICSE).

[23]  W. Marsden I and J , 2012 .

[24]  Myra B. Cohen,et al.  Directed test suite augmentation: techniques and tradeoffs , 2010, FSE '10.

[25]  Dawson R. Engler,et al.  KLEE: Unassisted and Automatic Generation of High-Coverage Tests for Complex Systems Programs , 2008, OSDI.

[26]  Guodong Li,et al.  SymJS: automatic symbolic testing of JavaScript web applications , 2014, SIGSOFT FSE.

[27]  Guodong Li,et al.  KLOVER: A Symbolic Execution and Automatic Test Generation Tool for C++ Programs , 2011, CAV.

[28]  Gordon Fraser,et al.  EvoSuite: automatic test suite generation for object-oriented software , 2011, ESEC/FSE '11.

[29]  Koushik Sen,et al.  Generating Succinct Test Cases Using Don't Care Analysis , 2015, 2015 IEEE 8th International Conference on Software Testing, Verification and Validation (ICST).

[30]  Phil McMinn,et al.  Search-based software test data generation: a survey: Research Articles , 2004 .

[31]  Koushik Sen,et al.  CUTE and jCUTE: Concolic Unit Testing and Explicit Path Model-Checking Tools , 2006, CAV.

[32]  David L. Dill,et al.  A Decision Procedure for Bit-Vectors and Arrays , 2007, CAV.