Improving Test Generation under Rich Contracts by Tight Bounds and Incremental SAT Solving

We present a novel and general technique for automated test generation that combines tight bounds with incremental SAT solving. The proposed technique uses incremental SAT to build test suites targeting a specific testing criterion, amongst various black-box and white-box criteria. As our experimental results show, the combination of tight bounds with incremental SAT, and the testing criterion driven approach implemented in our prototype tool FAJITA, enable us to effectively generate test suites for container classes with rich contracts, more efficiently than other state-of-the-art tools.

[1]  Emina Torlak,et al.  Kodkod: A Relational Model Finder , 2007, TACAS.

[2]  Michael D. Ernst,et al.  Feedback-Directed Random Test Generation , 2007, 29th International Conference on Software Engineering (ICSE'07).

[3]  Felix Sheng-Ho Chang,et al.  Modular verification of code with SAT , 2006, ISSTA '06.

[4]  Nikolaj Bjørner,et al.  Z3: An Efficient SMT Solver , 2008, TACAS.

[5]  Cem Kaner,et al.  Lessons Learned in Software Testing , 2001 .

[6]  Sarfraz Khurshid,et al.  Whispec: white-box testing of libraries using declarative specifications , 2007, LCSD '07.

[7]  G. G. Stokes "J." , 1890, The New Yale Book of Quotations.

[8]  Daniel Jackson,et al.  Finding bugs with a constraint solver , 2000, ISSTA '00.

[9]  Nazareno Aguirre,et al.  DynAlloy: upgrading alloy with actions , 2005, ICSE.

[10]  Bertrand Meyer,et al.  Reconciling Manual and Automated Testing: The AutoTest Experience , 2007, 2007 40th Annual Hawaii International Conference on System Sciences (HICSS'07).

[11]  Sarfraz Khurshid,et al.  Test generation through programming in UDITA , 2010, 2010 ACM/IEEE 32nd International Conference on Software Engineering.

[12]  Bertrand Meyer,et al.  Using Contracts and Boolean Queries to Improve the Quality of Automatic Test Generation , 2007, TAP.

[13]  Myra B. Cohen,et al.  Constructing Interaction Test Suites for Highly-Configurable Systems in the Presence of Constraints: A Greedy Approach , 2008, IEEE Transactions on Software Engineering.

[14]  Robby,et al.  Sireum/Topi LDP: a lightweight semi-decision procedure for optimizing symbolic execution-based analyses , 2009, ESEC/FSE '09.

[15]  Sarfraz Khurshid,et al.  TestEra: Specification-Based Testing of Java Programs Using SAT , 2004, Automated Software Engineering.

[16]  Hong Zhu,et al.  Software unit test coverage and adequacy , 1997, ACM Comput. Surv..

[17]  Frank Tip,et al.  Finding bugs efficiently with a SAT solver , 2007, ESEC-FSE '07.

[18]  Corina S. Pasareanu,et al.  Test input generation for java containers using state matching , 2006, ISSTA '06.

[19]  Gordon Fraser,et al.  EvoSuite: automatic test suite generation for object-oriented software , 2011, ESEC/FSE '11.

[20]  James M. Crawford,et al.  Symmetry-Breaking Predicates for Search Problems , 1996, KR.

[21]  Marcelo F. Frias,et al.  DynAlloy as a Formal Method for the Analysis of Java Programs , 2006, SET.

[22]  Elaine J. Weyuker,et al.  Analyzing Partition Testing Strategies , 1991, IEEE Trans. Software Eng..

[23]  Gordon Fraser,et al.  Testing Container Classes: Random or Systematic? , 2011, FASE.

[24]  Gary T. Leavens,et al.  Beyond Assertions: Advanced Specification and Verification with JML and ESC/Java2 , 2005, FMCO.

[25]  Nikolai Tillmann,et al.  Pex-White Box Test Generation for .NET , 2008, TAP.

[26]  Robby,et al.  Kiasan/KUnit: Automatic Test Case Generation and Analysis Feedback for Open Object-oriented Systems , 2007, Testing: Academic and Industrial Conference Practice and Research Techniques - MUTATION (TAICPART-MUTATION 2007).

[27]  Cesare Tinelli,et al.  Satisfiability Modulo Theories , 2021, Handbook of Satisfiability.

[28]  Marcelo F. Frias,et al.  Analysis of invariants for efficient bounded verification , 2010, ISSTA '10.

[29]  Sarfraz Khurshid,et al.  Korat: automated testing based on Java predicates , 2002, ISSTA '02.

[30]  Toby Walsh,et al.  Handbook of satisfiability , 2009 .