An Empirical Study of Structural Constraint Solving Techniques

Structural constraint solving allows finding object graphs that satisfy given constraints, thereby enabling software reliability tasks, such as systematic testing and error recovery. Since enumerating all possible object graphs is prohibitively expensive, researchers have proposed a number of techniques for reducing the number of potential object graphs to consider as candidate solutions. These techniques analyze the structural constraints to prune from search object graphs that cannot satisfy the constraints. Although, analytical and empirical evaluations of individual techniques have been done, comparative studies of different kinds of techniques are rare in the literature. We performed an experiment to evaluate the relative strengths and weaknesses of some key structural constraint solving techniques. The experiment considered four techniques using: a model checker, a SAT solver, a symbolic execution engine, and a specialized solver. It focussed on their relative abilities in expressing the constraints and formatting the output object graphs, and most importantly on their performance. Our results highlight the tradeoffs of different techniques and help choose a technique for practical use.

[1]  Sarfraz Khurshid,et al.  TestEra: Specification-Based Testing of Java Programs Using SAT , 2004, Automated Software Engineering.

[2]  Eugene Goldberg,et al.  BerkMin: A Fast and Robust Sat-Solver , 2002, Discret. Appl. Math..

[3]  Sarfraz Khurshid,et al.  Test input generation with java PathFinder , 2004, ISSTA '04.

[4]  Sarfraz Khurshid,et al.  Efficient solving of structural constraints , 2008, ISSTA '08.

[5]  Sriram K. Rajamani,et al.  Automatically validating temporal safety properties of interfaces , 2001, SPIN '01.

[6]  Sarfraz Khurshid,et al.  Generalized Symbolic Execution for Model Checking and Testing , 2003, TACAS.

[7]  Patrice Godefroid,et al.  Model checking for programming languages using VeriSoft , 1997, POPL '97.

[8]  Sarfraz Khurshid,et al.  Software assurance by bounded exhaustive testing , 2004, IEEE Transactions on Software Engineering.

[9]  James C. Corbett,et al.  Bandera: extracting finite-state models from Java source code , 2000, ICSE.

[10]  Daniel Jackson,et al.  Alcoa: the Alloy constraint analyzer , 2000, Proceedings of the 2000 International Conference on Software Engineering. ICSE 2000 the New Millennium.

[11]  Sarfraz Khurshid,et al.  TestEra: a novel framework for automated testing of Java programs , 2001, Proceedings 16th Annual International Conference on Automated Software Engineering (ASE 2001).

[12]  Andreas Podelski,et al.  ACSAR: Software Model Checking with Transfinite Refinement , 2007, SPIN.

[13]  Sarfraz Khurshid,et al.  Checking Java implementation of a naming architecture using Testera , 2001, Workshop on Software Model Checking @ CAV.

[14]  Oscar Nierstrasz,et al.  Software Engineering - ESEC/FSE '99 , 1999 .

[15]  Margus Veanes,et al.  Generating finite state machines from abstract state machines , 2002, ISSTA '02.

[16]  A. Jefferson Offutt,et al.  Generating Tests from UML Specifications , 1999, UML.

[17]  Sarfraz Khurshid,et al.  PKorat: Parallel Generation of Structurally Complex Test Inputs , 2009, 2009 International Conference on Software Testing Verification and Validation.

[18]  Thomas A. Henzinger,et al.  Software Verification with BLAST , 2003, SPIN.

[19]  J. C. Huang,et al.  An Approach to Program Testing , 1975, CSUR.

[20]  Chandrasekhar Boyapati,et al.  Efficient software model checking of data structure properties , 2006, OOPSLA '06.

[21]  Sarfraz Khurshid,et al.  Parallel test generation and execution with Korat , 2007, ESEC-FSE '07.

[22]  William R. Bush,et al.  A static analyzer for finding dynamic programming errors , 2000 .

[23]  Daniel Jackson,et al.  Software Abstractions - Logic, Language, and Analysis , 2006 .

[24]  Sarfraz Khurshid,et al.  Starc: static analysis for efficient repair of complex data , 2007, OOPSLA.

[25]  Rudolf Bayer,et al.  Symmetric binary B-Trees: Data structure and maintenance algorithms , 1972, Acta Informatica.

[26]  Debra J. Richardson,et al.  Structural specification-based testing: automated support and experimental evaluation , 1999, ESEC/FSE-7.

[27]  J. Michael Spivey,et al.  The Z notation - a reference manual , 1992, Prentice Hall International Series in Computer Science.

[28]  Sarfraz Khurshid,et al.  Korat: automated testing based on Java predicates , 2002, ISSTA '02.

[29]  David Notkin,et al.  Rostra: a framework for detecting redundant object-oriented unit tests , 2004 .

[30]  Koushik Sen,et al.  CUTE: a concolic unit testing engine for C , 2005, ESEC/FSE-13.

[31]  Mark Lillibridge,et al.  Extended static checking for Java , 2002, PLDI '02.

[32]  Bogdan Korel,et al.  Automated test data generation for programs with procedures , 1996, ISSTA '96.

[33]  Michael R. Lowry,et al.  Experiments with Test Case Generation and Runtime Analysis , 2003, Abstract State Machines.

[34]  John B. Goodenough,et al.  Toward a theory of test data selection , 1975 .

[35]  Sarfraz Khurshid,et al.  Optimizing Generation of Object Graphs in Java PathFinder , 2009, 2009 International Conference on Software Testing Verification and Validation.

[36]  Dawson R. Engler,et al.  EXE: automatically generating inputs of death , 2006, CCS '06.

[37]  Carlo Ghezzi,et al.  Using symbolic execution for verifying safety-critical systems , 2001, ESEC/FSE-9.

[38]  Ilya Shlyakhter Generating effective symmetry-breaking predicates for search problems , 2007, Discret. Appl. Math..

[39]  C. V. Ramamoorthy,et al.  On the Automated Generation of Program Test Data , 1976, IEEE Transactions on Software Engineering.

[40]  Koushik Sen,et al.  DART: directed automated random testing , 2005, PLDI '05.

[41]  Thomas A. Henzinger,et al.  Generating tests from counterexamples , 2004, Proceedings. 26th International Conference on Software Engineering.

[42]  Sharad Malik,et al.  Chaff: engineering an efficient SAT solver , 2001, Proceedings of the 38th Design Automation Conference (IEEE Cat. No.01CH37232).

[43]  Rajeev Alur,et al.  A Temporal Logic of Nested Calls and Returns , 2004, TACAS.

[44]  Leonidas J. Guibas,et al.  A dichromatic framework for balanced trees , 1978, 19th Annual Symposium on Foundations of Computer Science (sfcs 1978).

[45]  Donald Sanella What Does the Future Hold for Theoretical Computer Science , 1997 .

[46]  Edmund M. Clarke,et al.  The Birth of Model Checking , 2008, 25 Years of Model Checking.

[47]  James C. King,et al.  Symbolic execution and program testing , 1976, CACM.

[48]  Todd M. Austin,et al.  High Coverage Detection of Input-Related Security Faults , 2003, USENIX Security Symposium.

[49]  Klaus Havelund,et al.  Model Checking Programs , 2004, Automated Software Engineering.

[50]  Lori A. Clarke,et al.  A System to Generate Test Data and Symbolically Execute Programs , 1976, IEEE Transactions on Software Engineering.

[51]  John Penix,et al.  Verification of time partitioning in the DEOS scheduler kernel , 2000, Proceedings of the 2000 International Conference on Software Engineering. ICSE 2000 the New Millennium.

[52]  Angelo Gargantini,et al.  Abstract State Machines 2003 , 2003, Lecture Notes in Computer Science.

[53]  Daniel Jackson,et al.  Alloy: a lightweight object modelling notation , 2002, TSEM.

[54]  Niklas Sörensson,et al.  An Extensible SAT-solver , 2003, SAT.

[55]  Michael R. Donat Automating Formal Specification-Based Testing , 1997, TAPSOFT.

[56]  Arnaud Gotlieb,et al.  Automatic test data generation using constraint solving techniques , 1998, ISSTA '98.

[57]  Michael R. Lowry,et al.  Experimental Evaluation of Verification and Validation Tools on Martian Rover Software , 2013, Formal Methods Syst. Des..