Efficient Explicit -State Model Checking for Programs With Dynamically Allocated Data

Despite the technological advances in languages and tools to support program development, programmers still deliver software with lots of errors. Software testing has been the dominant approach in industry to improve the quality of software at code level. Testing software is not cheap though. A study done in 2002 by the National Institute of Standards and Technology (NIST) [NIS02] reports that between 70% and 80% of development costs is due to testing. We believe that automation of software testing can help to reduce this cost; automation can assist the programmer to test the code under development in a more systematic way. Software model checkers are becoming increasingly popular to assist in the automation of software testing. Model checkers are tools that systematically explore the state space of a model to demonstrate the presence of errors or to confirm their absence. For example, a model checker can report a test whenever it finds an "interesting" state, i.e., a state which violates a user-defined assertion or a state that makes execution throw a runtime exception. This dissertation focuses on software model checkers that operate directly on code. Model checking programs is becoming very popular, as it enables the discovery of errors in the implementation as opposed to errors in a system design. However, model checking programs poses particular challenges. It is the implementation that defines the model, not the design. As a consequence, a deterministic step of the exploration can be potentially more expensive as it may require the execution of a complex fragment of sequential code. In addition, the states that the model defines are potentially larger, making important operations that depend on the size of the state potentially slower. While state-space explosion has been traditionally the key issue in model checking, time efficiency becomes more relevant with the increasing appearance of model checkers that operate on programs. Our goal is to improve time efficiency for model checking programs that allocate dynamic data. Such programs are pervasive nowadays with the wide use of object-oriented languages. To that end, this dissertation presents Mixed execution and ΔExecution. Mixed execution speeds up the execution of deterministic steps in model checkers that use a special representation of state. Such representation can be convenient to perform some model checking operations but sacrifices the execution of deterministic steps (i.e., fragments of sequential code). ΔExecution uses sets of states to perform state-space exploration. The use of sets enables the model checker to take advantage of overlappings (of state and paths) that exist in a systematic exploration of state, speeding up several operations in software model checking. We implemented mixed execution in Java PathFinder (JPF), a model checker for Java programs. We evaluate mixed execution on seven simple subject programs and one large case study. The experimental results on the seven smaller programs show that mixed execution can improve the overall time for state exploration in JPF from 1.01x to 1.73x (with median 1.13x). The experimental results on a larger case study show that the exploration time on this experiment reduces from 1.14x to 1.41x (with median 1.23x). We implemented Delta Execution in two model checkers, JPF (Java PathFinder) and BOX (Bounded Object eXploration) to evaluate the effectiveness of this technique in model checkers with different designs. The results show that ΔExecution improves the overall exploration in both tools, but the improvements are due to different factors. We evaluate ΔExecution on ten simple subject programs and a larger case study. The results show that ΔExecution improves the exploration time for the smaller programs from 0.88x to 126.80x (with median 5.60x) in JPF and from 0.58x to 4.16x (with median 2.23x) in BOX, while takingfrom 0.46x to 11.50x (with median 1.48x) less memory in JPF and from 0.18x to 2.71x (with median 1.18x) memory in BOX. The experimental results on one larger case study show that ΔExecution improves the exploration time from 0.88x to 2.04x (with median 1.72x). Note that a ratio smaller than 1 means a slow down or use of more memory. Such ratios can arise in the exploration of small state-spaces.

[1]  David L. Dill,et al.  An Incremental Heap Canonicalization Algorithm , 2005, SPIN.

[2]  Marcelo d'Amorim,et al.  Optimized Execution of Deterministic Blocks in Java PathFinder , 2006, ICFEM.

[3]  Sheng Liang,et al.  Java Native Interface: Programmer's Guide and Specification , 1999 .

[4]  Thomas W. Reps,et al.  Symbolically Computing Most-Precise Abstract Operations for Shape Analysis , 2004, TACAS.

[5]  Hung-Ying Tyan,et al.  Design, realization and evaluation of a component-based compositional software architecture for network simulation , 2002 .

[6]  Claudio Demartini,et al.  A deadlock detection tool for concurrent Java programs , 1999, Softw. Pract. Exp..

[7]  Gerard J. Holzmann,et al.  The Model Checker SPIN , 1997, IEEE Trans. Software Eng..

[8]  B. Boehm Software risk management: principles and practices , 1991, IEEE Software.

[9]  A. Jefferson Offutt,et al.  The dynamic domain reduction procedure for test data generation , 1999, Softw. Pract. Exp..

[10]  Grzegorz Rozenberg,et al.  Linear Time, Branching Time and Partial Order in Logics and Models for Concurrency , 1988, Lecture Notes in Computer Science.

[11]  Marcelo d'Amorim,et al.  An Empirical Comparison of Automated Generation and Classification Techniques for Object-Oriented Unit Testing , 2006, 21st IEEE/ACM International Conference on Automated Software Engineering (ASE'06).

[12]  Augusto Sampaio,et al.  Model-checking CSP-Z: strategy, tool support and industrial application , 2001, Sci. Comput. Program..

[13]  Sarfraz Khurshid,et al.  TestEra: a novel framework for automated testing of Java programs , 2001, Proceedings 16th Annual International Conference on Automated Software Engineering (ASE 2001).

[14]  Edmund M. Clarke,et al.  Symbolic Model Checking: 10^20 States and Beyond , 1990, Inf. Comput..

[15]  Sheldon B. Akers,et al.  Binary Decision Diagrams , 1978, IEEE Transactions on Computers.

[16]  Somesh Jha,et al.  Exploiting symmetry in temporal logic model checking , 1993, Formal Methods Syst. Des..

[17]  P. David Stotts,et al.  An Informal Formal Method for Systematic JUnit Test Case Generation , 2002, XP/Agile Universe.

[18]  Eugene H. Spafford,et al.  Debugging with dynamic slicing and backtracking , 1993, Softw. Pract. Exp..

[19]  A. Tarski A LATTICE-THEORETICAL FIXPOINT THEOREM AND ITS APPLICATIONS , 1955 .

[20]  William H. Sanders,et al.  Delta execution for software reliability , 2007 .

[21]  Carl A. Gunter,et al.  Formal verification of standards for distance vector routing protocols , 2002, JACM.

[22]  Martin C. Rinard,et al.  ACM Conference on Object-Oriented Programming, Systems, Languages and Applications (OOPSLA), November 2002 Ownership Types for Safe Programming: Preventing Data Races and Deadlocks , 2022 .

[23]  Mark Weiser,et al.  Program Slicing , 1981, IEEE Transactions on Software Engineering.

[24]  Daniel Kroening,et al.  A Tool for Checking ANSI-C Programs , 2004, TACAS.

[25]  Kenneth L. McMillan,et al.  A methodology for hardware verification using compositional model checking , 2000, Sci. Comput. Program..

[26]  Dawson R. Engler,et al.  EXE: automatically generating inputs of death , 2006, CCS '06.

[27]  Sarfraz Khurshid,et al.  Generalized Symbolic Execution for Model Checking and Testing , 2003, TACAS.

[28]  Cormac Flanagan,et al.  A type and effect system for atomicity , 2003, PLDI.

[29]  Mahesh Viswanathan,et al.  Finding Bugs in Network Protocols Using Simulation Code and Protocol-Specific Heuristics , 2005, ICFEM.

[30]  Mary Shaw,et al.  Estimating the numbers of end users and end user programmers , 2005, 2005 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC'05).

[31]  Koushik Sen,et al.  DART: directed automated random testing , 2005, PLDI '05.

[32]  Dawson R. Engler,et al.  Proceedings of the 5th Symposium on Operating Systems Design and Implementation Cmc: a Pragmatic Approach to Model Checking Real Code , 2022 .

[33]  Patrice Godefroid,et al.  Model checking for programming languages using VeriSoft , 1997, POPL '97.

[34]  Jakob Rehof,et al.  Zing: A Model Checker for Concurrent Software , 2004, CAV.

[35]  James C. Corbett,et al.  Bandera: extracting finite-state models from Java source code , 2000, ICSE.

[36]  Edsger W. Dijkstra,et al.  Hierarchical ordering of sequential processes , 1971, Acta Informatica.

[37]  Daniel Jackson,et al.  Alcoa: the Alloy constraint analyzer , 2000, Proceedings of the 2000 International Conference on Software Engineering. ICSE 2000 the New Millennium.

[38]  Randal E. Bryant,et al.  Symbolic Boolean manipulation with ordered binary-decision diagrams , 1992, CSUR.

[39]  Alan J. Hu,et al.  Protocol verification as a hardware design aid , 1992, Proceedings 1992 IEEE International Conference on Computer Design: VLSI in Computers & Processors.

[40]  A classical mind: essays in honour of C. A. R. Hoare , 1994 .

[41]  Michael Burrows,et al.  Eraser: a dynamic data race detector for multithreaded programs , 1997, TOCS.

[42]  Corina S. Pasareanu,et al.  Test input generation for red-black trees using abstraction , 2005, ASE.

[43]  Edmund M. Clarke,et al.  Model Checking , 1999, Handbook of Automated Reasoning.

[44]  Beate Bollig,et al.  Improving the Variable Ordering of OBDDs Is NP-Complete , 1996, IEEE Trans. Computers.

[45]  David L. Dill,et al.  Java model checking , 2000, Proceedings ASE 2000. Fifteenth IEEE International Conference on Automated Software Engineering.

[46]  Marcelo d'Amorim,et al.  An Equational Specification for the Scheme Language , 2005, J. Univers. Comput. Sci..

[47]  Glynn Winskel,et al.  The formal semantics of programming languages - an introduction , 1993, Foundation of computing series.

[48]  Koushik Sen,et al.  CUTE: a concolic unit testing engine for C , 2005, ESEC/FSE-13.

[49]  Matthew B. Dwyer,et al.  Slicing Software for Model Construction , 2000, High. Order Symb. Comput..

[50]  Matthew B. Dwyer,et al.  Carving differential unit test cases from system test cases , 2006, SIGSOFT '06/FSE-14.

[51]  Charles E. Perkins,et al.  Ad-hoc on-demand distance vector routing , 1999, Proceedings WMCSA'99. Second IEEE Workshop on Mobile Computing Systems and Applications.

[52]  Kenneth L. McMillan,et al.  Symbolic model checking , 1992 .

[53]  A. Jefferson Offutt,et al.  The dynamic domain reduction procedure for test data generation , 1999, Softw. Pract. Exp..

[54]  Dawson R. Engler,et al.  Model Checking Large Network Protocol Implementations , 2004, NSDI.

[55]  Willem Visser,et al.  Addressing dynamic issues of program model checking , 2001, SPIN '01.

[56]  Corina S. Pasareanu,et al.  Test input generation for java containers using state matching , 2006, ISSTA '06.

[57]  Patrice Godefroid,et al.  Partial-Order Methods for the Verification of Concurrent Systems , 1996, Lecture Notes in Computer Science.

[58]  Jennifer C. Hou,et al.  Towards composable and extensible network simulation , 2005, 19th IEEE International Parallel and Distributed Processing Symposium.

[59]  Fausto Giunchiglia,et al.  NUSMV: A New Symbolic Model Verifier , 1999, CAV.

[60]  A. Prasad Sistla,et al.  Reasoning about systems with many processes , 1992, JACM.

[61]  Patrick Cousot,et al.  Abstract interpretation: a unified lattice model for static analysis of programs by construction or approximation of fixpoints , 1977, POPL.

[62]  Monica S. Lam,et al.  Cloning-based context-sensitive pointer alias analysis using binary decision diagrams , 2004, PLDI '04.

[63]  Klaus Havelund,et al.  Model Checking Programs , 2004, Automated Software Engineering.

[64]  Radu Iosif,et al.  A deadlock detection tool for concurrent Java programs , 1999, Softw. Pract. Exp..

[65]  Michael D. Ernst,et al.  Eclat: Automatic Generation and Classification of Test Inputs , 2005, ECOOP.

[66]  Radu Rugina Shape Analysis Quantitative Shape Analysis , 2004, SAS.

[67]  David Notkin,et al.  Rostra: a framework for detecting redundant object-oriented unit tests , 2004 .

[68]  James C. Corbett,et al.  A Formal Study of Slicing for Multi-threaded Programs with JVM Concurrency Primitives , 1999, SAS.

[69]  Sarfraz Khurshid,et al.  Test input generation with java PathFinder , 2004, ISSTA '04.

[70]  Scott D. Stoller,et al.  Runtime analysis for atomicity for multi-threaded programs , 2005 .

[71]  David Notkin,et al.  Symstra: A Framework for Generating Object-Oriented Unit Tests Using Symbolic Execution , 2005, TACAS.

[72]  Sriram K. Rajamani,et al.  Bebop: A Symbolic Model Checker for Boolean Programs , 2000, SPIN.

[73]  Antti Valmari A stubborn attack on state explosion , 1992, Formal Methods Syst. Des..

[74]  Cyrille Artho,et al.  Accurate Centralization for Applying Model Checking on Networked Applications , 2006, 21st IEEE/ACM International Conference on Automated Software Engineering (ASE'06).

[75]  Marcelo d'Amorim,et al.  Delta Execution for Efficient State-Space Exploration of Object-Oriented Programs , 2008, IEEE Trans. Software Eng..

[76]  Chandrasekhar Boyapati,et al.  Efficient software model checking of data structure properties , 2006, OOPSLA '06.

[77]  Ondrej Lhoták,et al.  Jedd: a BDD-based relational extension of Java , 2004, PLDI '04.

[78]  Matthew B. Dwyer,et al.  Space-Reduction Strategies for Model Checking Dynamic Software , 2003, SoftMC@CAV.

[79]  Mahesh Viswanathan,et al.  Incorporating Bounded Model Checking in Network Simulation: Theory, Implementation and Evaluation , 2004 .

[80]  Mitchell Wand,et al.  Essentials of programming languages , 2008 .

[81]  Boris Beizer,et al.  Software Testing Techniques , 1983 .

[82]  Viktor Schuppan,et al.  JNuke: Efficient Dynamic Analysis for Java , 2004, CAV.

[83]  Dawson R. Engler,et al.  EXE: automatically generating inputs of death , 2006, CCS '06.

[84]  Yannis Smaragdakis,et al.  JCrasher: an automatic robustness tester for Java , 2004, Softw. Pract. Exp..

[85]  Radu Iosif,et al.  Symmetry Reduction Criteria for Software Model Checking , 2002, SPIN.

[86]  Junfeng Yang,et al.  Using model checking to find serious file system errors , 2004, TOCS.

[87]  Margus Veanes,et al.  Online testing with model programs , 2005, ESEC/FSE-13.

[88]  Peter Sestoft,et al.  Partial evaluation and automatic program generation , 1993, Prentice Hall international series in computer science.

[89]  Viktor Kuncak,et al.  Role analysis , 2002, POPL '02.

[90]  Matthew B. Dwyer,et al.  Bogor: an extensible and highly-modular software model checking framework , 2003, ESEC/FSE-11.

[91]  Gregory Tassey,et al.  Prepared for what , 2007 .

[92]  Radu Iosif Exploiting heap symmetries in explicit-state model checking of software , 2001, Proceedings 16th Annual International Conference on Automated Software Engineering (ASE 2001).

[93]  Klaus Havelund,et al.  Java PathFinder, A Translator from Java to Promela , 1999, SPIN.

[94]  C. A. R. Hoare,et al.  Communicating sequential processes , 1978, CACM.

[95]  John Penix,et al.  Verification of time partitioning in the DEOS scheduler kernel , 2000, Proceedings of the 2000 International Conference on Software Engineering. ICSE 2000 the New Millennium.

[96]  Somesh Jha,et al.  Verification of the Futurebus+ cache coherence protocol , 1993, Formal Methods Syst. Des..

[97]  Saburo Muroga,et al.  Binary Decision Diagrams , 2000, The VLSI Handbook.

[98]  Carl A. Gunter Semantics of programming languages: structures and techniques , 1993, Choice Reviews Online.

[99]  John Penix,et al.  Using predicate abstraction to reduce object-oriented programs for model checking , 2000, FMSP '00.

[100]  Sérgio Vale Aguiar Campos,et al.  Symbolic Model Checking , 1993, CAV.

[101]  Sarfraz Khurshid,et al.  Korat: automated testing based on Java predicates , 2002, ISSTA '02.

[102]  Peter Huber,et al.  Towards reachability trees for high-level Petri nets , 1985, European Workshop on Applications and Theory in Petri Nets.