Assertion Checking in J-Sim Simulation Models of Network Protocols

Verification and validation (V&V) is a critically important phase in the development life cycle of a simulation model. In the context of network simulation, traditional network simulators perform well in using a simulation model for evaluating the performance of a network protocol but lack the capability to check the “correctness” of the simulation model being used. To address this problem, we have extended J-Sim—an open-source component-based network simulator written entirely in Java—with a state space exploration (SSE) capability that explores the state space created by a network simulation model, up to a configurable maximum depth, in order to find an execution (if any) that violates an assertion, i.e. a property specifying an invariant that must always hold true in all states. In this paper, we elaborate on the SSE framework in J-Sim and present one of our fairly complex case studies, namely verifying the simulation model of the Ad-hoc On-demand Distance Vector (AODV) routing protocol for wireless ad-hoc networks. The SSE framework makes use of protocol-specific properties along two orthogonal dimensions: state similarity and state ranking. State similarity determines whether a state is “similar to” another in order to enable the implementation of stateful search. State ranking determines whether a state is “better than” another in order to enable the implementation of best-first search (BeFS). Specifically, we develop protocol-specific search heuristics to guide SSE towards finding assertion violations in less time. We evaluate the efficiency of our SSE framework by comparing its performance with that of a state-of-the-art model checker for Java programs, namely Java PathFinder (JPF). The results of the comparison show that the time needed to find an assertion violation by our SSE framework in J-Sim can be significantly less than that in JPF unless a substantial amount of programming effort is spent in JPF to make its performance close to that of our SSE framework.

[1]  James D. Arthur,et al.  Expanding our horizons in verification, validation, and accreditation research and practice , 2002, Proceedings of the Winter Simulation Conference.

[2]  Mahesh Viswanathan,et al.  Verisim: Formal analysis of network simulations , 2000, ISSTA '00.

[3]  Carl A. Gunter,et al.  Formal verification of standards for distance vector routing protocols , 2002, JACM.

[4]  Marco Bernardo,et al.  An Algebra-Based Method to Associate Rewards with EMPA Terms , 1997, ICALP.

[5]  J. Michael Spivey,et al.  The Z notation - a reference manual , 1992, Prentice Hall International Series in Computer Science.

[6]  R. Sargent,et al.  Validation of Simulation Models via Simultaneous Confidence Intervals , 1984 .

[7]  Klaus Havelund,et al.  Model Checking Programs , 2004, Automated Software Engineering.

[8]  Osman Balci.  Verification, validation, and certification of modeling and simulation applications , 2003, Proceedings of the 2003 Winter Simulation Conference, 2003..

[9]  Kenneth L. McMillan,et al.  Symbolic model checking , 1992 .

[10]  Charles E. Perkins,et al.  Ad hoc On-Demand Distance Vector (AODV) Routing , 2001, RFC.

[11]  Patrice Godefroid,et al.  Dynamic partial-order reduction for model checking software , 2005, POPL '05.

[12]  Karl Pauwels,et al.  A Neural Network Approach to the Validation of Simulation Models , 2006, Proceedings of the 2006 Winter Simulation Conference.

[13]  Matthew B. Dwyer,et al.  Controlling factors in evaluating path-sensitive error detection techniques , 2006, SIGSOFT '06/FSE-14.

[14]  Edmund M. Clarke,et al.  Model Checking , 1999, Handbook of Automated Reasoning.

[15]  José Meseguer,et al.  Formal Analysis of Java Programs in JavaFAN , 2004, CAV.

[16]  Mark Saaltink,et al.  The Z/EVES System , 1997, ZUM.

[17]  Osman Balci,et al.  Principles of simulation model validation, verification, and testing , 1997 .

[18]  Mahesh Viswanathan,et al.  J-Sim: An Integrated Environment for Simulation and Model Checking of Network Protocols , 2007, 2007 IEEE International Parallel and Distributed Processing Symposium.

[19]  Rance Cleaveland,et al.  The NCSU Concurrency Workbench , 1996, CAV.

[20]  Hanêne Ben-Abdallah,et al.  Formally specified monitoring of temporal properties , 1999, Proceedings of 11th Euromicro Conference on Real-Time Systems. Euromicro RTS'99.

[21]  Mamadou Kaba Traoré,et al.  Analyzing Static and Temporal Properties of Simulation Models , 2006, Proceedings of the 2006 Winter Simulation Conference.

[22]  Alan J. Hu,et al.  Protocol verification as a hardware design aid , 1992, Proceedings 1992 IEEE International Conference on Computer Design: VLSI in Computers & Processors.

[23]  Robin Milner,et al.  Communication and concurrency , 1989, PHI Series in computer science.

[24]  Stefan Edelkamp,et al.  Directed explicit-state model checking in the validation of communication protocols , 2004, International Journal on Software Tools for Technology Transfer.

[25]  H. Rueb,et al.  From simulation to verification (and back) , 2003, Proceedings of the 2003 Winter Simulation Conference, 2003..

[26]  Peter Csaba Ölveczky,et al.  Specification and Analysis of Real-Time Systems Using Real-Time Maude , 2004, FASE.

[27]  Radu Iosif.  Exploiting heap symmetries in explicit-state model checking of software , 2001, Proceedings 16th Annual International Conference on Automated Software Engineering (ASE 2001).

[28]  Klaus Havelund,et al.  Java PathFinder, A Translator from Java to Promela , 1999, SPIN.

[29]  Dawson R. Engler,et al.  Model Checking Large Network Protocol Implementations , 2004, NSDI.

[30]  James D. Arthur,et al.  Improving the model development process: expanding our horizons in verification, validation, and accreditation research and practice , 2002, WSC '02.

[31]  David L. Dill,et al.  Validation with guided search of the state space , 1998, Proceedings 1998 Design and Automation Conference. 35th DAC. (Cat. No.98CH36175).

[32]  Corina S. Pasareanu,et al.  Test input generation for red-black trees using abstraction , 2005, ASE.

[33]  José Meseguer,et al.  The Maude LTL Model Checker , 2004, WRLA.

[34]  Etienne E. Kerre,et al.  A fuzzy set theoretic approach to validate simulation models , 2006, TOMC.

[35]  Hung-Ying Tyan,et al.  Design, realization and evaluation of a component-based compositional software architecture for network simulation , 2002 .

[36]  Gerard J. Holzmann,et al.  The Model Checker SPIN , 1997, IEEE Trans. Software Eng..

[37]  Jennifer C. Hou,et al.  Maintaining Sensing Coverage and Connectivity in Large Sensor Networks , 2005, Ad Hoc Sens. Wirel. Networks.

[38]  Charles E. Perkins,et al.  Ad-hoc on-demand distance vector routing , 1999, Proceedings WMCSA'99. Second IEEE Workshop on Mobile Computing Systems and Applications.

[39]  Osman Balci,et al.  Verification, Validation, and Testing , 2007 .

[40]  Osman Balci,et al.  Quality assessment, verification, and validation of modeling and simulation applications , 2004, Proceedings of the 2004 Winter Simulation Conference, 2004..

[41]  Corina S. Pasareanu,et al.  Test input generation for java containers using state matching , 2006, ISSTA '06.

[42]  Mahesh Viswanathan,et al.  Check and simulate: a case for incorporating model checking in network simulation , 2004, Proceedings. Second ACM and IEEE International Conference on Formal Methods and Models for Co-Design, 2004. MEMOCODE '04..

[43]  Donald Ervin Knuth,et al.  The Art of Computer Programming , 1968 .

[44]  Sneha Kumar Kasera,et al.  Scalable fair reliable multicast using active services , 2000, IEEE Netw..

[45]  Averill M. Law,et al.  How to build valid and credible simulation models , 2008, 2008 Winter Simulation Conference.

[46]  Dawson R. Engler,et al.  Proceedings of the 5th Symposium on Operating Systems Design and Implementation Cmc: a Pragmatic Approach to Model Checking Real Code , 2022 .

[47]  Sarfraz Khurshid,et al.  Test input generation with java PathFinder , 2004, ISSTA '04.

[48]  Marcelo d'Amorim,et al.  Verification of Simulation Models of Network Protocols Using State Space Exploration and Protocol-Specific Properties , 2007 .

[49]  Mahesh Viswanathan,et al.  Finding Bugs in Network Protocols Using Simulation Code and Protocol-Specific Heuristics , 2005, ICFEM.

[50]  Rance Cleaveland,et al.  TwoTowers: A Tool Integrating Functional and Performance Analysis of Concurrent Systems , 1998, FORTE.

[51]  Sarfraz Khurshid,et al.  Exploring very large state spaces using genetic algorithms , 2004, International Journal on Software Tools for Technology Transfer.

[52]  William J. Stewart,et al.  Introduction to the numerical solution of Markov Chains , 1994 .

[53]  Roberto Gorrieri,et al.  A Tutorial on EMPA: A Theory of Concurrent Processes with Nondeterminism, Priorities, Probabilities and Time , 1998, Theor. Comput. Sci..

[54]  Matthew B. Dwyer,et al.  Parallel Randomized State-Space Search , 2007, 29th International Conference on Software Engineering (ICSE'07).

[55]  J. Banks,et al.  Discrete-Event System Simulation , 1995 .

[56]  Marco Roccetti,et al.  Comparing the QoS of Internet audio mechanisms via formal methods , 2001, TOMC.

[57]  Alex Groce,et al.  Heuristics for model checking Java programs , 2004, International Journal on Software Tools for Technology Transfer.

[58]  Patrice Godefroid,et al.  Model checking for programming languages using VeriSoft , 1997, POPL '97.

[59]  Peter Csaba Ölveczky,et al.  Specification and analysis of the AER/NCA active network protocol suite in Real-Time Maude , 2006, Formal Methods Syst. Des..

[60]  Matthew B. Dwyer,et al.  Bandera: extracting finite-state models from Java source code , 2000, Proceedings of the 2000 International Conference on Software Engineering. ICSE 2000 the New Millennium.

[61]  Francisco Durán,et al.  Playing with Maude , 2007, All About Maude.

[62]  David L. Dill,et al.  Java model checking , 2000, Proceedings ASE 2000. Fifteenth IEEE International Conference on Automated Software Engineering.

[63]  George S. Avrunin,et al.  Heuristic-guided counterexample search in FLAVERS , 2004, SIGSOFT '04/FSE-12.

[64]  Peter Csaba Ölveczky,et al.  Formal modeling and analysis of wireless sensor network algorithms in Real-Time Maude , 2006, Proceedings 20th IEEE International Parallel & Distributed Processing Symposium.

[65]  Bernard P. Zeigler,et al.  Theory of modeling and simulation , 1976 .