Verification of Simulation Models of Network Protocols Using State Space Exploration and Protocol-Specific Properties

Verification and Validation (VV i.e., a safety property. In this paper, we elaborate on the state space exploration framework in J-Sim and demonstrate its usefulness and effectiveness in verifying complicated simulation models. Specifically, we verify the simulation models of two widely used and fairly complex network protocols: the Ad-Hoc On-Demand Distance Vector (AODV) routing protocol for wireless ad hoc networks and the directed diffusion data dissemination protocol for wireless sensor networks. To enable the verification of these fairly complex network simulation models, we make use of structural properties in the underlying state space along two orthogonal dimensions; the first uses a non-trivial simulation relation to prune the states to be searched, and the second is state ranking that determines whether a state is “better than” another in order to enable the implementation of a best-first search (BeFS). We also develop protocol-specific search heuristics to guide state space exploration towards finding assertion violations in less time. In particular, we report findings on how to devise good search heuristics for routing/data dissemination protocols similar to AODV and directed diffusion. We also show that the time needed to find an assertion violation by our state space exploration framework in J-Sim is comparable to that of Java PathFinder (JPF), a state-of-the-art model checker for Java programs.1

[1]  Gregg Rothermel,et al.  Empirical Studies of a Prediction Model for Regression Test Selection , 2001, IEEE Trans. Software Eng..

[2]  Sheng Liang,et al.  Java Native Interface: Programmer's Guide and Specification , 1999 .

[3]  Peter Csaba Ölveczky,et al.  Specification and Analysis of Real-Time Systems Using Real-Time Maude , 2004, FASE.

[4]  Mahesh Viswanathan,et al.  Bounded Model Checking of Network Protocols in Network Simulators by Exploiting Protocol-Specific Heuristics , 2005 .

[5]  David L. Dill,et al.  Java model checking , 2000, Proceedings ASE 2000. Fifteenth IEEE International Conference on Automated Software Engineering.

[6]  George S. Avrunin,et al.  Heuristic-guided counterexample search in FLAVERS , 2004, SIGSOFT '04/FSE-12.

[7]  Rance Cleaveland,et al.  TwoTowers: A Tool Integrating Functional and Performance Analysis of Concurrent Systems , 1998, FORTE.

[8]  Gregg Rothermel,et al.  A safe, efficient regression test selection technique , 1997, TSEM.

[9]  Osman Balci,et al.  Quality assessment, verification, and validation of modeling and simulation applications , 2004, Proceedings of the 2004 Winter Simulation Conference, 2004..

[10]  Sarfraz Khurshid,et al.  Exploring very large state spaces using genetic algorithms , 2004, International Journal on Software Tools for Technology Transfer.

[11]  Don S. Batory,et al.  The design and implementation of hierarchical software systems with reusable components , 1992, TSEM.

[12]  Jack P. C. Kleijnen,et al.  EUROPEAN JOURNAL OF OPERATIONAL , 1992 .

[13]  Sarfraz Khurshid,et al.  Test input generation with java PathFinder , 2004, ISSTA '04.

[14]  James D. Arthur,et al.  Expanding our horizons in verification, validation, and accreditation research and practice , 2002, Proceedings of the Winter Simulation Conference.

[15]  B. P. Ziegler,et al.  Theory of Modeling and Simulation , 1976 .

[16]  Peter Csaba Ölveczky,et al.  Formal modeling and analysis of wireless sensor network algorithms in Real-Time Maude , 2006, Proceedings 20th IEEE International Parallel & Distributed Processing Symposium.

[17]  Timo Latvala,et al.  Incremental and Complete Bounded Model Checking for Full PLTL , 2005, CAV.

[18]  Carl A. Gunter,et al.  Formal verification of standards for distance vector routing protocols , 2002, JACM.

[19]  José Meseguer,et al.  Formal Analysis of Java Programs in JavaFAN , 2004, CAV.

[20]  Sérgio Vale Aguiar Campos,et al.  Symbolic Model Checking , 1993, CAV.

[21]  Sneha Kumar Kasera,et al.  Scalable fair reliable multicast using active services , 2000, IEEE Netw..

[22]  Alessandro Orso,et al.  Regression test selection for Java software , 2001, OOPSLA '01.

[23]  Marco Bernardo,et al.  An Algebra-Based Method to Associate Rewards with EMPA Terms , 1997, ICALP.

[24]  Ahmed Sobeih,et al.  Incremental State Space Exploration in J-Sim , 2007 .

[25]  Armin Biere,et al.  Symbolic Model Checking without BDDs , 1999, TACAS.

[26]  Marco Conti,et al.  Dynamic tuning of the IEEE 802.11 protocol to achieve a theoretical throughput limit , 2000, TNET.

[27]  Mahesh Viswanathan,et al.  Finding Bugs in Network Protocols Using Simulation Code and Protocol-Specific Heuristics , 2005, ICFEM.

[28]  Mahesh Viswanathan,et al.  Incremental state-space exploration for programs with dynamically allocated data , 2008, 2008 ACM/IEEE 30th International Conference on Software Engineering.

[29]  Klaus Havelund,et al.  Model Checking Programs , 2004, Automated Software Engineering.

[30]  Osman Balci.  Verification, validation, and certification of modeling and simulation applications , 2003, Proceedings of the 2003 Winter Simulation Conference, 2003..

[31]  Gregg Rothermel,et al.  Whole program path-based dynamic impact analysis , 2003, 25th International Conference on Software Engineering, 2003. Proceedings..

[32]  Mark Saaltink,et al.  The Z/EVES System , 1997, ZUM.

[33]  Hyuk Lim,et al.  J-Sim: a simulation and emulation environment for wireless sensor networks , 2006, IEEE Wireless Communications.

[34]  Averill M. Law,et al.  How to build valid and credible simulation models , 2008, 2008 Winter Simulation Conference.

[35]  Mahesh Viswanathan,et al.  J-Sim: An Integrated Environment for Simulation and Model Checking of Network Protocols , 2007, 2007 IEEE International Parallel and Distributed Processing Symposium.

[36]  Robert G. Sargent,et al.  Verification and validation of simulation models , 1998, 1998 Winter Simulation Conference. Proceedings (Cat. No.98CH36274).

[37]  Thomas A. Henzinger,et al.  Extreme Model Checking , 2003, Verification: Theory and Practice.

[38]  H. Rueb,et al.  From simulation to verification (and back) , 2003, Proceedings of the 2003 Winter Simulation Conference, 2003..

[39]  Frank Tip,et al.  Chianti: a tool for change impact analysis of java programs , 2004, OOPSLA.

[40]  Alex Groce,et al.  Modular verification of software components in C , 2003, 25th International Conference on Software Engineering, 2003. Proceedings..

[41]  R. Srikant,et al.  How good are deterministic fluid models of Internet congestion control? , 2002, Proceedings.Twenty-First Annual Joint Conference of the IEEE Computer and Communications Societies.

[42]  J. Banks,et al.  Discrete-Event System Simulation , 1995 .

[43]  Dawson R. Engler,et al.  Model Checking Large Network Protocol Implementations , 2004, NSDI.

[44]  Stephen A. Edwards,et al.  Incremental Algorithms for Inter-procedural Analysis of Safety Properties , 2005, CAV.

[45]  D. J. Morrice,et al.  FROM SIMULATION TO VERIFICATION ( AND BACK ) , 2003 .

[46]  Scott A. Smolka,et al.  Incremental Model Checking in the Modal Mu-Calculus , 1994, CAV.

[47]  Marco Roccetti,et al.  Comparing the QoS of Internet audio mechanisms via formal methods , 2001, TOMC.

[48]  Mamadou Kaba Traoré,et al.  Analyzing Static and Temporal Properties of Simulation Models , 2006, Proceedings of the 2006 Winter Simulation Conference.

[49]  Alan J. Hu,et al.  Protocol verification as a hardware design aid , 1992, Proceedings 1992 IEEE International Conference on Computer Design: VLSI in Computers & Processors.

[50]  Stefan Edelkamp,et al.  Directed explicit-state model checking in the validation of communication protocols , 2004, International Journal on Software Tools for Technology Transfer.

[51]  Gregg Rothermel,et al.  Prioritizing test cases for regression testing , 2000, ISSTA '00.

[52]  Yechiam Yemini,et al.  NEST: a network simulation and prototyping testbed , 1990, CACM.

[53]  Alex Groce,et al.  Heuristics for model checking Java programs , 2004, International Journal on Software Tools for Technology Transfer.

[54]  Gregg Rothermel,et al.  An empirical study of regression test selection techniques , 2001, ACM Trans. Softw. Eng. Methodol..

[55]  José Meseguer,et al.  The Maude LTL Model Checker , 2004, WRLA.

[56]  Donald F. Towsley,et al.  Modeling TCP throughput: a simple model and its empirical validation , 1998, SIGCOMM '98.

[57]  Patrice Godefroid,et al.  Model checking for programming languages using VeriSoft , 1997, POPL '97.

[58]  Etienne E. Kerre,et al.  A fuzzy set theoretic approach to validate simulation models , 2006, TOMC.

[59]  Mahesh Viswanathan,et al.  Verisim: Formal analysis of network simulations , 2000, ISSTA '00.

[60]  J. Michael Spivey,et al.  The Z notation - a reference manual , 1992, Prentice Hall International Series in Computer Science.

[61]  R. Sargent,et al.  Validation of Simulation Models via Simultaneous Confidence Intervals , 1984 .

[62]  Mahesh Viswanathan,et al.  Incorporating Bounded Model Checking in Network Simulation: Theory, Implementation and Evaluation , 2004 .

[63]  Hyuk Lim,et al.  J-Sim: a simulation environment for wireless sensor networks , 2005, 38th Annual Simulation Symposium.

[64]  Gregg Rothermel,et al.  Incorporating varying test costs and fault severities into test case prioritization , 2001, Proceedings of the 23rd International Conference on Software Engineering. ICSE 2001.

[65]  Stephan Merz,et al.  Model Checking , 2000 .

[66]  Charles E. Perkins,et al.  Ad hoc On-Demand Distance Vector (AODV) Routing , 2001, RFC.

[67]  Patrice Godefroid,et al.  Dynamic partial-order reduction for model checking software , 2005, POPL '05.

[68]  Peter Csaba Ölveczky,et al.  Specification and analysis of the AER/NCA active network protocol suite in Real-Time Maude , 2006, Formal Methods Syst. Des..

[69]  Karl Pauwels,et al.  A Neural Network Approach to the Validation of Simulation Models , 2006, Proceedings of the 2006 Winter Simulation Conference.

[70]  Jennifer C. Hou,et al.  Towards composable and extensible network simulation , 2005, 19th IEEE International Parallel and Distributed Processing Symposium.

[71]  Rance Cleaveland,et al.  The NCSU Concurrency Workbench , 1996, CAV.

[72]  Matthew B. Dwyer,et al.  Controlling factors in evaluating path-sensitive error detection techniques , 2006, SIGSOFT '06/FSE-14.

[73]  Hanêne Ben-Abdallah,et al.  Formally specified monitoring of temporal properties , 1999, Proceedings of 11th Euromicro Conference on Real-Time Systems. Euromicro RTS'99.

[74]  Osman Balci,et al.  Principles of simulation model validation, verification, and testing , 1997 .

[75]  Gregg Rothermel,et al.  Test Case Prioritization: A Family of Empirical Studies , 2002, IEEE Trans. Software Eng..

[76]  William J. Stewart,et al.  Introduction to the numerical solution of Markov Chains , 1994 .

[77]  Roberto Gorrieri,et al.  A Tutorial on EMPA: A Theory of Concurrent Processes with Nondeterminism, Priorities, Probabilities and Time , 1998, Theor. Comput. Sci..

[78]  Matthew B. Dwyer,et al.  Parallel Randomized State-Space Search , 2007, 29th International Conference on Software Engineering (ICSE'07).

[79]  Alessandro Orso,et al.  Leveraging field data for impact analysis and regression testing , 2003, ESEC/FSE-11.

[80]  Arvinder Kaur,et al.  Component Based Software Engineering , 2010 .

[81]  Deborah Estrin,et al.  Directed diffusion: a scalable and robust communication paradigm for sensor networks , 2000, MobiCom '00.

[82]  Gregor von Bochmann,et al.  FSM-based incremental conformance testing methods , 2004, IEEE Transactions on Software Engineering.

[83]  Donald Ervin Knuth,et al.  The Art of Computer Programming, Volume II: Seminumerical Algorithms , 1970 .

[84]  Laurie A. Williams,et al.  Applying regression test selection for COTS-based applications , 2006, ICSE.

[85]  Radu Iosif.  Exploiting heap symmetries in explicit-state model checking of software , 2001, Proceedings 16th Annual International Conference on Automated Software Engineering (ASE 2001).

[86]  Klaus Havelund,et al.  Java PathFinder, A Translator from Java to Promela , 1999, SPIN.

[87]  James C. Corbett,et al.  Bandera: extracting finite-state models from Java source code , 2000, ICSE.

[88]  Corina S. Pasareanu,et al.  Test input generation for red-black trees using abstraction , 2005, ASE.

[89]  Srinivasan Keshav,et al.  REAL: A Network Simulator , 1988 .

[90]  Jennifer C. Hou,et al.  A Simulation Framework for Sensor Networks in J-Sim , 2003 .

[91]  Gregg Rothermel,et al.  Using component metacontent to support the regression testing of component-based software , 2001, Proceedings IEEE International Conference on Software Maintenance. ICSM 2001.

[92]  Hung-Ying Tyan,et al.  Design, realization and evaluation of a component-based compositional software architecture for network simulation , 2002 .

[93]  Gerard J. Holzmann,et al.  The Model Checker SPIN , 1997, IEEE Trans. Software Eng..

[94]  Jennifer C. Hou,et al.  Maintaining Sensing Coverage and Connectivity in Large Sensor Networks , 2005, Ad Hoc Sens. Wirel. Networks.

[95]  Carl A. Gunter,et al.  Fault origin adjudication , 2000, FMSP '00.

[96]  A. Orso,et al.  Efficient and precise dynamic impact analysis using execute-after sequences , 2005, Proceedings. 27th International Conference on Software Engineering, 2005. ICSE 2005..

[97]  Dawson R. Engler,et al.  Proceedings of the 5th Symposium on Operating Systems Design and Implementation Cmc: a Pragmatic Approach to Model Checking Real Code , 2022 .

[98]  David L. Dill,et al.  An Incremental Heap Canonicalization Algorithm , 2005, SPIN.

[99]  Marcelo d'Amorim,et al.  Optimized Execution of Deterministic Blocks in Java PathFinder , 2006, ICFEM.

[100]  Michael D. Ernst,et al.  Automatic test factoring for java , 2005, ASE '05.

[101]  Vishal Misray,et al.  Stochastic Differential Equation Modeling and Analysis of TCP-Windowsize Behavior , 2005 .

[102]  Charles E. Perkins,et al.  Ad-hoc on-demand distance vector routing , 1999, Proceedings WMCSA'99. Second IEEE Workshop on Mobile Computing Systems and Applications.

[103]  Osman Balci,et al.  Verification, Validation, and Testing , 2007 .

[104]  Mary Jean Harrold,et al.  Test-Suite Reduction and Prioritization for Modified Condition/Decision Coverage , 2003, IEEE Trans. Software Eng..

[105]  David L. Dill,et al.  Validation with guided search of the state space , 1998, Proceedings 1998 Design and Automation Conference. 35th DAC. (Cat. No.98CH36175).

[106]  Corina S. Pasareanu,et al.  Test input generation for java containers using state matching , 2006, ISSTA '06.

[107]  Alessandro Orso,et al.  Selective capture and replay of program executions , 2005, WODA '05.

[108]  Mahesh Viswanathan,et al.  Check and simulate: a case for incorporating model checking in network simulation , 2004, Proceedings. Second ACM and IEEE International Conference on Formal Methods and Models for Co-Design, 2004. MEMOCODE '04..