Probabilistic model checking : a comparison of tools

Model checking is a technique to establish the correctness of hardware or software systems in an automated fashion. The goal of this technique is to try to predict system behaviour, or more specifically, to formally prove that all possible executions of the system conform to the requirements. Probabilistic model checking focusses on proving correctness of stochastic systems (i. e. systems where probabilities play a role). A probabilistic model checker tool automates the correctness proving process. These tools can verify if a system – which is described by a model, written in a formal language – satisfies a formal specification, which is expressed using logics, such as Probabilistic Computation Tree Logic (PCTL). We have studied the efficiency of five probabilistic model checker tools, namely: PRISM (Sparse and Hybrid mode), MRMC, ETMCC, YMER and VESTA. We made a tool by tool comparison, analysing model check times and peak memory usage. This was achieved by using five representative case studies of fully probabilistic systems, namely; Synchronous Leader Election (SLE), Randomized Dining Philosophers (RDP), Birth-death process (BDP), Tandem Queuing Network (TQN) and Cyclic Server Polling System (CSP). Besides their performance, we also investigated the characteristics of each tool, comparing their implementation details, range of supported probabilistic models, model specification language, property specification language and supported algorithms and data structures. During our research, we have performed nearly 15,000 individual runs. By ensuring that our experiments are automated, repeatable, verifiable, statistically significant and free from external influences, our findings are based on a solid methodology.

[1]  Laurence G. Grimm,et al.  Statistical Applications for the Behavioral Sciences , 1993 .

[2]  Peter W. Glynn,et al.  Computing Poisson probabilities , 1988, CACM.

[3]  Christel Baier,et al.  Model Checking Continuous-Time Markov Chains by Transient Analysis , 2000, CAV.

[4]  John I. McCool,et al.  Probability and Statistics With Reliability, Queuing and Computer Science Applications , 2003, Technometrics.

[5]  Holger Hermanns,et al.  Compositional performance modelling with the TIPPtool , 2000, Perform. Evaluation.

[6]  Sri Gopal Mohanty,et al.  On the transient behavior of a finite birth-death process with an application , 1993, Comput. Oper. Res..

[7]  Casey A. Volino,et al.  A First Course in Stochastic Models , 2005, Technometrics.

[8]  Robert E. Tarjan,et al.  Depth-First Search and Linear Graph Algorithms , 1972, SIAM J. Comput..

[9]  DAVID G. KENDALL,et al.  Introduction to Mathematical Statistics , 1947, Nature.

[10]  Richard Lassaigne,et al.  Approximate Verification of Probabilistic Systems , 2002, PAPM-PROBMIV.

[11]  Håkan L. S. Younes.  Ymer: A Statistical Model Checker , 2005, CAV.

[12]  T. Y. WilliamJ,et al.  Numerical Methods in Markov Chain Modeling , 1992, Operational Research.

[13]  D. Cox.  A use of complex probabilities in the theory of stochastic processes , 1955, Mathematical Proceedings of the Cambridge Philosophical Society.

[14]  Håkan L. S. Younes,et al.  Numerical vs. statistical probabilistic model checking , 2006, International Journal on Software Tools for Technology Transfer.

[15]  D. Sorensen.  Numerical methods for large eigenvalue problems , 2002, Acta Numerica.

[16]  William J. Stewart,et al.  Introduction to the numerical solution of Markov Chains , 1994 .

[17]  S. Karlin,et al.  The differential equations of birth-and-death processes, and the Stieltjes moment problem , 1957 .

[18]  Håkan L. S. Younes,et al.  Probabilistic Verification of Discrete Event Systems Using Acceptance Sampling , 2002, CAV.

[19]  Joost-Pieter Katoen,et al.  Towards Model Checking Stochastic Process Algebra , 2000, IFM.

[20]  M. Siegle,et al.  Multi Terminal Binary Decision Diagrams to Represent and Analyse Continuous Time Markov Chains , 1999 .

[21]  Andrew Hinton,et al.  PRISM: A Tool for Automatic Verification of Probabilistic Systems , 2006, TACAS.

[22]  Jane Hillston,et al.  A compositional approach to performance modelling , 1996 .

[23]  F. Wilcoxon.  Individual Comparisons by Ranking Methods , 1945 .

[24]  Henk C. Tijms,et al.  A fast algorithm for the transient reward distribution in continuous-time Markov chains , 2000, Oper. Res. Lett..

[25]  J. Ben Atkinson,et al.  Modeling and Analysis of Stochastic Systems , 1996 .

[26]  Holger Hermanns,et al.  A tool for model-checking Markov chains , 2003, International Journal on Software Tools for Technology Transfer.

[27]  José Meseguer,et al.  PMaude: Rewrite-based Specification Language for Probabilistic Object Systems , 2006, QAPL.

[28]  Vitaly Shmatikov,et al.  Analysis of probabilistic contract signing , 2002, J. Comput. Secur..

[29]  Farhad Nourai.  Automata theory I , 1973, CSC '73.

[30]  Daniel Lehmann,et al.  On the advantages of free choice: a symmetric and fully distributed solution to the dining philosophers problem , 1981, POPL '81.

[31]  Ward Whitt,et al.  Continuity of Generalized Semi-Markov Processes , 1980, Math. Oper. Res..

[32]  Christel Baier,et al.  On the Logical Characterisation of Performability Properties , 2000, ICALP.

[33]  Gerald S. Shedler.  Regenerative Stochastic Simulation , 1992 .

[34]  Mahesh Viswanathan,et al.  On Statistical Model Checking of Stochastic Systems , 2005, CAV.

[35]  Christel Baier,et al.  Approximate Symbolic Model Checking of Continuous-Time Markov Chains , 1999, CONCUR.

[36]  Kenneth L. McMillan,et al.  Symbolic model checking , 1992 .

[37]  Marta Z. Kwiatkowska,et al.  Quantitative Analysis With the Probabilistic Model Checker PRISM , 2006, QAPL.

[38]  Conrado Daws,et al.  Automatic verification of the IEEE 1394 root contention protocol with KRONOS and PRISM , 2002, International Journal on Software Tools for Technology Transfer.

[39]  Marta Z. Kwiatkowska,et al.  PRISM 2.0: a tool for probabilistic model checking , 2004, First International Conference on the Quantitative Evaluation of Systems, 2004. QEST 2004. Proceedings..

[40]  Thomas A. Henzinger,et al.  Reactive Modules , 1996, Proceedings 11th Annual IEEE Symposium on Logic in Computer Science.

[41]  Joost-Pieter Katoen,et al.  Discrete-Time Rewards Model-Checked , 2003, FORMATS.

[42]  Marta Z. Kwiatkowska,et al.  Performance analysis of probabilistic timed automata using digital clocks , 2003, Formal Methods Syst. Des..

[43]  Amir Pnueli,et al.  Verification of multiprocess probabilistic protocols , 2005, Distributed Computing.

[44]  Edsger W. Dijkstra,et al.  Hierarchical ordering of sequential processes , 1971, Acta Informatica.

[45]  R. Khan,et al.  Sequential Tests of Statistical Hypotheses. , 1972 .

[46]  Christel Baier,et al.  Model checking performability properties , 2002, Proceedings International Conference on Dependable Systems and Networks.

[47]  Stephan Merz,et al.  Model Checking , 2000 .

[48]  William H. Sanders,et al.  A new methodology for calculating distributions of reward accumulated during a finite interval , 1996, Proceedings of Annual Symposium on Fault Tolerant Computing.

[49]  Peter Buchholz,et al.  Complexity of Memory-Efficient Kronecker Operations with Applications to the Solution of Markov Models , 2000, INFORMS J. Comput..

[50]  Sergio Pissanetzky,et al.  Sparse Matrix Technology , 1984 .

[51]  Mahesh Viswanathan,et al.  Statistical Model Checking of Black-Box Probabilistic Systems , 2004, CAV.

[52]  Frank Ciesinski,et al.  On Probabilistic Computation Tree Logic , 2004, Validation of Stochastic Systems.

[53]  Marta Z. Kwiatkowska,et al.  PRISM: Probabilistic Symbolic Model Checker , 2002, Computer Performance Evaluation / TOOLS.

[54]  Luca de Alfaro,et al.  Symbolic Model Checking of Probabilistic Processes Using MTBDDs and the Kronecker Representation , 2000, TACAS.

[55]  Christel Baier,et al.  Model-Checking Algorithms for Continuous-Time Markov Chains , 2002, IEEE Trans. Software Eng..

[56]  Kishor S. Trivedi,et al.  Numerical Transient Solution of Finite Markovian Queueing Systems , 1992 .

[57]  Andrea Bianco,et al.  Model Checking of Probabalistic and Nondeterministic Systems , 1995, FSTTCS.

[58]  Joost-Pieter Katoen,et al.  Faster and Symbolic CTMC Model Checking , 2001, PAPM-PROBMIV.

[59]  Bowen Alpern,et al.  Recognizing safety and liveness , 2005, Distributed Computing.

[60]  Sérgio Vale Aguiar Campos,et al.  Symbolic Model Checking , 1993, CAV.

[61]  Masahiro Fujita,et al.  Multi-Terminal Binary Decision Diagrams: An Efficient Data Structure for Matrix Representation , 1997, Formal Methods Syst. Des..

[62]  Holger Hermanns,et al.  A Markov Chain Model Checker , 2000, TACAS.

[63]  Matthew Simon,et al.  Automata Theory , 1999 .

[64]  David Anthony Parker,et al.  Implementation of symbolic model checking for probabilistic systems , 2003 .

[65]  A. Jensen,et al.  Markoff chains as an aid in the study of Markoff processes , 1953 .

[66]  Alon Itai,et al.  Symmetry breaking in distributed networks , 1990, Inf. Comput..

[67]  Robert V. Hogg,et al.  Introduction to Mathematical Statistics. , 1966 .

[68]  Joost-Pieter Katoen,et al.  Safe On-The-Fly Steady-State Detection for Time-Bounded Reachability , 2005, Third International Conference on the Quantitative Evaluation of Systems - (QEST'06).

[69]  Kishor S. Trivedi.  Probability and Statistics with Reliability, Queuing, and Computer Science Applications , 1984 .

[70]  Håkan L. S. Younes,et al.  Statistical probabilistic model checking with a focus on time-bounded properties , 2006, Inf. Comput..

[71]  Robert K. Brayton,et al.  Verifying Continuous Time Markov Chains , 1996, CAV.

[72]  M. L. Plume,et al.  SPSS (Statistical Package for the Social Sciences) , 2002, Encyclopedia of Information Systems.

[73]  Wan Fokkink,et al.  Simplifying Itai-Rodeh Leader Election for Anonymous Rings , 2005, Electron. Notes Theor. Comput. Sci..

[74]  Christel Baier,et al.  Model checking for a probabilistic branching time logic with fairness , 1998, Distributed Computing.

[75]  Håkan L. S. Younes,et al.  Verification and planning for stochastic processes with asynchronous events , 2004 .

[76]  Louis A. Hageman,et al.  Iterative Solution of Large Linear Systems. , 1971 .

[77]  Bengt Jonsson,et al.  A logic for reasoning about time and reliability , 1990, Formal Aspects of Computing.

[78]  Rajiv Gupta,et al.  On randomization in sequential and distributed algorithms , 1994, CSUR.

[79]  Alan Bundy,et al.  Constructing Induction Rules for Deductive Synthesis Proofs , 2006, CLASE.

[80]  Paola Lecca,et al.  Cell Cycle Control in Eukaryotes: A BioSpi model , 2007, Electron. Notes Theor. Comput. Sci..

[81]  Kishor S. Trivedi,et al.  Stochastic Petri Net Models of Polling Systems , 1990, IEEE J. Sel. Areas Commun..