Reduction Techniques for Model Checking Markov Decision Processes

The quantitative analysis of a randomized system, modeled by a Markov decision process, against an LTL formula can be performed by a combination of graph algorithms, automata-theoretic concepts and numerical methods to compute maximal or minimal reachability probabilities. In this paper, we present various reduction techniques that serve to improve the performance of the quantitative analysis, and report on their implementation on the top of the probabilistic model checker \LiQuor. Although our techniques are purely heuristic and cannot improve the worst-case time complexity of standard algorithms for the quantitative analysis, a series of examples illustrates that the proposed methods can yield a major speed-up.

[1]  Doron A. Peled.  Partial order reduction: Linear and branching temporal logics and process algebras , 1996, Partial Order Methods in Verification.

[2]  Andrea Bianco,et al.  Model Checking of Probabalistic and Nondeterministic Systems , 1995, FSTTCS.

[3]  L. D. Alfaro.  The Verification of Probabilistic Systems Under Memoryless Partial-Information Policies is Hard , 1999 .

[4]  Christel Baier,et al.  Experiments with deterministic omega-automata for formulas of linear temporal logic , 2006, Theor. Comput. Sci..

[5]  Peng Dai,et al.  FASTER DYNAMIC PROGRAMMING FOR MARKOV DECISION PROCESSES , 2007 .

[6]  Kim G. Larsen,et al.  Reduction and Refinement Strategies for Probabilistic Analysis , 2002, PAPM-PROBMIV.

[7]  Doron A. Peled,et al.  All from One, One for All: on Model Checking Using Representatives , 1993, CAV.

[8]  Pedro R. D'Argenio,et al.  Partial order reduction on concurrent probabilistic programs , 2004, First International Conference on the Quantitative Evaluation of Systems, 2004. QEST 2004. Proceedings..

[9]  Seif Haridi,et al.  Distributed Algorithms , 1992, Lecture Notes in Computer Science.

[10]  Gerard J. Holzmann,et al.  The SPIN Model Checker - primer and reference manual , 2003 .

[11]  Christel Baier,et al.  Model checking for a probabilistic branching time logic with fairness , 1998, Distributed Computing.

[12]  Christel Baier,et al.  On-the-Fly Stuttering in the Construction of Deterministic omega -Automata , 2007, CIAA.

[13]  Christel Baier,et al.  PROBMELA: a modeling language for communicating probabilistic processes , 2004, Proceedings. Second ACM and IEEE International Conference on Formal Methods and Models for Co-Design, 2004. MEMOCODE '04..

[14]  Luca de Alfaro,et al.  Temporal Logics for the Specification of Performance and Reliability , 1997, STACS.

[15]  Henrik Ejersbo Jensen,et al.  Reachability Analysis of Probabilistic Systems by Successive Refinements , 2001, PAPM-PROBMIV.

[16]  Martin L. Puterman,et al.  Markov Decision Processes: Discrete Stochastic Dynamic Programming , 1994 .

[17]  Gerard J. Holzmann,et al.  Partial Order Methods in Verification , 1997 .

[18]  Peng Dai,et al.  Topological Value Iteration Algorithm for Markov Decision Processes , 2007, IJCAI.

[19]  Mihalis Yannakakis,et al.  The complexity of probabilistic verification , 1995, JACM.

[20]  S. Safra,et al.  On the complexity of omega -automata , 1988, [Proceedings 1988] 29th Annual Symposium on Foundations of Computer Science.

[21]  Gerard J. Holzmann,et al.  An improvement in formal verification , 1994, FORTE.

[22]  Mihalis Yannakakis,et al.  Markov Decision Processes and Regular Events (Extended Abstract) , 1990, ICALP.

[23]  Christel Baier,et al.  Partial order reduction for probabilistic systems , 2004, First International Conference on the Quantitative Evaluation of Systems, 2004. QEST 2004. Proceedings..

[24]  Joost-Pieter Katoen,et al.  Bisimulation Minimisation Mostly Speeds Up Probabilistic Model Checking , 2007, TACAS.

[25]  Peng Dai,et al.  Prioritizing Bellman Backups without a Priority Queue , 2007, ICAPS.

[26]  Doron A. Peled,et al.  Static Partial Order Reduction , 1998, TACAS.

[27]  C. Atkeson,et al.  Prioritized Sweeping : Reinforcement Learning withLess Data and Less Real , 1993 .

[28]  Edmund M. Clarke,et al.  Model Checking , 1999, Handbook of Automated Reasoning.

[29]  Marta Z. Kwiatkowska,et al.  Probabilistic symbolic model checking with PRISM: a hybrid approach , 2004, International Journal on Software Tools for Technology Transfer.

[30]  Christel Baier,et al.  LiQuor: A tool for Qualitative and Quantitative Linear Time analysis of Reactive Systems , 2006, Third International Conference on the Quantitative Evaluation of Systems - (QEST'06).

[31]  Andrew W. Moore,et al.  Prioritized Sweeping: Reinforcement Learning with Less Data and Less Time , 1993, Machine Learning.