The 10, 000 Facets of MDP Model Checking

This paper presents a retrospective view on probabilistic model checking. We focus on Markov decision processes (MDPs, for short). We survey the basic ingredients of MDP model checking and discuss its enormous developments since the seminal works by Courcoubetis and Yannakakis in the early 1990s. We discuss in particular the manifold facets of this field of research by surveying the verification of various MDP extensions, rich classes of properties, and their applications.

[1]  Marta Z. Kwiatkowska,et al.  Symmetry Reduction for Probabilistic Model Checking , 2006, CAV.

[2]  Pedro R. D'Argenio,et al.  Partial Order Reduction for Probabilistic Systems: A Revision for Distributed Schedulers , 2009, CONCUR.

[3]  Lijun Zhang,et al.  The Quest for Minimal Quotients for Probabilistic Automata , 2013, TACAS.

[4]  Kousha Etessami,et al.  Approximating the termination value of one-counter MDPs and stochastic games , 2013, Inf. Comput..

[5]  Moshe Y. Vardi.  Automatic verification of probabilistic concurrent finite state programs , 1985, 26th Annual Symposium on Foundations of Computer Science (sfcs 1985).

[6]  Sven Schewe,et al.  Finite optimal control for time-bounded reachability in CTMDPs and continuous-time Markov games , 2010, Acta Informatica.

[7]  Bernd Becker,et al.  Transient Reward Approximation for Continuous-Time Markov Chains , 2015, IEEE Transactions on Reliability.

[8]  Christel Baier,et al.  Principles of model checking , 2008 .

[9]  Christel Baier,et al.  Partial order reduction for probabilistic systems , 2004, First International Conference on the Quantitative Evaluation of Systems, 2004. QEST 2004. Proceedings..

[10]  Christel Baier,et al.  Probabilistic ω-automata , 2012, JACM.

[11]  Arnd Hartmanns,et al.  Sound statistical model checking for MDP using partial order and confluence reduction , 2014, International Journal on Software Tools for Technology Transfer.

[12]  Lu Feng,et al.  Learning-Based Compositional Verification for Synchronous Probabilistic Systems , 2011, ATVA.

[13]  Christel Baier,et al.  Deciding Bisimilarity and Similarity for Probabilistic Processes , 2000, J. Comput. Syst. Sci..

[14]  Florent Teichteil-Königsbuch.  Path-Constrained Markov Decision Processes: bridging the gap between probabilistic model-checking and decision-theoretic planning , 2012, ECAI.

[15]  Anja Vogler,et al.  Continuous Time Markov Decision Processes Theory And Applications , 2016 .

[16]  Sebastian Junges,et al.  Parameter Synthesis for Markov Models: Faster Than Ever , 2016, ATVA.

[17]  Dragan Bosnacki,et al.  Efficient GPU algorithms for parallel decomposition of graphs into strongly connected and maximal end components , 2016, Formal Methods Syst. Des..

[18]  Marta Z. Kwiatkowska,et al.  The PRISM Benchmark Suite , 2012, 2012 Ninth International Conference on Quantitative Evaluation of Systems.

[19]  Mahesh Viswanathan,et al.  A counterexample-guided abstraction-refinement framework for markov decision processes , 2008, TOCL.

[20]  Martin L. Puterman,et al.  Markov Decision Processes: Discrete Stochastic Dynamic Programming , 1994 .

[21]  Lijun Zhang,et al.  Deciding Bisimilarities on Distributions , 2013, QEST.

[22]  Pedro R. D'Argenio,et al.  Partial order reduction on concurrent probabilistic programs , 2004, First International Conference on the Quantitative Evaluation of Systems, 2004. QEST 2004. Proceedings..

[23]  Henrik Ejersbo Jensen,et al.  Reachability Analysis of Probabilistic Systems by Successive Refinements , 2001, PAPM-PROBMIV.

[24]  Luca Pulina,et al.  Evaluating probabilistic model checking tools for verification of robot control policies , 2016, AI Commun..

[25]  Kim G. Larsen,et al.  Learning deterministic probabilistic automata from a model checking perspective , 2016, Machine Learning.

[26]  R. I. Bahar,et al.  Algebraic decision diagrams and their applications , 1993, Proceedings of 1993 International Conference on Computer Aided Design (ICCAD).

[27]  Luca de Alfaro,et al.  How to Specify and Verify the Long-Run Average Behavior of Probabilistic Systems , 1998, LICS.

[28]  Nils Jansen,et al.  Minimal counterexamples for linear-time probabilistic verification , 2014, Theor. Comput. Sci..

[29]  Joost-Pieter Katoen,et al.  Three-valued abstraction for probabilistic systems , 2012, J. Log. Algebraic Methods Program..

[30]  Dragan Bosnacki,et al.  Parallel probabilistic model checking on general purpose graphics processors , 2010, International Journal on Software Tools for Technology Transfer.

[31]  Greg N. Frederickson,et al.  Sequencing Tasks with Exponential Service Times to Minimize the Expected Flow Time or Makespan , 1981, JACM.

[32]  Kim G. Larsen,et al.  Infinite Runs in Weighted Timed Automata with Energy Constraints , 2008, FORMATS.

[33]  Christian von Essen,et al.  Synthesizing efficient systems in probabilistic environments , 2015, Acta Informatica.

[34]  Kim Guldstrand Larsen,et al.  Specification and refinement of probabilistic processes , 1991, [1991] Proceedings Sixth Annual IEEE Symposium on Logic in Computer Science.

[35]  R. Bellman.  A Markovian Decision Process , 1957 .

[36]  Reiner Hähnle,et al.  Deductive Software Verification: From Pen-and-Paper Proofs to Industrial Tools , 2019, Computing and Software Science.

[37]  Amir Pnueli,et al.  Verification of multiprocess probabilistic protocols , 2005, Distributed Computing.

[38]  Christel Baier,et al.  Weight monitoring with linear temporal logic: complexity and decidability , 2014, CSL-LICS.

[39]  Christel Baier,et al.  Computing Conditional Probabilities in Markovian Models Efficiently , 2014, TACAS.

[40]  Lijun Zhang,et al.  Probabilistic Logical Characterization , 2011, Inf. Comput..

[41]  Krishnendu Chatterjee,et al.  Trading Performance for Stability in Markov Decision Processes , 2013, 2013 28th Annual ACM/IEEE Symposium on Logic in Computer Science.

[42]  Kousha Etessami,et al.  One-counter Markov decision processes , 2009, SODA '10.

[43]  Mickael Randour,et al.  Percentile queries in multi-dimensional Markov decision processes , 2017, Formal Methods Syst. Des..

[44]  Sebastian Junges,et al.  Sequential Convex Programming for the Efficient Verification of Parametric MDPs , 2017, TACAS.

[45]  Christel Baier,et al.  Quantitative Analysis under Fairness Constraints , 2009, ATVA.

[46]  Christel Baier,et al.  Maximizing the Conditional Expected Reward for Reaching the Goal , 2017, TACAS.

[47]  Christel Baier,et al.  Reachability in continuous-time Markov reward decision processes , 2008, Logic and Automata.

[48]  Anne Condon,et al.  On the undecidability of probabilistic planning and related stochastic optimization problems , 2003, Artif. Intell..

[49]  Véronique Bruyère,et al.  Symblicit algorithms for optimal strategy synthesis in monotonic Markov decision processes , 2014, SYNT.

[50]  Kousha Etessami,et al.  Recursive Markov Decision Processes and Recursive Stochastic Games , 2005, ICALP.

[51]  Bernd Becker,et al.  Fiber-Optic Fabry–Pérot Sensor Based on Periodic Focusing Effect of Graded-Index Multimode Fibers , 2010, IEEE Photonics Technology Letters.

[52]  Hongfei Fu,et al.  Maximal Cost-Bounded Reachability Probability on Continuous-Time Markov Decision Processes , 2013, FoSSaCS.

[53]  Ezio Bartocci,et al.  Policy Learning for Time-Bounded Reachability in Continuous-Time Markov Decision Processes via Doubly-Stochastic Gradient Ascent , 2016, QEST.

[54]  Hugo Gimbert,et al.  Deciding the Value 1 Problem for ]-acyclic Partially Observable Markov Decision Processes , 2013 .

[55]  Sebastian Junges,et al.  Markov automata with multiple objectives , 2017, Formal Methods in System Design.

[56]  Enrico Macii,et al.  Markovian analysis of large finite state machines , 1996, IEEE Trans. Comput. Aided Des. Integr. Circuits Syst..

[57]  Peter Buchholz,et al.  Multi-Objective Approaches to Markov Decision Processes with Uncertain Transition Parameters , 2017, VALUETOOLS.

[58]  Enrico Macii,et al.  Algebric Decision Diagrams and Their Applications , 1997, ICCAD '93.

[59]  Clare Dixon,et al.  Analysing robot swarm behaviour via probabilistic model checking , 2012, Robotics Auton. Syst..

[60]  Bengt Jonsson,et al.  A logic for reasoning about time and reliability , 1990, Formal Aspects of Computing.

[61]  K. Subramani,et al.  Compositional Bisimulation Minimization for Interval Markov Decision Processes , 2016, LATA.

[62]  Krishnendu Chatterjee,et al.  What is decidable about partially observable Markov decision processes with ω-regular objectives , 2013, J. Comput. Syst. Sci..

[63]  Kim G. Larsen,et al.  Continuous-Time Models for System Design and Analysis , 2019, Computing and Software Science.

[64]  A. Shwartz,et al.  Handbook of Markov decision processes : methods and applications , 2002 .

[65]  Lijun Zhang,et al.  On Probabilistic Automata in Continuous Time , 2010, 2010 25th Annual IEEE Symposium on Logic in Computer Science.

[66]  Lijun Zhang,et al.  Synthesis for PCTL in Parametric Markov Decision Processes , 2011, NASA Formal Methods.

[67]  Shimon Whiteson,et al.  A Survey of Multi-Objective Sequential Decision-Making , 2013, J. Artif. Intell. Res..

[68]  Eugene A. Feinberg,et al.  Handbook of Markov Decision Processes , 2002 .

[69]  Gethin Norman,et al.  Model checking for probabilistic timed automata , 2012, Formal Methods in System Design.

[70]  Michael Ben-Or,et al.  Another advantage of free choice (Extended Abstract): Completely asynchronous agreement protocols , 1983, PODC '83.

[71]  Zohar Manna,et al.  Formal verification of probabilistic systems , 1997 .

[72]  Holger Hermanns,et al.  Explicit Model Checking of Very Large MDP Using Partitioning and Secondary Storage , 2015, ATVA.

[73]  Annabelle McIver,et al.  Operational versus weakest pre-expectation semantics for the probabilistic guarded command language , 2014, Perform. Evaluation.

[74]  Petr Novotný,et al.  Optimizing the Expected Mean Payoff in Energy Markov Decision Processes , 2016, ATVA.

[75]  Lijun Zhang,et al.  Time-Bounded Reachability Probabilities in Continuous-Time Markov Decision Processes , 2010, 2010 Seventh International Conference on the Quantitative Evaluation of Systems.

[76]  Luca de Alfaro,et al.  Computing Minimum and Maximum Reachability Times in Probabilistic Systems , 1999, CONCUR.

[77]  Krishnendu Chatterjee,et al.  Optimal cost almost-sure reachability in POMDPs , 2014, Artif. Intell..

[78]  Sebastian Junges,et al.  Safety-Constrained Reinforcement Learning for MDPs , 2015, TACAS.

[79]  Christoph Haase,et al.  The Odds of Staying on Budget , 2014, ICALP.

[80]  Mihalis Yannakakis,et al.  The complexity of probabilistic verification , 1995, JACM.

[81]  Ronald A. Howard,et al.  Dynamic Probabilistic Systems , 1971 .

[82]  Marta Z. Kwiatkowska,et al.  Automatic verification of real-time systems with discrete probability distributions , 1999, Theor. Comput. Sci..

[83]  Xianping Guo,et al.  Continuous-Time Markov Decision Processes: Theory and Applications , 2009 .

[84]  Thomas A. Henzinger,et al.  Markov Decision Processes with Multiple Objectives , 2006, STACS.

[85]  Luca de Alfaro,et al.  Magnifying-Lens Abstraction for Markov Decision Processes , 2007, CAV.

[86]  Edmund M. Clarke,et al.  Symbolic Model Checking: 10^20 States and Beyond , 1990, Inf. Comput..

[87]  Christel Baier,et al.  Symbolic Model Checking for Probabilistic Processes , 1997, ICALP.

[88]  Hongyang Qu,et al.  Quantitative Multi-objective Verification for Probabilistic Systems , 2011, TACAS.

[89]  Pedro R. D'Argenio,et al.  Partial order reduction on concurrent probabilistic programs , 2004 .

[90]  Masahiro Fujita,et al.  Multi-Terminal Binary Decision Diagrams: An Efficient Data Structure for Matrix Representation , 1997, Formal Methods Syst. Des..

[91]  Christel Baier,et al.  Computing Quantiles in Markov Reward Models , 2013, FoSSaCS.

[92]  Holger Hermanns,et al.  Multi-objective Robust Strategy Synthesis for Interval Markov Decision Processes , 2017, QEST.

[93]  Michael Ben-Or,et al.  Another advantage of free choice (Extended Abstract): Completely asynchronous agreement protocols , 1983, PODC '83.

[94]  Jan Kretínský,et al.  From LTL to Deterministic Automata: A Safraless Compositional Approach , 2014, CAV.

[95]  Christel Baier,et al.  Model checking for a probabilistic branching time logic with fairness , 1998, Distributed Computing.

[96]  Markus Siegle,et al.  Markov Automata: Deciding weak bisimulation by means of non-naïvely vanishing states , 2012, Inf. Comput..

[97]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[98]  Holger Hermanns,et al.  Reward-Bounded Reachability Probability for Uncertain Weighted MDPs , 2016, VMCAI.

[99]  D. J. White,et al.  A Survey of Applications of Markov Decision Processes , 1993 .

[100]  Marta Z. Kwiatkowska,et al.  A game-based abstraction-refinement framework for Markov decision processes , 2010, Formal Methods Syst. Des..

[101]  Lijun Zhang,et al.  Best Probabilistic Transformers , 2010, VMCAI.

[102]  R. Segala,et al.  Automatic Verification of Real-Time Systems with Discrete Probability Distributions , 1999, ARTS.

[103]  Kousha Etessami,et al.  Analysis of probabilistic processes and automata theory , 2021, Handbook of Automata Theory.

[104]  Vahid Hashemi,et al.  Decision algorithms for modelling, optimal control and verification of probabilistic systems , 2017 .

[105]  Marta Z. Kwiatkowska,et al.  Probabilistic symbolic model checking with PRISM: a hybrid approach , 2004, International Journal on Software Tools for Technology Transfer.

[106]  Holger Hermanns,et al.  Deciding probabilistic automata weak bisimulation: theory and practice , 2016, Formal Aspects of Computing.

[107]  Daniel Gebler,et al.  On Abstraction of Probabilistic Systems , 2012, ROCKS.

[108]  Alastair F. Donaldson,et al.  Language-Level Symmetry Reduction for Probabilistic Model Checking , 2009, 2009 Sixth International Conference on the Quantitative Evaluation of Systems.

[109]  Luca de Alfaro,et al.  Symbolic Model Checking of Probabilistic Processes Using MTBDDs and the Kronecker Representation , 2000, TACAS.

[110]  Christel Baier,et al.  Stochastic Timed Automata , 2014, Log. Methods Comput. Sci..

[111]  Benjamin Monmege,et al.  Reachability in MDPs: Refining Convergence of Value Iteration , 2014, RP.

[112]  Kim G. Larsen,et al.  Abstract Probabilistic Automata , 2011, VMCAI.

[113]  Hadas Kress-Gazit,et al.  Analyzing and revising synthesized controllers for robots with sensing and actuation errors , 2015, Int. J. Robotics Res..

[114]  Joost-Pieter Katoen,et al.  The Probabilistic Model Checking Landscape* , 2016, 2016 31st Annual ACM/IEEE Symposium on Logic in Computer Science (LICS).

[115]  Hugo Gimbert,et al.  Deciding the Value 1 Problem for $\sharp$ -acyclic Partially Observable Markov Decision Processes , 2014, SOFSEM.

[116]  Kim G. Larsen,et al.  A modal process logic , 1988, [1988] Proceedings. Third Annual Information Symposium on Logic in Computer Science.

[117]  Krishnendu Chatterjee,et al.  Temporal Specifications with Accumulative Values , 2011, 2011 IEEE 26th Annual Symposium on Logic in Computer Science.

[118]  Véronique Bruyère,et al.  Meet Your Expectations With Guarantees: Beyond Worst-Case Synthesis in Quantitative Games , 2013, STACS.

[119]  Pierre Wolper,et al.  An Automata-Theoretic Approach to Automatic Program Verification (Preliminary Report) , 1986, LICS.

[120]  Christel Baier,et al.  Trade-off analysis meets probabilistic model checking , 2014, CSL-LICS.

[121]  Holger Hermanns,et al.  Probabilistic Bisimulation: Naturally on Distributions , 2014, CONCUR.

[122]  David Anthony Parker,et al.  Implementation of symbolic model checking for probabilistic systems , 2003 .

[123]  Marta Z. Kwiatkowska,et al.  Model checking for probability and time: from theory to practice , 2003, 18th Annual IEEE Symposium of Logic in Computer Science, 2003. Proceedings..

[124]  Lijun Zhang,et al.  Model Checking Algorithms for CTMDPs , 2011, CAV.

[125]  Krishnendu Chatterjee,et al.  Energy and Mean-Payoff Parity Markov Decision Processes , 2011, MFCS.

[126]  Nicolas Halbwachs,et al.  Minimal Model Generation , 1990, CAV.

[127]  Nancy A. Lynch,et al.  Probabilistic Simulations for Probabilistic Processes , 1994, Nord. J. Comput..

[128]  Sven Schewe,et al.  MDPs with energy-parity objectives , 2017, 2017 32nd Annual ACM/IEEE Symposium on Logic in Computer Science (LICS).

[129]  Joost-Pieter Katoen,et al.  Quantitative model-checking of controlled discrete-time Markov processes , 2014, Inf. Comput..

[130]  Christel Baier,et al.  On Decision Problems for Probabilistic Büchi Automata , 2008, FoSSaCS.

[131]  Kim G. Larsen,et al.  Reduction and Refinement Strategies for Probabilistic Analysis , 2002, PAPM-PROBMIV.

[132]  Marta Z. Kwiatkowska,et al.  PRISM 4.0: Verification of Probabilistic Real-Time Systems , 2011, CAV.

[133]  Krishnendu Chatterjee,et al.  Two Views on Multiple Mean-Payoff Objectives in Markov Decision Processes , 2011, 2011 IEEE 26th Annual Symposium on Logic in Computer Science.

[134]  Sebastian Junges,et al.  A Storm is Coming: A Modern Probabilistic Model Checker , 2017, CAV.

[135]  Lijun Zhang,et al.  Probabilistic CEGAR , 2008, CAV.

[136]  Krishnendu Chatterjee,et al.  Verification of Markov Decision Processes Using Learning Algorithms , 2014, ATVA.

[137]  Jan Kretínský,et al.  From LTL to deterministic automata , 2014, Formal Methods Syst. Des..

[138]  Rajeev Alur,et al.  A Theory of Timed Automata , 1994, Theor. Comput. Sci..

[139]  W. Lovejoy.  A survey of algorithmic methods for partially observed Markov decision processes , 1991 .

[140]  Andrea Bianco,et al.  Model Checking of Probabalistic and Nondeterministic Systems , 1995, FSTTCS.

[141]  L. D. Alfaro.  The Verification of Probabilistic Systems Under Memoryless Partial-Information Policies is Hard , 1999 .

[142]  Robert Givan,et al.  Bounded Parameter Markov Decision Processes , 1997, ECP.

[143]  Kim G. Larsen,et al.  Uppaal SMC tutorial , 2015, International Journal on Software Tools for Technology Transfer.

[144]  Christel Baier,et al.  Efficient computation of time-bounded reachability probabilities in uniform continuous-time Markov decision processes , 2005, Theor. Comput. Sci..

[145]  Holger Hermanns,et al.  Polynomial time decision algorithms for probabilistic automata , 2015, Inf. Comput..

[146]  Peter van Rossum,et al.  Conditional Probabilities over Probabilistic and Nondeterministic Systems , 2008, TACAS.

[147]  Kousha Etessami,et al.  Multi-objective Model Checking of Markov Decision Processes , 2007, TACAS.

[148]  Annabelle McIver,et al.  Abstraction, Refinement and Proof for Probabilistic Systems , 2004, Monographs in Computer Science.

[149]  Nils Jansen,et al.  Understanding Probabilistic Programs , 2015, Correct System Design.

[150]  John S. Edwards,et al.  Linear Programming and Finite Markovian Control Problems , 1983 .

[151]  Kim G. Larsen,et al.  Bisimulation through Probabilistic Testing , 1991, Inf. Comput..

[152]  Christel Baier,et al.  Energy-Utility Quantiles , 2014, NASA Formal Methods.

[153]  Nancy A. Lynch,et al.  Impossibility of distributed consensus with one faulty process , 1985, JACM.

[154]  Joost-Pieter Katoen,et al.  Delayed Nondeterminism in Continuous-Time Markov Decision Processes , 2009, FoSSaCS.

[155]  Holger Hermanns,et al.  Optimal Continuous Time Markov Decisions , 2015, ATVA.

[156]  Christel Baier,et al.  Verifying nondeterministic probabilistic channel systems against ω-regular linear-time properties , 2005, TOCL.

[157]  Joost-Pieter Katoen,et al.  Time-Bounded Reachability in Tree-Structured QBDs by Abstraction , 2009, 2009 Sixth International Conference on the Quantitative Evaluation of Systems.

[158]  Marta Z. Kwiatkowska,et al.  Permissive Controller Synthesis for Probabilistic Systems , 2014, TACAS.

[159]  Ana Sokolova,et al.  Information Hiding in Probabilistic Concurrent Systems , 2010, 2010 Seventh International Conference on the Quantitative Evaluation of Systems.

[160]  John N. Tsitsiklis,et al.  An Analysis of Stochastic Shortest Path Problems , 1991, Math. Oper. Res..

[161]  Nick Hawes,et al.  Optimal Policy Generation for Partially Satisfiable Co-Safe LTL Specifications , 2015, IJCAI.