Scalable Verification of Markov Decision Processes

Markov decision processes (MDP) are useful to model concurrent process optimisation problems, but verifying them with numerical methods is often intractable. Existing approximative approaches do not scale well and are limited to memoryless schedulers. Here we present the basis of scalable verification for MDPSs, using an \(\mathcal {O}(1)\) memory representation of history-dependent schedulers. We thus facilitate scalable learning techniques and the use of massively parallel verification.

[1]  Holger Hermanns,et al.  Partial Order Methods for Statistical Model Checking and Simulation , 2011, FMOODS/FORTE.

[2]  Sean R Eddy,et al.  What is dynamic programming? , 2004, Nature Biotechnology.

[3]  W. Marsden.  I and J , 2012 .

[4]  Bengt Jonsson,et al.  A logic for reasoning about time and reliability , 1990, Formal Aspects of Computing.

[5]  W. G. Horner,et al.  A new method of solving numerical equations of all orders, by continuous approximation , 1815 .

[6]  Yishay Mansour,et al.  A Sparse Sampling Algorithm for Near-Optimal Planning in Large Markov Decision Processes , 1999, Machine Learning.

[7]  Xin-She Yang,et al.  Introduction to Algorithms , 2021, Nature-Inspired Optimization Algorithms.

[8]  H. Chernoff.  A Measure of Asymptotic Efficiency for Tests of a Hypothesis Based on the sum of Observations , 1952 .

[9]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[10]  Joseph Sifakis,et al.  Model checking , 1996, Handbook of Automated Reasoning.

[11]  Richard Lassaigne,et al.  Approximate planning and verification for large Markov decision processes , 2012, SAC '12.

[12]  M. Okamoto.  Some inequalities relating to the partial sum of binomial probabilities , 1959 .

[13]  Christel Baier,et al.  Principles of model checking , 2008 .

[14]  Arnd Hartmanns,et al.  On-the-Fly Confluence Detection for Statistical Model Checking , 2013, NASA Formal Methods.

[15]  David Thomas,et al.  The Art in Computer Programming , 2001 .

[16]  Steven I. Marcus,et al.  A survey of some simulation-based algorithms for Markov decision processes , 2007, Commun. Inf. Syst..

[17]  Pierre L'Ecuyer,et al.  Improved long-period generators based on linear recurrences modulo 2 , 2004, TOMS.

[18]  D. J. White,et al.  A Survey of Applications of Markov Decision Processes , 1993 .

[19]  Andrea Bianco,et al.  Model Checking of Probabalistic and Nondeterministic Systems , 1995, FSTTCS.

[20]  Martin L. Puterman,et al.  Markov Decision Processes: Discrete Stochastic Dynamic Programming , 1994 .

[21]  J. Wrench.  Table errata: The art of computer programming, Vol. 2: Seminumerical algorithms (Addison-Wesley, Reading, Mass., 1969) by Donald E. Knuth , 1970 .

[22]  Axel Legay,et al.  PLASMA-lab: A Flexible, Distributable Statistical Model Checking Library , 2013, QEST.

[23]  Edmund M. Clarke,et al.  Statistical Model Checking for Markov Decision Processes , 2012, 2012 Ninth International Conference on Quantitative Evaluation of Systems.

[24]  Cyrille Jégourel,et al.  Importance Splitting for Statistical Model Checking Rare Properties , 2013, CAV.

[25]  R. Khan,et al.  Sequential Tests of Statistical Hypotheses. , 1972 .

[26]  Marta Z. Kwiatkowska,et al.  Probabilistic Model Checking of the IEEE 802.11 Wireless Local Area Network Protocol , 2002, PAPM-PROBMIV.

[27]  Annabelle McIver,et al.  An expectation transformer approach to predicate abstraction and data independence for probabilistic programs , 2010, QAPL.

[28]  Håkan L. S. Younes,et al.  Probabilistic Verification of Discrete Event Systems Using Acceptance Sampling , 2002, CAV.