PAC Statistical Model Checking for Markov Decision Processes and Stochastic Games

Statistical model checking (SMC) is a technique for analysis of probabilistic systems that may be (partially) unknown. We present an SMC algorithm for (unbounded) reachability yielding probably approximately correct (PAC) guarantees on the results. We consider both the setting (i) with no knowledge of the transition function (with the only quantity required a bound on the minimum transition probability) and (ii) with knowledge of the topology of the underlying graph. On the one hand, it is the first algorithm for stochastic games. On the other hand, it is the first practical algorithm even for Markov decision processes. Compared to previous approaches where PAC guarantees require running times longer than the age of universe even for systems with a handful of states, our algorithm often yields reasonably precise results within minutes, not requiring the knowledge of mixing time or the topology of the whole model.

[1]  Kim G. Larsen,et al.  Statistical Model Checking for Networks of Priced Timed Automata , 2011, FORMATS.

[2]  Michael L. Littman,et al.  Markov Games as a Framework for Multi-Agent Reinforcement Learning , 1994, ICML.

[3]  Daniel Kroening,et al.  Certified Reinforcement Learning with Logic Guidance , 2019, Artif. Intell..

[4]  Edmund M. Clarke,et al.  Bayesian statistical model checking with application to Stateflow/Simulink verification , 2010, Formal Methods in System Design.

[5]  Kim G. Larsen,et al.  Priced Timed Automata and Statistical Model Checking , 2013, IFM.

[6]  Marta Z. Kwiatkowska,et al.  PRISM 4.0: Verification of Probabilistic Real-Time Systems , 2011, CAV.

[7]  Taolue Chen,et al.  Verifying Team Formation Protocols with Probabilistic Model Checking , 2011, CLIMA.

[8]  Kim G. Larsen,et al.  Time for Statistical Model Checking of Real-Time Systems , 2011, CAV.

[9]  Axel Legay,et al.  Scalable Verification of Markov Decision Processes , 2013, SEFM Workshops.

[10]  S. Shankar Sastry,et al.  A learning based approach to control synthesis of Markov decision processes for linear temporal logic specifications , 2014, 53rd IEEE Conference on Decision and Control.

[11]  Krishnendu Chatterjee,et al.  Verification of Markov Decision Processes Using Learning Algorithms , 2014, ATVA.

[12]  Ufuk Topcu,et al.  Probably Approximately Correct MDP Learning and Control With Temporal Logic Constraints , 2014, Robotics: Science and Systems.

[13]  W. Hoeffding.  Probability Inequalities for sums of Bounded Random Variables , 1963 .

[14]  Benjamin Monmege,et al.  Interval iteration algorithm for MDPs and IMDPs , 2017, Theor. Comput. Sci..

[15]  Daniel Kroening,et al.  Logically-Correct Reinforcement Learning , 2018, ArXiv.

[16]  Axel Legay,et al.  A Bayesian Approach to Model Checking Biological Systems , 2009, CMSB.

[17]  Thomas Hérault,et al.  Approximate Probabilistic Model Checking , 2004, VMCAI.

[18]  Axel Legay,et al.  PLASMA-lab: A Flexible, Distributable Statistical Model Checking Library , 2013, QEST.

[19]  Ufuk Topcu,et al.  Probably Approximately Correct Learning in Stochastic Games with Temporal Logic Specifications , 2016, IJCAI.

[20]  Edmund M. Clarke,et al.  Statistical Model Checking for Markov Decision Processes , 2012, 2012 Ninth International Conference on Quantitative Evaluation of Systems.

[21]  Kim G. Larsen,et al.  UPPAAL-SMC: Statistical Model Checking for Priced Timed Automata , 2012, QAPL.

[22]  Radu Calinescu,et al.  Compositional Reverification of Probabilistic Safety Properties for Large-Scale Complex IT Systems , 2012, Monterey Workshop.

[23]  Axel Legay,et al.  Statistical Abstraction and Model-Checking of Large Heterogeneous Systems , 2010 .

[24]  Håkan L. S. Younes,et al.  Probabilistic Verification of Discrete Event Systems Using Acceptance Sampling , 2002, CAV.

[25]  Cyrille Jégourel,et al.  A Platform for High Performance Statistical Model Checking - PLASMA , 2012, TACAS.

[26]  Nihal Pekergin,et al.  Statistical Model Checking Using Perfect Simulation , 2009, ATVA.

[27]  Edmund M. Clarke,et al.  Statistical Model Checking for Cyber-Physical Systems , 2011, ATVA.

[28]  Lihong Li,et al.  PAC model-free reinforcement learning , 2006, ICML.

[29]  Thomas A. Henzinger,et al.  Faster Statistical Model Checking for Unbounded Temporal Properties , 2016, TACAS.

[30]  Håkan L. S. Younes,et al.  Statistical Verification of Probabilistic Properties with Unbounded Until , 2010, SBMF.

[31]  Anne Condon,et al.  The Complexity of Stochastic Games , 1992, Inf. Comput..

[32]  Geoffrey J. Gordon,et al.  Bounded real-time dynamic programming: RTDP with monotone upper bounds and performance guarantees , 2005, ICML.

[33]  Sven Schewe,et al.  Omega-Regular Objectives in Model-Free Reinforcement Learning , 2018, TACAS.

[34]  Krishnendu Chatterjee,et al.  A survey of stochastic ω-regular games , 2012, J. Comput. Syst. Sci..

[35]  David Hsu,et al.  Statistical Model Checking Based Calibration and Analysis of Bio-pathway Models , 2013, CMSB.

[36]  Martin L. Puterman,et al.  Markov Decision Processes: Discrete Stochastic Dynamic Programming , 1994 .

[37]  Ronen I. Brafman,et al.  A Near-Optimal Poly-Time Algorithm for Learning a class of Stochastic Games , 1999, IJCAI.

[38]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[39]  S. Lakshmivarahan,et al.  Learning Algorithms for Two-Person Zero-Sum Stochastic Games with Incomplete Information , 1981, Math. Oper. Res..

[40]  Jan Kretínský,et al.  Value Iteration for Simple Stochastic Games: Stopping Criterion and Learning Algorithm , 2018, CAV.

[41]  Sean Sedwards,et al.  Lightweight Statistical Model Checking in Nondeterministic Continuous Time , 2018, ISoLA.

[42]  Jan Kretínský,et al.  Of Cores: A Partial-Exploration Framework for Markov Decision Processes , 2019, CONCUR.

[43]  Holger Hermanns,et al.  Simulation and Statistical Model Checking for Modestly Nondeterministic Models , 2012, MMB/DFT.

[44]  Arnd Hartmanns,et al.  The Quantitative Verification Benchmark Set , 2019, TACAS.

[45]  Fabrice Saffre,et al.  Host selection through collective decision , 2012, TAAS.

[46]  Mahesh Viswanathan,et al.  Statistical Model Checking of Black-Box Probabilistic Systems , 2004, CAV.

[47]  Richard Lassaigne,et al.  Approximate planning and verification for large Markov decision processes , 2012, SAC '12.

[48]  Holger Hermanns,et al.  Partial Order Methods for Statistical Model Checking and Simulation , 2011, FMOODS/FORTE.

[49]  Axel Legay,et al.  Smart sampling for lightweight verification of Markov decision processes , 2014, International Journal on Software Tools for Technology Transfer.

[50]  Martin Fränzle,et al.  Confidence Bounds for Statistical Model Checking of Probabilistic Hybrid Systems , 2012, FORMATS.

[51]  Richard Lassaigne,et al.  Probabilistic Verification and Approximation , 2006, Electron. Notes Theor. Comput. Sci..

[52]  Håkan L. S. Younes,et al.  Numerical vs. statistical probabilistic model checking , 2006, International Journal on Software Tools for Technology Transfer.

[53]  Taolue Chen,et al.  PRISM-games: A Model Checker for Stochastic Multi-Player Games , 2013, TACAS.

[54]  T. E. S. Raghavan,et al.  Algorithms for stochastic games — A survey , 1991, ZOR Methods Model. Oper. Res..

[55]  Samik Basu,et al.  A bounded statistical approach for model checking of unbounded until properties , 2010, ASE.

[56]  Chong Li,et al.  Model-Free Reinforcement Learning , 2019, Reinforcement Learning for Cyber-Physical Systems.

[57]  Kim G. Larsen,et al.  Optimizing Control Strategy Using Statistical Model Checking , 2013, NASA Formal Methods.

[58]  Krishnendu Chatterjee,et al.  Value Iteration , 2008, 25 Years of Model Checking.

[59]  Taolue Chen,et al.  Automatic verification of competitive stochastic systems , 2012, Formal Methods in System Design.

[60]  Mahesh Viswanathan,et al.  On Statistical Model Checking of Stochastic Systems , 2005, CAV.

[61]  Kim G. Larsen,et al.  Statistical Model Checking for Stochastic Hybrid Systems , 2012, HSB.

[62]  Kim G. Larsen.  Statistical Model Checking, Refinement Checking, Optimization, ... for Stochastic Hybrid Systems , 2012, FORMATS.