On the connections between PCTL and dynamic programming

Probabilistic Computation Tree Logic (PCTL) is a well-known modal logic which has become a standard for expressing temporal properties of finite-state Markov chains in the context of automated model checking. In this paper, we consider PCTL for noncountable-space Markov chains, and we show that there is a substantial affinity between certain of its operators and problems of Dynamic Programming. We prove some basic properties of the solutions to the latter. We also provide two examples and demonstrate how recovery strategies in practical applications, which are naturally stated as reach-avoid problems, can be viewed as particular cases of PCTL formulas.

[1]  Holger Hermanns,et al.  A Markov Chain Model Checker , 2000, TACAS.

[2]  Nicolás Wolovick,et al.  Nondeterministic Labeled Markov Processes: Bisimulations and Logical Characterization , 2009, 2009 Sixth International Conference on the Quantitative Evaluation of Systems.

[3]  Michael Huth,et al.  On finite-state approximants for probabilistic computation tree logic , 2005, Theor. Comput. Sci..

[4]  Simin Nadjm-Tehrani,et al.  Formal Support for Quantitative Analysis of Residual Risks in Safety-Critical Systems , 2008, 2008 11th IEEE High Assurance Systems Engineering Symposium.

[5]  Ian M. Mitchell,et al.  Level Set Methods for Computation in Hybrid Systems , 2000, HSCC.

[6]  W. Fleming.  Book Review: Discrete-time Markov control processes: Basic optimality criteria , 1997 .

[7]  John Lygeros,et al.  Maximizing the probability of attaining a target prior to extinction , 2009, 0904.4143.

[8]  Hailiang Yang,et al.  Ruin problems for a discrete time risk model with random interest rate , 2006, Math. Methods Oper. Res..

[9]  Dimitri P. Bertsekas,et al.  Approximate Dynamic Programming , 2017, Encyclopedia of Machine Learning and Data Mining.

[10]  Jonathan Timmis,et al.  Modelling the Tunability of Early T Cell Signalling Events , 2008, ICARIS.

[11]  De Kok,et al.  Ruin probabilities with compounding assets for discrete time finite horizon problems , independent period claim sizes and general premium structure , 2010 .

[12]  Marta Z. Kwiatkowska,et al.  Probabilistic model checking of complex biological pathways , 2008, Theor. Comput. Sci..

[13]  John Lygeros,et al.  On reachability and minimum cost optimal control , 2004, Autom..

[14]  Qihe Tang,et al.  Precise estimates for the ruin probability in finite horizon in a discrete-time model with heavy-tai , 2003 .

[15]  Jun Cai,et al.  DISCRETE TIME RISK MODELS UNDER RATES OF INTEREST , 2002, Probability in the Engineering and Informational Sciences.

[16]  de Ag Ton Kok,et al.  Ruin probabilities with compounding assets for discrete time finite horizon problems, independent period claim sizes and general premium structure , 2003 .

[17]  Stanisa Dautovic,et al.  Dynamic Power Management of a System With a Two-Priority Request Queue Using Probabilistic-Model Checking , 2008, IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems.

[18]  John Lygeros,et al.  Reachability Analysis for Controlled Discrete Time Stochastic Hybrid Systems , 2006, HSCC.

[19]  Marta Z. Kwiatkowska,et al.  Using probabilistic model checking in systems biology , 2008, PERV.

[20]  Warren B. Powell,et al.  Approximate Dynamic Programming: Solving the Curses of Dimensionality (Wiley Series in Probability and Statistics) , 2007 .

[21]  Michel Loève,et al.  Probability Theory I , 1977 .

[22]  Gianfranco Ciardo,et al.  Logical and Stochastic Modeling with SMART , 2003, Computer Performance Evaluation / TOOLS.

[23]  Andrew Hinton,et al.  PRISM: A Tool for Automatic Verification of Probabilistic Systems , 2006, TACAS.

[24]  Jostein Paulsen.  Ruin Models with Investment Income , 2010 .

[25]  John Lygeros,et al.  Reachability Questions in Piecewise Deterministic Markov Processes , 2003, HSCC.

[26]  Panagiotis Katsaros,et al.  A Probabilistic Attacker Model for Quantitative Verification of DoS Security Threats , 2008, 2008 32nd Annual IEEE International Computer Software and Applications Conference.

[27]  Dimitri P. Bertsekas,et al.  Dynamic Programming and Optimal Control, Vol. II , 1976 .

[28]  John N. Tsitsiklis,et al.  Neuro-Dynamic Programming , 1996, Encyclopedia of Machine Learning.

[29]  Abbas Edalat,et al.  A logical characterization of bisimulation for labeled Markov processes , 1998, Proceedings. Thirteenth Annual IEEE Symposium on Logic in Computer Science (Cat. No.98CB36226).

[30]  Jean-Pierre Aubin,et al.  Viability theory , 1991 .

[31]  Joost-Pieter Katoen,et al.  A Markov reward model checker , 2005, Second International Conference on the Quantitative Evaluation of Systems (QEST'05).

[32]  H. Nyrhinen,et al.  On the ruin probabilities in a general economic environment , 1999 .

[33]  Bengt Jonsson,et al.  A logic for reasoning about time and reliability , 1990, Formal Aspects of Computing.

[34]  Jonathan W. Pitchford,et al.  Uncertainty and sustainability in fisheries and the benefit of marine protected areas , 2007 .

[35]  Dimitri P. Bertsekas,et al.  Dynamic Programming and Optimal Control, Two Volume Set , 1995 .

[36]  Gianfranco Ciardo,et al.  Logic and stochastic modeling with S m A r T , 2006, Perform. Evaluation.

[37]  Colin W. Clark,et al.  Mathematical Bioeconomics: The Optimal Management of Renewable Resources. , 1993 .

[38]  John Lygeros,et al.  A Probabilistic Reach-Avoid Problem for Controlled Discrete Time Stochastic Hybrid Systems , 2009, ADHS.

[39]  Jerzy A. Filar,et al.  Stochastic target hitting time and the problem of early retirement , 2004, IEEE Transactions on Automatic Control.

[40]  M. M. Rao,et al.  Probability Theory with Applications (Mathematics and Its Applications) , 2006 .

[41]  M. Rao.  Probability theory with applications , 1984 .

[42]  John Lygeros,et al.  Probabilistic reachability and safety for controlled discrete time stochastic hybrid systems , 2008, Autom..

[43]  Marta Z. Kwiatkowska,et al.  Stochastic Model Checking , 2007, SFM.

[44]  P. Strevens.  Iii , 1985 .

[45]  W. Rudin.  Principles of mathematical analysis , 1964 .

[46]  Jun Cai,et al.  Ruin probabilities with a Markov chain interest model , 2004 .

[47]  V. Borkar.  Probability Theory: An Advanced Course , 1995 .

[48]  Yijun Hu,et al.  Ruin probabilities for discrete time risk models with stochastic rates of interest , 2008 .

[49]  O. Hernández-Lerma,et al.  Discrete-time Markov control processes , 1999 .