Analysis of probabilistic processes and automata theory

This chapter surveys some basic algorithms for analyzing Ma rkov chains (MCs) and Markov decision processes (MDPs), and discusses their comp utational complexity. We focus on discrete-time processes, and we consider both finite-state mod ls as well as countably infinite-state models that are finitely-presented. The analyses we will pri ma ily focus on are hitting (reachability) probabilities andω-regular model checking, but we will also discuss various re ward-based analyses. Although it may not be evident at first, there are fruitful con nections between automata theory and stochastic processes. Firstly, and not surprisingly, ω-automata play a naturally important role for specifyingω-regular properties of sample paths (trajectories) of stoc hastic processes. Computing the probability of the event that a random sample path s isfies a givenω-regular property constitutes the (linear-time) model checking problem for p r babilistic systems. Secondly, it turns out that there are close relationships be tween classic infinite-state automatatheoretic models and classic denumerably infinite-state st ochastic processes, even though these models were developed independently in separate mathemati cal communities. Roughly speaking, some classic stochastic processes share their underlying s tate transition systems with corresponding classic automata-theoretic models. Furthermore, exploit ing these connections to automata theory is fruitful for the algorithmic analysis of such stochastic processes, and for their controlled MDP extensions. This holds even when the analyses are much simpl er than model checking, such as computing (optimal) hitting probabilities. A number of important infinite-state stochastic models conn ected with automata theory can be captured as (restricted fragments of) recursive Markov chainsand recursive Markov decision processes , which are obtained by adding a natural recursion feature to finite-state MCs and MDPs. Key computational problems for analyzing classes of recurs ive MCs and MDPs can be reduced to computing theleast fixed point(LFP) solution of corresponding classes of monotonesystems of nonlinear equations. The complexity of computing the LFP for such equations is a intriguing problem, with connections to several areas of research in th eoretical computer science. D R A FT 2 K. Etessami

[1]  Kousha Etessami,et al.  Recursive Markov chains, stochastic grammars, and monotone systems of nonlinear equations , 2005, JACM.

[2]  Javier Esparza,et al.  Model checking probabilistic pushdown automata , 2004, Proceedings of the 19th Annual IEEE Symposium on Logic in Computer Science, 2004..

[3]  Kousha Etessami,et al.  Recursive Markov Decision Processes and Recursive Stochastic Games , 2005, ICALP.

[4]  Kousha Etessami,et al.  Recursive Stochastic Games with Positive Rewards , 2008, ICALP.

[5]  Kousha Etessami,et al.  One-counter Markov decision processes , 2009, SODA '10.

[6]  Pierre Wolper,et al.  An Automata-Theoretic Approach to Automatic Program Verification (Preliminary Report) , 1986, LICS.

[7]  Uri Zwick,et al.  The Complexity of Mean Payoff Games on Graphs , 1996, Theor. Comput. Sci..

[8]  Thomas A. Henzinger,et al.  Concurrent reachability games , 2007, Theor. Comput. Sci..

[9]  Peter Bro Miltersen,et al.  Strategy Iteration Is Strongly Polynomial for 2-Player Turn-Based Stochastic Games with a Constant Discount Factor , 2010, JACM.

[10]  Yinyu Ye,et al.  A New Complexity Result on Solving the Markov Decision Problem , 2005, Math. Oper. Res..

[11]  Kousha Etessami,et al.  Quasi-Birth-Death Processes, Tree-Like QBDs, Probabilistic 1-Counter Automata, and Pushdown Systems , 2008, 2008 Fifth International Conference on Quantitative Evaluation of Systems.

[12]  Markov decision processes and regular events , 1998, IEEE Trans. Autom. Control..

[13]  Kousha Etessami,et al.  Upper Bounds for Newton’s Method on Monotone Polynomial Systems, and P-Time Model Checking of Probabilistic One-Counter Automata , 2013, J. ACM.

[14]  Tomás Brázdil,et al.  Efficient Analysis of Probabilistic Programs with an Unbounded Counter , 2014, JACM.

[15]  Peter Bro Miltersen,et al.  2 The Task of a Numerical Analyst , 2022 .

[16]  Kousha Etessami,et al.  Recursive Concurrent Stochastic Games , 2008, Log. Methods Comput. Sci..

[17]  Beatrice Meini,et al.  Numerical methods for structured Markov chains , 2005 .

[18]  Kousha Etessami,et al.  One-Counter Stochastic Games , 2010, FSTTCS.

[19]  Kousha Etessami,et al.  Polynomial Time Algorithms for Multi-Type Branching Processes and Stochastic Context-Free Grammars , 2012, ArXiv.

[20]  Yinyu Ye,et al.  A New Complexity Result on Solving the , 2005 .

[21]  L. Shapley,et al.  Stochastic Games* , 1953, Proceedings of the National Academy of Sciences.

[22]  Kousha Etessami,et al.  Approximating the Termination Value of One-Counter MDPs and Stochastic Games , 2011, ICALP.

[23]  J. Filar,et al.  Competitive Markov Decision Processes , 1996 .

[24]  Tomás Brázdil,et al.  Reachability in recursive Markov decision processes , 2008, Inf. Comput..

[25]  Tom Burr,et al.  Introduction to Matrix Analytic Methods in Stochastic Modeling , 2001, Technometrics.

[26]  Michael Luttenberger,et al.  Computing the Least Fixed Point of Positive Polynomial Systems , 2010, SIAM J. Comput..

[27]  Marta Z. Kwiatkowska,et al.  PRISM 4.0: Verification of Probabilistic Real-Time Systems , 2011, CAV.

[28]  D. Welsh.  Review: Theodore E. Harris, The Theory of Branching Processes , 1965 .

[29]  Fred Kröger,et al.  Temporal Logic of Programs , 1987, EATCS Monographs on Theoretical Computer Science.

[30]  PatersonMike,et al.  The complexity of mean payoff games on graphs , 1996 .

[31]  Mihalis Yannakakis,et al.  Markov Decision Processes and Regular Events (Extended Abstract) , 1990, ICALP.

[32]  T. E. Harris,et al.  The Theory of Branching Processes. , 1963 .

[33]  Moshe Y. Vardi.  Automatic verification of probabilistic concurrent finite state programs , 1985, 26th Annual Symposium on Foundations of Computer Science (sfcs 1985).

[34]  Mihalis Yannakakis,et al.  Verifying temporal properties of finite-state probabilistic programs , 1988, [Proceedings 1988] 29th Annual Symposium on Foundations of Computer Science.

[35]  Kousha Etessami,et al.  Quasi-Birth-Death Processes, Tree-Like QBDs, Probabilistic 1-Counter Automata, and Pushdown Systems , 2008, QEST.

[36]  Anne Condon,et al.  The Complexity of Stochastic Games , 1992, Inf. Comput..

[37]  D. Vere-Jones.  Markov Chains , 1972, Nature.

[38]  Kai Lai Chung,et al.  A Course in Probability Theory , 1949 .

[39]  Mihalis Yannakakis,et al.  The complexity of probabilistic verification , 1995, JACM.

[40]  Christel Baier,et al.  Principles of model checking , 2008 .

[41]  Kousha Etessami,et al.  Model Checking of Recursive Probabilistic Systems , 2012, TOCL.

[42]  Kousha Etessami,et al.  Stochastic Context-Free Grammars, Regular Languages, and Newton's Method , 2013, ICALP.

[43]  U. Rieder,et al.  Markov Decision Processes , 2010 .