Statistic Analysis for Probabilistic Processes

We associate a statistical vector to a trace and a geometrical embedding to a Markov Decision Process, based on a distance on words, and study basic Membership and Equivalence problems. The Membership problem for a trace \textit{w} and a Markov Decision Process \textit{S} decides if there exists a strategy on \textit{S} which generates with high probability traces close to \textit{w}. We prove that Membership of a trace is \emph{testable} and Equivalence of MDPs is polynomial time approximable. For Probabilistic Automata, Membership is not testable, and approximate Equivalence is undecidable. We give a class of properties, based on results concerning the structure of the tail sigma-field of a finite Markov chain, which characterizes equivalent Markov Decision Processes in this context.

[1]  Thomas A. Henzinger,et al.  Equivalence of Labeled Markov Chains , 2008, Int. J. Found. Comput. Sci..

[2]  Radha Jagadeesan,et al.  Metrics for labelled Markov processes , 2004, Theor. Comput. Sci..

[3]  John N. Tsitsiklis,et al.  On the Empirical State-Action Frequencies in Markov Decision Processes Under General Policies , 2005, Math. Oper. Res..

[4]  Luca de Alfaro,et al.  Computing Minimum and Maximum Reachability Times in Probabilistic Systems , 1999, CONCUR.

[5]  Mihalis Yannakakis,et al.  The complexity of probabilistic verification , 1995, JACM.

[6]  James Worrell,et al.  A behavioural pseudometric for probabilistic transition systems , 2005, Theor. Comput. Sci..

[7]  Wen-Guey Tzeng,et al.  A Polynomial-Time Algorithm for the Equivalence of Probabilistic Automata , 1992, SIAM J. Comput..

[8]  GoldreichOded,et al.  Property testing and its connection to learning and approximation , 1998 .

[9]  Allan F. Abrahamse.  The Tail Field of a Markov Chain , 1969 .

[10]  Ronitt Rubinfeld,et al.  Robust Characterizations of Polynomials with Applications to Program Testing , 1996, SIAM J. Comput..

[11]  Salil P. Vadhan,et al.  Computational Complexity , 2005, Encyclopedia of Cryptography and Security.

[12]  K. Fernow.  New York , 1896, American Potato Journal.

[13]  Dana Ron,et al.  Property testing and its connection to learning and approximation , 1998, JACM.

[14]  Mathieu Tracol,et al.  Fast convergence to state-action frequency polytopes for MDPs , 2009, Oper. Res. Lett..

[15]  Subhash Khot,et al.  Linear Equations Modulo 2 and the L1 Diameter of Convex Bodies , 2008, 48th Annual IEEE Symposium on Foundations of Computer Science (FOCS'07).

[16]  Dmitry Krass,et al.  Achieving Target State-Action Frequencies in Multichain Average-Reward Markov Decision Processes , 2002, Math. Oper. Res..

[17]  Anne Condon,et al.  On the Undecidability of Probabilistic Planning and Infinite-Horizon Partially Observable Markov Decision Problems , 1999, AAAI/IAAI.

[18]  David A. Freedman,et al.  The Tail $\sigma$-Field of a Markov Chain and a Theorem of Orey , 1964 .

[19]  John S. Edwards,et al.  Linear Programming and Finite Markovian Control Problems , 1983 .

[20]  Frédéric Magniez,et al.  Approximate Satisfiability and Equivalence , 2006, 21st Annual IEEE Symposium on Logic in Computer Science (LICS'06).

[21]  P. Shields.  The Ergodic Theory of Discrete Sample Paths , 1996 .

[22]  L. D. Alfaro.  The Verification of Probabilistic Systems Under Memoryless Partial-Information Policies is Hard , 1999 .

[23]  Vincent D. Blondel,et al.  Undecidable Problems for Probabilistic Automata of Fixed Dimension , 2003, Theory of Computing Systems.

[24]  Azaria Paz,et al.  Probabilistic automata , 2003 .

[25]  John N. Tsitsiklis,et al.  A survey of computational complexity results in systems and control , 2000, Autom..

[26]  Martin L. Puterman,et al.  Markov Decision Processes: Discrete Stochastic Dynamic Programming , 1994 .

[27]  Cyrus Derman,et al.  Finite State Markovian Decision Processes , 1970 .

[28]  Benjamin Weiss,et al.  How Sampling Reveals a Process , 1990 .

[29]  Moshe Y. Vardi.  Automatic verification of probabilistic concurrent finite state programs , 1985, 26th Annual Symposium on Foundations of Computer Science (sfcs 1985).

[30]  John N. Tsitsiklis,et al.  The Complexity of Markov Decision Processes , 1987, Math. Oper. Res..

[31]  Nancy A. Lynch,et al.  Probabilistic Simulations for Probabilistic Processes , 1994, Nord. J. Comput..