Trace Refinement in Labelled Markov Decision Processes

Given two labelled Markov decision processes (MDPs), the trace-refinement problem asks whether for all strategies of the first MDP there exists a strategy of the second MDP such that the induced labelled Markov chains are trace-equivalent. We show that this problem is decidable in polynomial time if the second MDP is a Markov chain. The algorithm is based on new results on a particular notion of bisimulation between distributions over the states. However, we show that the general trace-refinement problem is undecidable, even if the first MDP is a Markov chain. Decidability of those problems was stated as open in 2008. We further study the decidability and complexity of the trace-refinement problem provided that the strategies are restricted to be memoryless.

[1]  Richard J. Lipton,et al.  On the complexity of space bounded interactive proofs , 1989, 30th Annual Symposium on Foundations of Computer Science.

[2]  H. James Hoover,et al.  Limits to Parallel Computation: P-Completeness Theory , 1995 .

[3]  Thomas A. Henzinger,et al.  Equivalence of Labeled Markov Chains , 2008, Int. J. Found. Comput. Sci..

[4]  John F. Canny,et al.  Some algebraic and geometric computations in PSPACE , 1988, STOC '88.

[5]  Stephen A. Vavasis,et al.  On the Complexity of Nonnegative Matrix Factorization , 2007, SIAM J. Optim..

[6]  Wen-Guey Tzeng,et al.  A Polynomial-Time Algorithm for the Equivalence of Probabilistic Automata , 1992, SIAM J. Comput..

[7]  Sanjeev Arora,et al.  Computing a nonnegative matrix factorization -- provably , 2011, STOC '12.

[8]  James Renegar,et al.  On the Computational Complexity and Geometry of the First-Order Theory of the Reals, Part I: Introduction. Preliminaries. The Geometry of Semi-Algebraic Sets. The Decision Problem for the Existential Theory of the Reals , 1992, J. Symb. Comput..

[9]  Yaroslav Shitov,et al.  A universality theorem for nonnegative matrix factorizations , 2016, 1606.09068.

[10]  Lijun Zhang,et al.  When Equivalence and Bisimulation Join Forces in Probabilistic Automata , 2014, FM.

[11]  Stefan Kiefer,et al.  Stability and Complexity of Minimising Probabilistic Automata , 2014, ICALP.

[12]  Xin-She Yang,et al.  Introduction to Algorithms , 2021, Nature-Inspired Optimization Algorithms.

[13]  James Renegar,et al.  On the Computational Complexity and Geometry of the First-Order Theory of the Reals, Part III: Quantifier Elimination , 1992, J. Symb. Comput..

[14]  Holger Hermanns,et al.  Probabilistic Bisimulation: Naturally on Distributions , 2014, CONCUR.

[15]  Marcel Paul Schützenberger,et al.  On the Definition of a Family of Automata , 1961, Inf. Control..

[16]  Mariëlle Stoelinga,et al.  An Introduction to Probabilistic Automata , 2002, Bull. EATCS.

[17]  James Renegar.  On the computational complexity and geome-try of the first-order theory of the reals , 1992 .

[18]  Joël Ouaknine,et al.  APEX: An Analyzer for Open Probabilistic Programs , 2012, CAV.

[19]  Yann Strozecki,et al.  Approximate Verification and Enumeration Problems , 2012, ICTAC.

[20]  Nathanaël Fijalkow,et al.  Undecidability results for probabilistic automata , 2017, SIGL.

[21]  J. Renegar,et al.  On the Computational Complexity and Geometry of the First-Order Theory of the Reals, Part I , 1989 .

[22]  Wen-Guey Tzeng.  On Path Equivalence of Nondeterministic Finite Automata , 1996, Inf. Process. Lett..

[23]  John Fearnley,et al.  Reachability in two-clock timed automata is PSPACE-Complete , 2013, ICALP 2013.

[24]  L. G. H. Cijan.  A polynomial algorithm in linear programming , 1979 .

[25]  Vincent D. Blondel,et al.  Undecidable Problems for Probabilistic Automata of Fixed Dimension , 2003, Theory of Computing Systems.

[26]  Eric Allender,et al.  Improved Lower Bounds for the Cycle Detection Problem , 1985, Theor. Comput. Sci..

[27]  Yuan Feng,et al.  Quantum Markov chains: Description of hybrid systems, decidability of equivalence, and model checking linear-time properties , 2015, Inf. Comput..

[28]  Joël Ouaknine,et al.  Language Equivalence for Probabilistic Automata , 2011, CAV.

[29]  Joel E. Cohen,et al.  Nonnegative ranks, decompositions, and factorizations of nonnegative matrices , 1993 .

[30]  Marieke Huisman,et al.  Confidentiality for Probabilistic Multi-threaded Programs and Its Verification , 2012, ESSoS.

[31]  John Fearnley,et al.  Reachability in two-clock timed automata is PSPACE-complete , 2013, Inf. Comput..