Control of Markov decision processes from PCTL specifications

We address the problem of controlling a Markov Decision Process (MDP) such that the probability of satisfying a temporal logic specification over a set of properties associated to its states is maximized. We focus on specifications given as formulas of Probabilistic Computation Tree Logic (PCTL) and show that controllers can be synthesized by adapting existing PCTL model checking algorithms. We illustrate the approach by applying it to the automatic deployment of a mobile robot in an indoor-like environment with respect to a PCTL specification.

[1]  Thomas Bak,et al.  Multi-Robot Motion Planning: A Timed Automata Approach , 2004 .

[2]  Calin Belta,et al.  A Fully Automated Framework for Control of Linear Systems from Temporal Logic Specifications , 2008, IEEE Transactions on Automatic Control.

[3]  Hadas Kress-Gazit,et al.  Where's Waldo? Sensor-Based Temporal Logic Motion Planning , 2007, Proceedings 2007 IEEE International Conference on Robotics and Automation.

[4]  Jan van Leeuwen,et al.  Handbook of Theoretical Computer Science, Vol. B: Formal Models and Semantics , 1994 .

[5]  Christel Baier,et al.  Model checking for a probabilistic branching time logic with fairness , 1998, Distributed Computing.

[6]  Thomas Bak,et al.  Planning : A Timed Automata Approach , 2004 .

[7]  K.J. Kyriakopoulos,et al.  Automatic synthesis of multi-agent motion tasks based on LTL specifications , 2004, 2004 43rd IEEE Conference on Decision and Control (CDC) (IEEE Cat. No.04CH37601).

[8]  George J. Pappas,et al.  Translating Temporal Logic to Controller Specifications , 2006, Proceedings of the 45th IEEE Conference on Decision and Control.

[9]  Ivana Černá,et al.  Distributed Qualitative LTL Model Checking of Markov DecisionProcesses , 2006 .

[10]  Kousha Etessami,et al.  Multi-Objective Model Checking of Markov Decision Processes , 2007, Log. Methods Comput. Sci..

[11]  Calin Belta,et al.  Dealing with Nondeterminism in Symbolic Control , 2008, HSCC.

[12]  L. D. Alfaro.  The Verification of Probabilistic Systems Under Memoryless Partial-Information Policies is Hard , 1999 .

[13]  Jan J. M. M. Rutten,et al.  Mathematical techniques for analyzing concurrent and probabilistic systems , 2004, CRM monograph series.

[14]  Mihalis Yannakakis,et al.  Markov Decision Processes and Regular Events (Extended Abstract) , 1990, ICALP.

[15]  Calin Belta,et al.  Motion planning and control from temporal logic specifications with probabilistic satisfaction guarantees , 2010, 2010 IEEE International Conference on Robotics and Automation.

[16]  Bengt Jonsson,et al.  A logic for reasoning about time and reliability , 1990, Formal Aspects of Computing.

[17]  Amir Pnueli,et al.  Synthesis of Reactive(1) Designs , 2006, VMCAI.

[18]  Edmund M. Clarke,et al.  Model Checking , 1999, Handbook of Automated Reasoning.