Optimal Control of Markov Decision Processes With Linear Temporal Logic Constraints

In this paper, we develop a method to automatically generate a control policy for a dynamical system modeled as a Markov Decision Process (MDP). The control specification is given as a Linear Temporal Logic (LTL) formula over a set of propositions defined on the states of the MDP. Motivated by robotic applications requiring persistent tasks, such as environmental monitoring and data gathering, we synthesize a control policy that minimizes the expected cost between satisfying instances of a particular proposition over all policies that maximize the probability of satisfying the given LTL specification. Our approach is based on the definition of a novel optimization problem that extends the existing average cost per stage problem. We propose a sufficient condition for a policy to be optimal, and develop a dynamic programming algorithm that synthesizes a policy that is optimal for a set of LTL specifications.

[1]  Calin Belta,et al.  A Fully Automated Framework for Control of Linear Systems from Temporal Logic Specifications , 2008, IEEE Transactions on Automatic Control.

[2]  K.J. Kyriakopoulos,et al.  Automatic synthesis of multi-agent motion tasks based on LTL specifications , 2004, 2004 43rd IEEE Conference on Decision and Control (CDC) (IEEE Cat. No.04CH37601).

[3]  Calin Belta,et al.  Optimal path planning for surveillance with temporal-logic constraints* , 2011, Int. J. Robotics Res..

[4]  Calin Belta,et al.  LTL Control in Uncertain Environments with Probabilistic Satisfaction Guarantees , 2011, ArXiv.

[5]  Ufuk Topcu,et al.  Receding Horizon Temporal Logic Planning , 2012, IEEE Transactions on Automatic Control.

[6]  Leslie Pack Kaelbling,et al.  Collision Avoidance for Unmanned Aircraft using Markov Decision Processes , 2010 .

[7]  Amir Pnueli,et al.  Synthesis of Reactive(1) Designs , 2006, VMCAI.

[8]  Edmund M. Clarke,et al.  Model Checking , 1999, Handbook of Automated Reasoning.

[9]  Thierry Siméon,et al.  The Stochastic Motion Roadmap: A Sampling Framework for Planning with Markov Motion Uncertainty , 2007, Robotics: Science and Systems.

[10]  James E. Ward,et al.  Approaches to sensitivity analysis in linear programming , 1991 .

[11]  C. Baier,et al.  Experiments with Deterministic ω-Automata for Formulas of Linear Temporal Logic , 2005 .

[12]  Dimitri P. Bertsekas,et al.  Dynamic Programming and Optimal Control, Two Volume Set , 1995 .

[13]  L. Hogben.  Handbook of Linear Algebra , 2006 .

[14]  Christel Baier,et al.  Principles of model checking , 2008 .

[15]  Calin Belta,et al.  Dealing with Nondeterminism in Symbolic Control , 2008, HSCC.

[16]  Christel Baier,et al.  Experiments with deterministic omega-automata for formulas of linear temporal logic , 2006, Theor. Comput. Sci..

[17]  L. D. Alfaro.  The Verification of Probabilistic Systems Under Memoryless Partial-Information Policies is Hard , 1999 .

[18]  Martin L. Puterman,et al.  Markov Decision Processes: Discrete Stochastic Dynamic Programming , 1994 .

[19]  Calin Belta,et al.  MDP optimal control under temporal logic constraints , 2011, IEEE Conference on Decision and Control and European Control Conference.

[20]  Moshe Y. Vardi.  Probabilistic Linear-Time Model Checking: An Overview of the Automata-Theoretic Approach , 1999, ARTS.

[21]  Christel Baier,et al.  Controller Synthesis for Probabilistic Systems , 2004, IFIP TCS.

[22]  Yushan Chen,et al.  LTL robot motion control based on automata learning of environmental dynamics , 2012, 2012 IEEE International Conference on Robotics and Automation.

[23]  Dimitri P. Bertsekas,et al.  Dynamic Programming and Optimal Control, Vol. II , 1976 .

[24]  Calin Belta,et al.  Temporal logic control of discrete-time piecewise affine systems , 2009, Proceedings of the 48h IEEE Conference on Decision and Control (CDC) held jointly with 2009 28th Chinese Control Conference.

[25]  Mihalis Yannakakis,et al.  Markov Decision Processes and Regular Events (Extended Abstract) , 1990, ICALP.

[26]  Christel Baier,et al.  PROBMELA: a modeling language for communicating probabilistic processes , 2004, Proceedings. Second ACM and IEEE International Conference on Formal Methods and Models for Co-Design, 2004. MEMOCODE '04..

[27]  Calin Belta,et al.  Temporal Logic Motion Planning and Control With Probabilistic Satisfaction Guarantees , 2012, IEEE Transactions on Robotics.

[28]  Hadas Kress-Gazit,et al.  Where's Waldo? Sensor-Based Temporal Logic Motion Planning , 2007, Proceedings 2007 IEEE International Conference on Robotics and Automation.

[29]  Arkadi Nemirovski,et al.  Robust solutions of uncertain linear programs , 1999, Oper. Res. Lett..

[30]  Emilio Frazzoli,et al.  Sampling-based motion planning with deterministic μ-calculus specifications , 2009, Proceedings of the 48h IEEE Conference on Decision and Control (CDC) held jointly with 2009 28th Chinese Control Conference.

[31]  Thomas Wilke,et al.  Automata logics, and infinite games: a guide to current research , 2002 .

[32]  Thomas Wilke,et al.  Automata Logics, and Infinite Games , 2002, Lecture Notes in Computer Science.