MDP optimal control under temporal logic constraints

In this paper, we develop a method to automatically generate a control policy for a dynamical system modeled as a Markov Decision Process (MDP). The control specification is given as a Linear Temporal Logic (LTL) formula over a set of propositions defined on the states of the MDP. We synthesize a control policy such that the MDP satisfies the given specification almost surely, if such a policy exists. In addition, we designate an “optimizing proposition” to be repeatedly satisfied, and we formulate a novel optimization criterion in terms of minimizing the expected cost in between satisfactions of this proposition. We propose a sufficient condition for a policy to be optimal, and develop a dynamic programming algorithm that synthesizes a policy that is optimal under some conditions, and sub-optimal otherwise. This problem is motivated by robotic applications requiring persistent tasks, such as environmental monitoring or data gathering, to be performed.

[1]  Dimitri P. Bertsekas,et al.  Dynamic Programming and Optimal Control, Two Volume Set , 1995 .

[2]  Calin Belta,et al.  Optimal path planning for surveillance with temporal-logic constraints* , 2011, Int. J. Robotics Res..

[3]  L. Hogben.  Handbook of Linear Algebra , 2006 .

[4]  Christel Baier,et al.  Principles of model checking , 2008 .

[5]  Amir Pnueli,et al.  Synthesis of Reactive(1) designs , 2006, J. Comput. Syst. Sci..

[6]  Leslie Pack Kaelbling,et al.  Collision Avoidance for Unmanned Aircraft using Markov Decision Processes , 2010 .

[7]  Christel Baier,et al.  Experiments with deterministic omega-automata for formulas of linear temporal logic , 2006, Theor. Comput. Sci..

[8]  Calin Belta,et al.  Motion planning and control from temporal logic specifications with probabilistic satisfaction guarantees , 2010, 2010 IEEE International Conference on Robotics and Automation.

[9]  Martin L. Puterman,et al.  Markov Decision Processes: Discrete Stochastic Dynamic Programming , 1994 .

[10]  Hadas Kress-Gazit,et al.  Where's Waldo? Sensor-Based Temporal Logic Motion Planning , 2007, Proceedings 2007 IEEE International Conference on Robotics and Automation.

[11]  Calin Belta,et al.  A Fully Automated Framework for Control of Linear Systems from Temporal Logic Specifications , 2008, IEEE Transactions on Automatic Control.

[12]  Thomas Wilke,et al.  Automata logics, and infinite games: a guide to current research , 2002 .

[13]  Ufuk Topcu,et al.  Receding horizon temporal logic planning for dynamical systems , 2009, Proceedings of the 48h IEEE Conference on Decision and Control (CDC) held jointly with 2009 28th Chinese Control Conference.

[14]  Thierry Siméon,et al.  The Stochastic Motion Roadmap: A Sampling Framework for Planning with Markov Motion Uncertainty , 2007, Robotics: Science and Systems.

[15]  Edmund M. Clarke,et al.  Model Checking , 1999, Handbook of Automated Reasoning.

[16]  Dimitri P. Bertsekas,et al.  Dynamic Programming and Optimal Control, Vol. II , 1976 .

[17]  Emilio Frazzoli,et al.  Sampling-based motion planning with deterministic μ-calculus specifications , 2009, Proceedings of the 48h IEEE Conference on Decision and Control (CDC) held jointly with 2009 28th Chinese Control Conference.

[18]  Christel Baier,et al.  PROBMELA: a modeling language for communicating probabilistic processes , 2004, Proceedings. Second ACM and IEEE International Conference on Formal Methods and Models for Co-Design, 2004. MEMOCODE '04..

[19]  Zohar Manna,et al.  Formal verification of probabilistic systems , 1997 .

[20]  K.J. Kyriakopoulos,et al.  Automatic synthesis of multi-agent motion tasks based on LTL specifications , 2004, 2004 43rd IEEE Conference on Decision and Control (CDC) (IEEE Cat. No.04CH37601).

[21]  Calin Belta,et al.  Dealing with Nondeterminism in Symbolic Control , 2008, HSCC.

[22]  Moshe Y. Vardi.  Probabilistic Linear-Time Model Checking: An Overview of the Automata-Theoretic Approach , 1999, ARTS.

[23]  Christel Baier,et al.  Controller Synthesis for Probabilistic Systems , 2004, IFIP TCS.

[24]  Calin Belta,et al.  LTL Control in Uncertain Environments with Probabilistic Satisfaction Guarantees , 2011, ArXiv.