Probabilistically-sound and asymptotically-optimal algorithm for stochastic control with trajectory constraints

In this paper, we consider a class of stochastic optimal control problems with trajectory constraints. As a special case, we can constrain the probability that a system enters undesirable regions to remain below a certain threshold. We extend the incremental Markov Decision Process (iMDP) algorithm, which is a new computationally-efficient and asymptotically-optimal sampling-based tool for stochastic optimal control, to approximate arbitrarily well an optimal feedback policy of the constrained problem. We show that with probability one, in the presence of trajectory constraints, the sequence of policies returned from the algorithm is both probabilistically sound and asymptotically optimal. We demonstrate the proposed algorithm on motion planning and control problems subject to bounded collision probability in uncertain cluttered environments.

[1]  John N. Tsitsiklis,et al.  A survey of computational complexity results in systems and control , 2000, Autom..

[2]  S. LaValle.  Rapidly-exploring random trees : a new tool for path planning , 1998 .

[3]  Thierry Siméon,et al.  The Stochastic Motion Roadmap: A Sampling Framework for Planning with Markov Motion Uncertainty , 2007, Robotics: Science and Systems.

[4]  Ioannis Karatzas,et al.  Brownian Motion and Stochastic Calculus , 1987 .

[5]  James P. Ostrowski,et al.  Motion planning a aerial robot using rapidly-exploring random trees with dynamic constraints , 2003, 2003 IEEE International Conference on Robotics and Automation (Cat. No.03CH37422).

[6]  Boumediène Chentouf,et al.  The finite element approximation of Hamilton-Jacobi-Bellman equations: the noncoercive case , 2004, Appl. Math. Comput..

[7]  Emanuel Todorov,et al.  Stochastic Optimal Control and Estimation Methods Adapted to the Noise Characteristics of the Sensorimotor System , 2005, Neural Computation.

[8]  Emilio Frazzoli,et al.  An incremental sampling-based algorithm for stochastic optimal control , 2012, 2012 IEEE International Conference on Robotics and Automation.

[9]  B. Faverjon,et al.  Probabilistic Roadmaps for Path Planning in High-Dimensional Con(cid:12)guration Spaces , 1996 .

[10]  Nicholas Roy,et al.  Finite-Time Regional Verification of Stochastic Nonlinear Systems , 2012 .

[11]  Emilio Frazzoli,et al.  Sampling-based algorithms for optimal motion planning , 2011, Int. J. Robotics Res..

[12]  N. Roy,et al.  Regression-based LP solver for chance-constrained finite horizon optimal control with nonconvex constraints , 2011, Proceedings of the 2011 American Control Conference.

[13]  J. Tsitsiklis,et al.  Efficient algorithms for globally optimal trajectories , 1994, Proceedings of 1994 33rd IEEE Conference on Decision and Control.

[14]  J. Quadrat.  Numerical methods for stochastic control problems in continuous time , 1994 .

[15]  Masahiro Ono,et al.  A Probabilistic Particle-Control Approximation of Chance-Constrained Stochastic Predictive Control , 2010, IEEE Transactions on Robotics.

[16]  Andrew W. Moore,et al.  Variable Resolution Discretization in Optimal Control , 2002, Machine Learning.

[17]  Donald E. Kirk,et al.  Optimal control theory : an introduction , 1970 .

[18]  Lydia E. Kavraki,et al.  Probabilistic roadmaps for path planning in high-dimensional configuration spaces , 1996, IEEE Trans. Robotics Autom..

[19]  John Rust.  Using Randomization to Break the Curse of Dimensionality , 1997 .

[20]  J. Tsitsiklis,et al.  An optimal one-way multigrid algorithm for discrete-time stochastic control , 1991 .

[21]  G. Thompson,et al.  Optimal Control Theory: Applications to Management Science and Economics , 2000 .

[22]  Kok Lay Teo,et al.  Numerical Solution of Hamilton-Jacobi-Bellman Equations by an Upwind Finite Volume Method , 2003, J. Glob. Optim..

[23]  G. Thompson,et al.  Optimal control theory : applications to management science , 1984 .

[24]  W. Fleming,et al.  Stochastic Optimal Control, International Finance and Debt , 2002, SSRN Electronic Journal.

[25]  M. Pavon,et al.  Lagrange approach to the optimal control of diffusions , 1993 .

[26]  L. Grüne.  An adaptive grid scheme for the discrete Hamilton-Jacobi-Bellman equation , 1997 .

[27]  Jonathan P. How,et al.  Real-Time Motion Planning With Applications to Autonomous Urban Driving , 2009, IEEE Transactions on Control Systems Technology.

[28]  John Rust.  A Comparison of Policy Iteration Methods for Solving Continuous-State, Infinite-Horizon Markovian Decision Problems Using Random, Quasi-Random, and Deterministic Discretizations , 1997 .

[29]  Michele Pavon,et al.  Solving optimal control problems by means of general Lagrange functionals , 2001, Autom..