Path integrals and symmetry breaking for optimal control theory

This paper considers linear-quadratic control of a non-linear dynamical system subject to arbitrary cost. I show that for this class of stochastic control problems the non-linear Hamilton–Jacobi–Bellman equation can be transformed into a linear equation. The transformation is similar to the transformation used to relate the classical Hamilton–Jacobi equation to the Schrodinger equation. As a result of the linearity, the usual backward computation can be replaced by a forward diffusion process that can be computed by stochastic integration or by the evaluation of a path integral. It is shown how in the deterministic limit the Pontryagin minimum principle formalism is recovered. The significance of the path integral approach is that it forms the basis for a number of efficient computational methods, such as Monte Carlo sampling, the Laplace approximation and the variational approximation. We show the effectiveness of the first two methods in a number of examples. Examples are given that show the qualitative difference between stochastic and deterministic control and the occurrence of symmetry breaking as a function of the noise.

[1]  E. Madelung,et al.  Quantentheorie in hydrodynamischer Form , 1927 .

[2]  L. S. Pontryagin,et al.  Mathematical Theory of Optimal Processes , 1962 .

[3]  M. L. Chambers.  The Mathematical Theory of Optimal Processes , 1965 .

[4]  R. Feynman,et al.  Quantum Mechanics and Path Integrals , 1965 .

[5]  Edward Nelson.  Dynamical Theories of Brownian Motion , 1967 .

[6]  W. Miller.  Path integral representation of the reaction rate constant in quantum mechanical transition state theory , 1975 .

[7]  W. Fleming,et al.  Deterministic and Stochastic Optimal Control , 1975 .

[8]  Robert E. Kalaba,et al.  Selected Papers On Mathematical Trends In Control Theory , 1977 .

[9]  W. Fleming.  Exit probabilities and optimal stochastic control , 1977 .

[10]  P. Wolynes,et al.  Convenient and accurate discretized path integral methods for equilibrium quantum mechanical calculations , 1981 .

[11]  Francesco Guerra,et al.  Structural aspects of stochastic mechanics and stochastic field theory , 1981 .

[12]  C. Morris,et al.  Voltage oscillations in the barnacle giant muscle fiber. , 1981, Biophysical journal.

[13]  J. Doll,et al.  A Monte Carlo method for quantum Boltzmann statistical mechanics using Fourier representations of path integrals , 1984 .

[14]  R. Feynman,et al.  Effective classical partition functions. , 1986, Physical review. A, General physics.

[15]  Mario Lefebvre.  Optimal control of an Ornstein-Uhlenbeck process , 1987 .

[16]  B. Øksendal.  Stochastic differential equations : an introduction with applications , 1987 .

[17]  H. Kleinert.  Path Integrals in Quantum Mechanics Statistics and Polymer Physics , 1990 .

[18]  W. Fleming,et al.  Controlled Markov processes and viscosity solutions , 1992 .

[19]  Robert F. Stengel,et al.  Optimal Control and Estimation , 1994 .

[20]  F. Guerra,et al.  Introduction to Nelson Stochastic Mechanics as a Model for Quantum Mechanics , 1995 .

[21]  Hagen Kleinert,et al.  Path Integrals in Quantum Mechanics , 1995 .

[22]  Peter Whittle,et al.  Optimal Control: Basics and Beyond , 1996 .

[23]  X. Zhou,et al.  Stochastic Controls: Hamiltonian Systems and HJB Equations , 1999 .

[24]  On the approximation of Feynman-Kac path integrals for quantum statistical mechanics , 2000, cond-mat/0007112.

[25]  David Chandler,et al.  Transition path sampling: throwing ropes over rough mountain passes, in the dark. , 2002, Annual review of physical chemistry.

[26]  Jianfeng Feng,et al.  Optimal control of neuronal activity. , 2003, Physical review letters.

[27]  Lawrence F. Shampine,et al.  Solving ODEs with MATLAB , 2002 .

[28]  Luigi M. Ricciardi,et al.  On the parameter estimation for diffusion models of single neuron's activities , 1995, Biological Cybernetics.

[29]  K. Pakdaman,et al.  Random dynamics of the Morris-Lecar neural model. , 2004, Chaos.

[30]  H. Kleinert.  Path Integrals in Quantum Mechanics, Statistics, Polymer Physics, and Financial Markets , 2006 .

[31]  Eric T. Shea-Brown,et al.  Optimal Inputs for Phase Models of Spiking Neurons , 2006 .