Variational and optimal control representations of conditioned and driven processes

We have shown recently that a Markov process conditioned on rare events involving time-integrated random variables can be described in the long-time limit by an effective Markov process, called the driven process, which is given mathematically by a generalization of Doob's $h$-transform. We show here that this driven process can be represented in two other ways: first, as a process satisfying various variational principles involving large deviation functions and relative entropies and, second, as an optimal stochastic control process minimizing a cost function also related to large deviation functions. These interpretations of the driven process generalize and unify many previous results on maximum entropy approaches to nonequilibrium systems, spectral characterizations of positive operators, and control approaches to large deviation theory. They also lead, as briefly discussed, to new methods for analytically or numerically approximating large deviation functions.

[1]  Peter Whittle Risk-sensitivity, large deviations and stochastic control , 1994 .

[2]  C. Maes,et al.  Steady state statistics of driven diffusions , 2007, 0708.0489.

[3]  L. Bertini,et al.  From level 2.5 to level 2 large deviations for continuous time Markov chains , 2012, 1212.6908.

[4]  Peter V. E. McClintock,et al.  Analogue studies of nonlinear systems , 1998 .

[5]  R. Evans Statistical physics of shear flow: a non-equilibrium problem , 2010 .

[6]  R. Graham Noise in nonlinear dynamical systems: Macroscopic potentials, bifurcations and noise in dissipative systems , 1989 .

[7]  Kazuyuki Aihara,et al.  Pathwise thermodynamic structure in population dynamics. , 2015, Physical review. E, Statistical, nonlinear, and soft matter physics.

[8]  A. C. Barato,et al.  A Formal View on Level 2.5 Large Deviations and Fluctuation Relations , 2014, 1408.5033.

[9]  C. Holland A minimum principle for the principal eigenvalue for second-order linear elliptic equations with natural boundary conditions , 1978 .

[10]  H. Kappen Path integrals and symmetry breaking for optimal control theory , 2005, physics/0505066.

[11]  Stochastic interacting particle systems out of equilibrium , 2007, 0705.1247.

[12]  M. Brelot Classical potential theory and its probabilistic counterpart , 1986 .

[13]  B. Derrida,et al.  of Statistical Mechanics : Theory and Experiment Non-equilibrium steady states : fluctuations and large deviations of the density and of the current , 2007 .

[14]  J. Elgin The Fokker-Planck Equation: Methods of Solution and Applications , 1984 .

[15]  Michael Schulz,et al.  Control Theory in Physics and Other Fields of Science: Concepts, Tools, and Applications , 2005 .

[16]  Hugo Touchette,et al.  Nonequilibrium Markov Processes Conditioned on Large Deviations , 2014, 1405.5157.

[17]  Rosemary J. Harris,et al.  Large Deviation Approach to Nonequilibrium Systems , 2011, 1110.5216.

[18]  H. Kappen,et al.  Linear PDEs and eigenvalue problems corresponding to ergodic stochastic optimization problems on compact manifolds , 2013, 1303.0126.

[19]  H. Kappen Linear theory for control of nonlinear stochastic systems. , 2004, Physical review letters.

[20]  Vivien Lecomte,et al.  Simulating Rare Events in Dynamical Processes , 2011, 1106.4929.

[21]  W. Fleming A stochastic control approach to some large deviations problems , 1985 .

[22]  Yoshitsugu Oono,et al.  Large Deviation and Statistical Physics , 1989 .

[23]  E. Jaynes The Minimum Entropy Production Principle , 1980 .

[24]  V. Lecomte,et al.  Thermodynamic Formalism for Systems with Markov Dynamics , 2007 .

[25]  Charles J. Holland,et al.  A new energy characterization of the smallest eigenvalue of the schrödinger equation , 1977 .

[26]  K. Vahala Handbook of stochastic methods for physics, chemistry and the natural sciences , 1986, IEEE Journal of Quantum Electronics.

[27]  R. Evans,et al.  Comment on `Detailed balance has a counterpart in non-equilibrium steady states' , 2004, 0901.4879.

[28]  C. Schütte,et al.  Efficient rare event simulation by optimal nonequilibrium forcing , 2012, 1208.3232.

[29]  Chaotic properties of systems with Markov dynamics. , 2005, Physical review letters.

[30]  Gregory L. Eyink,et al.  Rayleigh-Ritz Calculation of Effective Potential Far from Equilibrium , 1997 .

[31]  H. Kappen Optimal control theory and the linear bellman equation , 2011 .

[32]  W. Fleming,et al.  Risk sensitive optimal control and differential games , 1992 .

[33]  R M L Evans Rules for transition rates in nonequilibrium steady states. , 2004, Physical review letters.

[34]  Carsten Hartmann,et al.  Characterization of Rare Events in Molecular Dynamics , 2013, Entropy.

[35]  Large Deviation Principles and Complete Equivalence and Nonequivalence Results for Pure and Mixed Ensembles , 2000, math/0012081.

[36]  V. I. Mel'Nikov,et al.  The Kramers problem : fifty years of development , 1991 .

[37]  W. Fleming Stochastic Control for Small Noise Intensities , 1971 .

[38]  P. Whittle A risk-sensitive maximum principle: the case of imperfect state observation , 1991 .

[39]  K. Dill,et al.  Principles of maximum entropy and maximum caliber in statistical physics , 2013 .

[40]  Wendell H. Fleming Stochastic Control and Large Deviations , 1992, 25th Anniversary of INRIA.

[41]  Robert L. Jack,et al.  Effective interactions and large deviations in stochastic processes , 2015, The European Physical Journal Special Topics.

[42]  Peter Sollich,et al.  Large deviations and ensembles of trajectories in stochastic models , 2009, 0911.0211.

[43]  Lars Onsager,et al.  Fluctuations and Irreversible Processes , 1953 .

[44]  Variational formula for experimental determination of high-order correlations of current fluctuations in driven systems. , 2010, Physical review. E, Statistical, nonlinear, and soft matter physics.

[45]  Richard Bellman,et al.  Introduction to the mathematical theory of control processes , 1967 .

[46]  W. Fleming,et al.  Asymptotics for the principal eigenvalue and eigenfunction of a nearly first-order operator with large potential , 1997 .

[47]  Hugo Touchette,et al.  Equivalence and Nonequivalence of Ensembles: Thermodynamic, Macrostate, and Measure Levels , 2014, 1403.6608.

[48]  W. Fleming,et al.  A remark on the large deviations of an ergodic markov process , 1987 .

[49]  Pierre Collet,et al.  Quasi-stationary distributions , 2011 .

[50]  S. Sheu Some Estimates of the Transition Density of a Nondegenerate Diffusion Markov Process , 1991 .

[51]  Hugo Touchette,et al.  Nonequilibrium microcanonical and canonical ensembles and their equivalence. , 2013, Physical review letters.

[52]  Vivek S. Borkar,et al.  Peformance Analysis Conditioned on Rare Events: An Adaptive Simulation Scheme , 2003, Commun. Inf. Syst..

[53]  Wolfgang J. Runggaldier,et al.  Connections between stochastic control and dynamic games , 1996, Math. Control. Signals Syst..

[54]  W. Fleming Stochastic calculus of variations and mechanics , 1983 .

[55]  P. Hänggi,et al.  Reaction-rate theory: fifty years after Kramers , 1990 .

[56]  S. Varadhan,et al.  Asymptotic evaluation of certain Markov process expectations for large time , 1975 .

[57]  T. Nemoto,et al.  Thermodynamic formula for the cumulant generating function of time-averaged current. , 2011, Physical review. E, Statistical, nonlinear, and soft matter physics.

[58]  C. Gardiner Handbook of Stochastic Methods , 1983 .

[59]  S. R. S. Varadhan,et al.  Large deviations for Markov processes and the asymptotic evaluation of certain Markov process expectations for large times , 1975 .

[60]  R. Graham Macroscopic potentials, bifurcations and noise in dissipative systems , 1987 .

[61]  E. Seneta,et al.  On quasi-stationary distributions in absorbing continuous-time finite Markov chains , 1967, Journal of Applied Probability.

[62]  W. Fleming Exit probabilities and optimal stochastic control , 1977 .

[63]  S. Varadhan,et al.  On the principal eigenvalue of second‐order elliptic differential operators , 1976 .

[64]  D. Simon Construction of a coordinate Bethe ansatz for the asymmetric simple exclusion process with open boundaries , 2009, 0903.4968.

[65]  Physics of Large Deviation , 2012, 1204.5584.

[66]  Robert F. Stengel,et al.  Optimal Control and Estimation , 1994 .

[67]  Kingshuk Ghosh,et al.  Maximum Caliber: a variational approach applied to two-state dynamics. , 2008, The Journal of chemical physics.

[68]  C. Maes,et al.  Canonical structure of dynamical fluctuations in mesoscopic nonequilibrium steady states , 2007, 0705.2344.

[69]  W. Fleming Logarithmic Transformations and Stochastic Control , 1982 .

[70]  James A. Bucklew,et al.  Introduction to Rare Event Simulation , 2010 .

[71]  J. P. Garrahan,et al.  First-order dynamical phase transition in models of glasses: an approach based on ensembles of histories , 2008, 0810.5298.

[72]  Hugo Touchette,et al.  A basic introduction to large deviations: Theory, applications, simulations , 2011, 1106.4146.

[73]  H. Touchette The large deviation approach to statistical mechanics , 2008, 0804.0327.

[74]  J. Lynch,et al.  A weak convergence approach to the theory of large deviations , 1997 .

[75]  A. A. Filyukov,et al.  Description of steady transport processes by the method of the most probable path of evolution , 1967 .

[76]  M. Kac On Some Connections between Probability Theory and Differential and Integral Equations , 1951 .

[77]  M. Chertkov,et al.  Non-Equilibrium Thermodynamics and Topology of Currents , 2009, 0907.3481.

[78]  J.S. Sadowsky,et al.  On large deviations theory and asymptotically efficient Monte Carlo estimation , 1990, IEEE Trans. Inf. Theory.

[79]  Paolo Baldi,et al.  A representation formula for the large deviation rate function for the empirical law of a continuous time Markov chain , 1999 .

[80]  Michael Chertkov,et al.  Stochastic Optimal Control as Non-equilibrium Statistical Mechanics: Calculus of Variations over Density and Current , 2013, ArXiv.

[81]  A. Si,et al.  Entropy,Large Deviations,and Statistical Mechanics , 2011 .

[82]  W. Fleming,et al.  Deterministic and Stochastic Optimal Control , 1975 .

[83]  H. Risken Fokker-Planck Equation , 1996 .

[84]  W. Fleming,et al.  Controlled Markov processes and viscosity solutions , 1992 .

[85]  Variational Principle in Langevin Processes , 2013, 1307.6111.

[86]  Cecile Monthus,et al.  Non-equilibrium steady states: maximization of the Shannon entropy associated with the distribution of dynamical trajectories in the presence of constraints , 2010, 1011.1342.

[87]  K. Symanzik,et al.  Small distance behaviour in field theory and power counting , 1970 .

[88]  Robert L. Jack,et al.  Large deviations of the dynamical activity in the East model: analysing structure in biased trajectories , 2013, 1308.6124.

[89]  Vivek S. Borkar,et al.  A Learning Algorithm for Risk-Sensitive Cost , 2008, Math. Oper. Res..

[90]  Hilbert J. Kappen,et al.  Explicit solution of relative entropy weighted control , 2012, Syst. Control. Lett..

[91]  Action Principle in Statistical Dynamics , 1998 .

[92]  W. Fleming,et al.  Risk-Sensitive Control on an Infinite Time Horizon , 1995 .

[93]  S. R. S. Varadhan,et al.  Chapter Nine. Large Deviations and Entropy , 2003 .

[94]  Vivien Lecomte,et al.  A numerical approach to large deviations in continuous time , 2007 .

[95]  W. Fleming Logarithmic transformations with applications in probability and stochastic control , 1989 .

[96]  Wendell H. Fleming,et al.  Stochastic variational formula for fundamental solutions of parabolic PDE , 1985 .

[97]  Peter W. Glynn,et al.  Stochastic Simulation: Algorithms and Analysis , 2007 .

[98]  A. A. Filyukov The information approach to the theory of irreversible quasi-equilibrium processes , 1968 .

[99]  J. L. Doob,et al.  Conditional brownian motion and the boundary limits of harmonic functions , 1957 .

[100]  P. Whittle A risk-sensitive maximum principle , 1990 .

[101]  Takahiro Nemoto,et al.  Computation of large deviation statistics via iterative measurement-and-feedback procedure. , 2013, Physical review letters.

[102]  Gunter M. Schütz,et al.  Dynamics of Instantaneous Condensation in the ZRP Conditioned on an Atypical Current , 2013, Entropy.

[103]  Han Wang,et al.  Applications of the Cross-Entropy Method to Importance Sampling and Optimal Control of Diffusions , 2014, SIAM J. Sci. Comput..

[104]  G. Eyink,et al.  Action principle in nonequilibrium statistical dynamics. , 1996, Physical review. E, Statistical physics, plasmas, fluids, and related interdisciplinary topics.

[105]  A. Faggionato,et al.  Large deviations of the empirical flow for continuous time Markov chains , 2012, 1210.2004.

[106]  A. A. Filyukov,et al.  Method of the most probable path of evolution in the theory of stationary irreversible processes , 1967 .

[107]  E. Seneta,et al.  On Quasi-Stationary distributions in absorbing discrete-time finite Markov chains , 1965, Journal of Applied Probability.

[108]  V. Popkov,et al.  Transition Probabilities and Dynamic Structure Function in the ASEP Conditioned on Strong Flux , 2010, 1011.3913.

[109]  V. Popkov,et al.  ASEP on a ring conditioned on enhanced flux , 2010 .

[110]  Simulation of large deviation functions using population dynamics , 2008, 0811.1041.

[111]  S. Sheu Stochastic control and principal eigenvaluet , 1984 .

[112]  S. Varadhan,et al.  On a variational formula for the principal eigenvalue for operators with maximum principle. , 1975, Proceedings of the National Academy of Sciences of the United States of America.