Deterministic particle flows for constraining SDEs

Devising optimal interventions for diffusive systems often requires the solution of the Hamilton-Jacobi-Bellman (HJB) equation, a nonlinear backward partial differential equation (PDE), that is, in general, nontrivial to solve. Existing control methods either tackle the HJB directly with grid-based PDE solvers, or resort to iterative stochastic path sampling to obtain the necessary controls. Here, we present a framework that interpolates between these two approaches. By reformulating the optimal interventions in terms of logarithmic gradients ( scores ) of two forward probability flows, and by employing deterministic particle methods for solving Fokker-Planck equations, we introduce a novel fully deterministic framework that computes the required optimal interventions in one shot.

[1]  R Bellman,et al.  DYNAMIC PROGRAMMING AND LAGRANGE MULTIPLIERS. , 1956, Proceedings of the National Academy of Sciences of the United States of America.

[2]  I. V. Girsanov.  On Transforming a Certain Class of Stochastic Processes by Absolutely Continuous Substitution of Measures , 1960 .

[3]  L. Baum,et al.  A Maximization Technique Occurring in the Statistical Analysis of Probabilistic Functions of Markov Chains , 1970 .

[4]  W. Fleming.  Exit probabilities and optimal stochastic control , 1977 .

[5]  M. Brelot.  Classical potential theory and its probabilistic counterpart , 1986 .

[6]  L. Taylor,et al.  Sewall Wright and Evolutionary Biology , 1987 .

[7]  M. Whitlock,et al.  VARIANCE‐INDUCED PEAK SHIFTS , 1995, Evolution; international journal of organic evolution.

[8]  F. Maytag.  Evolution , 1996, Arch. Mus. Informatics.

[9]  S. Shreve,et al.  Stochastic differential equations , 1955, Mathematical Proceedings of the Cambridge Philosophical Society.

[10]  J. M. Hoekstra,et al.  The Strength of Phenotypic Selection in Natural Populations , 2001, The American Naturalist.

[11]  Neri Merhav,et al.  Hidden Markov processes , 2002, IEEE Trans. Inf. Theory.

[12]  Hagai Attias,et al.  Planning by Probabilistic Inference , 2003, AISTATS.

[13]  S. Scott.  Optimal feedback control and the neural basis of volitional motor control , 2004, Nature Reviews Neuroscience.

[14]  H. Kappen.  Linear theory for control of nonlinear stochastic systems. , 2004, Physical review letters.

[15]  H. Kappen.  Path integrals and symmetry breaking for optimal control theory , 2005, physics/0505066.

[16]  Emanuel Todorov,et al.  General duality between optimal control and estimation , 2008, 2008 47th IEEE Conference on Decision and Control.

[17]  Hilbert J. Kappen,et al.  Graphical Model Inference in Optimal Control of Stochastic Multi-Agent Systems , 2008, J. Artif. Intell. Res..

[18]  Emanuel Todorov,et al.  Efficient computation of optimal actions , 2009, Proceedings of the National Academy of Sciences.

[19]  Michael Werman,et al.  Fast and robust Earth Mover's Distances , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[20]  H. Orland.  Generating transition paths by Langevin bridges. , 2011, The Journal of chemical physics.

[21]  Evangelos A. Theodorou,et al.  An iterative path integral stochastic optimal control approach for learning robotic tasks , 2011 .

[22]  Evangelos Theodorou,et al.  Relative entropy and free energy dualities: Connections to Path Integral and KL control , 2012, 2012 IEEE 51st IEEE Conference on Decision and Control (CDC).

[23]  C. Schütte,et al.  Efficient rare event simulation by optimal nonequilibrium forcing , 2012, 1208.3232.

[24]  Vicenç Gómez,et al.  Optimal control as a graphical model inference problem , 2009, Machine Learning.

[25]  Alfio Borzì,et al.  A Fokker-Planck control framework for multidimensional stochastic processes , 2013, J. Comput. Appl. Math..

[26]  Sebastian Reich,et al.  A Nonparametric Ensemble Transform Method for Bayesian Inference , 2012, SIAM J. Sci. Comput..

[27]  Marc Toussaint,et al.  Path Integral Control by Reproducing Kernel Hilbert Space Embedding , 2013, IJCAI.

[28]  Armita Nourmohammad,et al.  Evolution of molecular phenotypes under stabilizing selection , 2013, 1301.3981.

[29]  André Longtin,et al.  Stochastic optimal control of single neuron spike trains. , 2014, Journal of neural engineering.

[30]  Joel W. Burdick,et al.  Linear Hamilton Jacobi Bellman Equations in high dimensions , 2014, 53rd IEEE Conference on Decision and Control.

[31]  Han Wang,et al.  Applications of the Cross-Entropy Method to Importance Sampling and Optimal Control of Diffusions , 2014, SIAM J. Sci. Comput..

[32]  Adilson E Motter,et al.  Control of Stochastic and Induced Switching in Biophysical Networks. , 2015, Physical review. X.

[33]  H. Kappen,et al.  Path integral control and state-dependent feedback. , 2014, Physical review. E, Statistical, nonlinear, and soft matter physics.

[34]  S. Majumdar,et al.  Effective Langevin equations for constrained stochastic processes , 2015, 1503.02639.

[35]  Hilbert J. Kappen,et al.  Adaptive Importance Sampling for Control and Inference , 2015, ArXiv.

[36]  J. Szavits-Nossan,et al.  Inequivalence of nonequilibrium path ensembles: the example of stochastic bridges , 2015, 1508.04969.

[37]  Hugo Touchette,et al.  Variational and optimal control representations of conditioned and driven processes , 2015, 1506.05291.

[38]  Jochen Garcke,et al.  Suboptimal Feedback Control of PDEs by Solving HJB Equations on Adaptive Sparse Grids , 2017, J. Sci. Comput..

[39]  A. Mazzolo.  Constrained Brownian processes and constrained Brownian bridges , 2017 .

[40]  Evangelos Theodorou,et al.  Stochastic optimal control via forward and backward stochastic differential equations and importance sampling , 2018, Autom..

[41]  Sergey Levine,et al.  Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review , 2018, ArXiv.

[42]  M. Timme,et al.  Inferring Network Connectivity from Event Timing Patterns. , 2018, Physical review letters.

[43]  J. W. Kim,et al.  An Optimal Control Derivation of Nonlinear Smoothing Equations , 2019, Advances in Dynamics, Optimization and Computation.

[44]  Armita Nourmohammad,et al.  Optimal evolutionary control for artificial selection on molecular phenotypes , 2019, bioRxiv.

[45]  Maxim Raginsky,et al.  Theoretical guarantees for sampling and inference in generative models with latent diffusions , 2019, COLT.

[46]  David Duvenaud,et al.  Scalable Gradients for Stochastic Differential Equations , 2020, AISTATS.

[47]  Guillaume Hennequin,et al.  Optimal anticipatory control as a theory of motor preparation: A thalamo-cortical circuit model , 2020, Neuron.

[48]  Nicolas Macris,et al.  Solving Non-linear Kolmogorov Equations in Large Dimensions by Using Deep Learning: A Numerical Comparison of Discretization Schemes , 2020, Journal of Scientific Computing.

[49]  M. Opper,et al.  Interacting Particle Solutions of Fokker–Planck Equations Through Gradient–Log–Density Estimation , 2020, Entropy.

[50]  Ziwei Zhong,et al.  Automated continuous evolution of proteins in vivo , 2020, bioRxiv.

[51]  Lorenz Richter,et al.  Solving high-dimensional Hamilton-Jacobi-Bellman PDEs using neural networks: perspectives from the theory of controlled diffusions and measures on path space , 2020, ArXiv.

[52]  Michele Pavon,et al.  The Data‐Driven Schrödinger Bridge , 2021, Communications on Pure and Applied Mathematics.

[53]  Valentin De Bortoli,et al.  Diffusion Schrödinger Bridge with Applications to Score-Based Generative Modeling , 2021, NeurIPS.

[54]  Nikola B. Kovachki,et al.  Fourier Neural Operator for Parametric Partial Differential Equations , 2020, ICLR.

[55]  Abhishek Kumar,et al.  Score-Based Generative Modeling through Stochastic Differential Equations , 2020, ICLR.

[56]  A. Doucet,et al.  Differentiable Particle Filtering via Entropy-Regularized Optimal Transport , 2021, ICML.

[57]  Gefei Wang,et al.  Deep Generative Learning via Schrödinger Bridge , 2021, ICML.

[58]  Arnaud Doucet,et al.  Schrödinger Bridge Samplers , 2021 .

[59]  Michael L. Waskom,et al.  Seaborn: Statistical Data Visualization , 2021, J. Open Source Softw..

[60]  Pierre Thodoroff,et al.  Solving Schrödinger Bridges via Maximum Likelihood , 2021, Entropy.

[61]  Kenneth F. Caluya,et al.  Wasserstein Proximal Algorithms for the Schrödinger Bridge Problem: Density Control With Nonlinear Drift , 2019, IEEE Transactions on Automatic Control.