DeepParticle: learning invariant measure by a deep neural network minimizing Wasserstein distance on data generated from an interacting particle method

We introduce DeepParticle, a method to learn and generate invariant measures of stochastic dynamical systems with physical parameters based on data computed from an interacting particle method (IPM). We utilize the expressiveness of deep neural networks (DNNs) to represent the transform of samples from a given input (source) distribution to an arbitrary target distribution, neither assuming distribution functions in closed form nor the sample transforms to be invertible, nor the sample state space to be finite. In the training stage, we update the weights of the network by minimizing a discrete Wasserstein distance between the input and target samples. To reduce the computational cost, we propose an iterative divide-and-conquer (a mini-batch interior point) algorithm, to find the optimal transition matrix in the Wasserstein distance. We present numerical results to demonstrate the performance of our method for accelerating the computation of invariant measures of stochastic dynamical systems arising in computing reaction-diffusion front speeds in 3D chaotic flows by using the IPM. The physical parameter is a large Péclet number reflecting the advection-dominated regime of our interest. AMS subject classification: 35K57, 37M25, 49Q22, 65C35, 68T07.

[1]  Dimitrios I. Fotiadis,et al.  Artificial neural networks for solving ordinary and partial differential equations , 1997, IEEE Trans. Neural Networks.

[2]  Jinchao Xu,et al.  Relu Deep Neural Networks and Linear Finite Elements , 2018, Journal of Computational Mathematics.

[3]  Jinchao Xu,et al.  MgNet: A unified framework of multigrid and convolutional neural network , 2019, Science China Mathematics.

[4]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[5]  Xinyu Liu,et al.  Deep least-squares methods: an unsupervised learning-based numerical method for solving elliptic PDEs , 2020, J. Comput. Phys..

[6]  Lukasz Kaiser,et al.  Attention is All you Need , 2017, NIPS.

[7]  Jack Xin,et al.  Global Convergence and Geometric Characterization of Slow to Fast Weight Evolution in Neural Network Training for Classifying Linearly Non-Separable Data , 2020, ArXiv.

[8]  Richard Sinkhorn.  A Relationship Between Arbitrary Positive Matrices and Doubly Stochastic Matrices , 1964 .

[9]  Jack Xin,et al.  An Introduction to Fronts in Random Media , 2009 .

[10]  Steven L. Brunton,et al.  Deep learning of dynamics and signal-noise decomposition with time-stepping constraints , 2018, J. Comput. Phys..

[11]  Jitesh H. Panchal,et al.  Simulator-free solution of high-dimensional stochastic elliptic partial differential equations using deep neural networks , 2019, J. Comput. Phys..

[12]  Edward Ott,et al.  Stretch, Twist, Fold: The Fast Dynamo , 1995 .

[13]  J. Nathan Kutz,et al.  Deep learning in fluid dynamics , 2017, Journal of Fluid Mechanics.

[14]  Justin A. Sirignano,et al.  DGM: A deep learning algorithm for solving partial differential equations , 2017, J. Comput. Phys..

[15]  J. Xin,et al.  Finite Element Computation of KPP Front Speeds in 3D Cellular and ABC Flows , 2013 .

[16]  Kailiang Wu,et al.  Data Driven Governing Equations Approximation Using Deep Neural Networks , 2018, J. Comput. Phys..

[17]  Jiancheng Lyu,et al.  Computing Residual Diffusivity by Adaptive Basis Learning via Spectral Method , 2017 .

[18]  Bin Dong,et al.  PDE-Net 2.0: Learning PDEs from Data with A Numeric-Symbolic Hybrid Deep Network , 2018, J. Comput. Phys..

[19]  Stanley Osher,et al.  Computations of Optimal Transport Distance with Fisher Information Regularization , 2017, J. Sci. Comput..

[20]  S. W. Ellacott,et al.  Aspects of the numerical analysis of neural networks , 1994, Acta Numerica.

[21]  E Weinan,et al.  The Deep Ritz Method: A Deep Learning-Based Numerical Algorithm for Solving Variational Problems , 2017, Communications in Mathematics and Statistics.

[22]  Allan Pinkus,et al.  Approximation theory of the MLP model in neural networks , 1999, Acta Numerica.

[23]  Jack Xin,et al.  A convergent interacting particle method and computation of KPP front speeds in chaotic flows , 2021, SIAM J. Numer. Anal..

[24]  Paris Perdikaris,et al.  Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations , 2019, J. Comput. Phys..

[25]  Mathieu Lauriere,et al.  Convergence Analysis of Machine Learning Algorithms for the Numerical Solution of Mean Field Control and Games I: The Ergodic Case , 2019, SIAM J. Numer. Anal..

[26]  Andrew J. Meade,et al.  The numerical solution of linear ordinary differential equations by feedforward neural networks , 1994 .

[27]  Ivan Kobyzev,et al.  Normalizing Flows: An Introduction and Review of Current Methods , 2020, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[28]  Gang Bao,et al.  Weak Adversarial Networks for High-dimensional Partial Differential Equations , 2019, J. Comput. Phys..

[29]  Nicholas Zabaras,et al.  Bayesian Deep Convolutional Encoder-Decoder Networks for Surrogate Modeling and Uncertainty Quantification , 2018, J. Comput. Phys..

[30]  Kurt Hornik,et al.  Multilayer feedforward networks are universal approximators , 1989, Neural Networks.

[31]  Jack Xin,et al.  Computing Residual Diffusivity by Adaptive Basis Learning via Super-Resolution Deep Neural Networks , 2019, ICCSAMA.

[32]  Lexing Ying,et al.  Solving parametric PDE problems with artificial neural networks , 2017, European Journal of Applied Mathematics.

[33]  Aaas News,et al.  Book Reviews , 1893, Buffalo Medical and Surgical Journal.

[34]  Zuowei Shen,et al.  Deep Network with Approximation Error Being Reciprocal of Width to Power of Square Root of Depth , 2020, Neural Computation.

[35]  Dmitry Yarotsky,et al.  Error bounds for approximations with deep ReLU networks , 2016, Neural Networks.

[36]  Gabriel Peyré,et al.  Computational Optimal Transport , 2018, Found. Trends Mach. Learn..

[37]  Stephen J. Wright.  Primal-Dual Interior-Point Methods , 1997, Other Titles in Applied Mathematics.

[38]  Zhongjian Wang,et al.  A mesh-free method for interface problems using the deep learning approach , 2019, J. Comput. Phys..

[39]  Jack Xin,et al.  Existence of KPP fronts in spatially-temporally periodic advection and variational principle for propagation speeds , 2005 .

[40]  Liu Yang,et al.  B-PINNs: Bayesian Physics-Informed Neural Networks for Forward and Inverse PDE Problems with Noisy Data , 2020, J. Comput. Phys..

[41]  Arnulf Jentzen,et al.  Solving high-dimensional partial differential equations using deep learning , 2017, Proceedings of the National Academy of Sciences.

[42]  Kamyar Azizzadenesheli,et al.  Fourier Neural Operator for Parametric Partial Differential Equations , 2021, ICLR.

[43]  George Em Karniadakis,et al.  DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators , 2019, ArXiv.

[44]  Yalchin Efendiev,et al.  Deep Multiscale Model Learning , 2018, J. Comput. Phys..

[45]  Jan S. Hesthaven,et al.  Physics-informed machine learning for reduced-order modeling of nonlinear problems , 2021, J. Comput. Phys..

[46]  Bin Dong,et al.  PDE-Net: Learning PDEs from Data , 2017, ICML.

[47]  Paris Perdikaris,et al.  Physics-Constrained Deep Learning for High-dimensional Surrogate Modeling and Uncertainty Quantification without Labeled Data , 2019, J. Comput. Phys..

[48]  Levon Nurbekyan,et al.  A machine learning framework for solving high-dimensional mean field game and mean field control problems , 2020, Proceedings of the National Academy of Sciences.

[49]  Mathieu Laurière,et al.  Convergence Analysis of Machine Learning Algorithms for the Numerical Solution of Mean Field Control and Games: I - The Ergodic Case , 2019, The Annals of Applied Probability.

[50]  C. Villani.  Topics in Optimal Transportation , 2003 .

[51]  M. Freidlin.  Functional Integration And Partial Differential Equations , 1985 .

[52]  Zuowei Shen,et al.  Deep Network Approximation for Smooth Functions , 2020, ArXiv.

[53]  Alternating the population and control neural networks to solve high-dimensional stochastic mean-field games , 2021, Proceedings of the National Academy of Sciences.

[54]  Alexander Schrijver,et al.  Combinatorial optimization. Polyhedra and efficiency. , 2003 .

[55]  Hyuk Lee,et al.  Neural algorithm for solving differential equations , 1990 .

[56]  Qian Wang,et al.  Recurrent neural network closure of parametric POD-Galerkin reduced-order models based on the Mori-Zwanzig formalism , 2020, J. Comput. Phys..

[57]  Marco Cuturi,et al.  Sinkhorn Distances: Lightspeed Computation of Optimal Transport , 2013, NIPS.

[58]  Nadav Cohen,et al.  On the Expressive Power of Deep Learning: A Tensor Analysis , 2015, COLT 2016.

[59]  Qiang Du,et al.  New Error Bounds for Deep ReLU Networks Using Sparse Grids , 2017, SIAM J. Math. Data Sci..

[60]  J. Xin,et al.  A Semi-Lagrangian Computation of Front Speeds of G-equation in ABC and Kolmogorov Flows with Estimation via Ballistic Orbits , 2020, Multiscale Model. Simul..

[61]  P. Alam.  ‘S’ , 2021, Composites Engineering: An A–Z Guide.

[62]  Zhiqiang Cai,et al.  Least-Squares ReLU Neural Network (LSNN) Method For Linear Advection-Reaction Equation , 2021, J. Comput. Phys..

[63]  Kailiang Wu,et al.  Data-Driven Deep Learning of Partial Differential Equations in Modal Space , 2020, J. Comput. Phys..

[64]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[65]  S. Osher,et al.  Connections between deep learning and partial differential equations , 2021, European Journal of Applied Mathematics.

[66]  Jack Xin,et al.  Finite Element Computation of KPP Front Speeds in Cellular and Cat’s Eye Flows , 2013, J. Sci. Comput..

[67]  STRETCH , 2019, Proceedings of the 13th ACM International Conference on Distributed and Event-based Systems.

[68]  G. Karniadakis,et al.  Multistep Neural Networks for Data-driven Discovery of Nonlinear Dynamical Systems , 2018, 1801.01236.

[69]  Jack Xin,et al.  A computational study of residual KPP front speeds in time-periodic cellular flows in the small diffusion limit , 2014, 1406.2268.

[70]  George Cybenko,et al.  Approximation by superpositions of a sigmoidal function , 1989, Math. Control. Signals Syst..

[71]  M. Proctor,et al.  Numerical calculations of fast dynamos in smooth velocity fields with realistic diffusion , 1992, Nature.