PPINN: Parareal Physics-Informed Neural Network for time-dependent PDEs

Physics-informed neural networks (PINNs) encode physical conservation laws and prior physical knowledge into the neural networks, ensuring the correct physics is represented accurately while alleviating the need for supervised learning to a great degree. While effective for relatively short-term time integration, when long time integration of the time-dependent PDEs is sought, the time-space domain may become arbitrarily large and hence training of the neural network may become prohibitively expensive. To this end, we develop a parareal physics-informed neural network (PPINN), hence decomposing a long-time problem into many independent short-time problems supervised by an inexpensive/fast coarse-grained (CG) solver. In particular, the serial CG solver is designed to provide approximate predictions of the solution at discrete times, while initiate many fine PINNs simultaneously to correct the solution iteratively. There is a two-fold benefit from training PINNs with small-data sets rather than working on a large-data set directly, i.e., training of individual PINNs with small-data is much faster, while training the fine PINNs can be readily parallelized. Consequently, compared to the original PINN approach, the proposed PPINN approach may achieve a significant speedup for long-time integration of PDEs, assuming that the CG solver is fast and can provide reasonable predictions of the solution, hence aiding the PPINN solution to converge in just a few iterations. To investigate the PPINN performance on solving time-dependent PDEs, we first apply the PPINN to solve the Burgers equation, and subsequently we apply the PPINN to solve a two-dimensional nonlinear diffusion-reaction equation. Our results demonstrate that PPINNs converge in a couple of iterations with significant speed-ups proportional to the number of time-subdomains employed.

[1]  Yvon Maday,et al.  A Parareal in Time Semi-implicit Approximation of the Navier-Stokes Equations , 2005 .

[2]  Jason Sanders,et al.  CUDA by example: an introduction to general purpose GPU programming , 2010 .

[3]  Todd A. Oliver,et al.  Solving Irregular and Data-enriched Differential Equations using Deep Neural Networks , 2019, ArXiv.

[4]  Zhiping Mao,et al.  DeepXDE: A Deep Learning Library for Solving Differential Equations , 2019, AAAI Spring Symposium: MLPS.

[5]  Y. Maday,et al.  A parareal in time procedure for the control of partial differential equations , 2002 .

[6]  Giovanni Samaey,et al.  A Micro-Macro Parareal Algorithm: Application to Singularly Perturbed Ordinary Differential Equations , 2012, SIAM J. Sci. Comput..

[7]  Liu Yang,et al.  Physics-Informed Generative Adversarial Networks for Stochastic Differential Equations , 2018, SIAM J. Sci. Comput..

[8]  M. Meerschaert,et al.  Finite difference methods for two-dimensional fractional dispersion equation , 2006 .

[9]  N. Phan-Thien,et al.  Neural-network-based approximations for solving partial differential equations , 1994 .

[10]  Yvon Maday.  Parareal in time algorithm for kinetic systems based on model reduction , 2007 .

[11]  Gavriel Salomon,et al.  T RANSFER OF LEARNING , 1992 .

[12]  Roi Livni,et al.  On the Computational Efficiency of Training Neural Networks , 2014, NIPS.

[13]  Ilias Bilionis,et al.  Deep UQ: Learning deep neural network surrogate models for high dimensional uncertainty quantification , 2018, J. Comput. Phys..

[14]  M MeerschaertMark,et al.  Finite difference methods for two-dimensional fractional dispersion equation , 2006 .

[15]  B. Ong,et al.  Applications of time parallelization , 2020, Computing and Visualization in Science.

[16]  Tor M. Aamodt,et al.  Analyzing Machine Learning Workloads Using a Detailed GPU Simulator , 2018, 2019 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS).

[17]  Yvon Maday,et al.  The Parareal in Time Iterative Solver: a Further Direction to Parallel Implementation , 2005 .

[18]  Mengying Shu,et al.  Deep learning for image classification on very small datasets using transfer learning , 2019 .

[19]  G. Beylkin,et al.  A Multiresolution Strategy for Reduction of Elliptic PDEs and Eigenvalue Problems , 1998 .

[20]  Andrew J. Christlieb,et al.  Implicit Parallel Time Integrators , 2011, J. Sci. Comput..

[21]  Paris Perdikaris,et al.  Adversarial Uncertainty Quantification in Physics-Informed Neural Networks , 2018, J. Comput. Phys..

[22]  INEXACT AND TRUNCATED PARAREAL-INTIME KRYLOV SUBSPACE METHODS FOR PARABOLIC OPTIMAL CONTROL PROBLEMS , 2012 .

[23]  George Em Karniadakis,et al.  Learning in Modal Space: Solving Time-Dependent Stochastic PDEs Using Physics-Informed Neural Networks , 2019, SIAM J. Sci. Comput..

[24]  George Em Karniadakis,et al.  Quantifying total uncertainty in physics-informed neural networks for solving forward and inverse stochastic problems , 2018, J. Comput. Phys..

[25]  Wolfgang Polifke,et al.  Physics-Informed Echo State Networks for Chaotic Systems Forecasting , 2019, ICCS.

[26]  Paris Perdikaris,et al.  Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations , 2019, J. Comput. Phys..

[27]  George Em Karniadakis,et al.  Supervised parallel-in-time algorithm for long-time Lagrangian simulations of stochastic dynamics: Application to hydrodynamics , 2019, J. Comput. Phys..

[28]  Yoram Singer,et al.  Adaptive Subgradient Methods for Online Learning and Stochastic Optimization , 2011, J. Mach. Learn. Res..

[29]  Jorge Nocedal,et al.  Optimization Methods for Large-Scale Machine Learning , 2016, SIAM Rev..

[30]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[31]  James Demmel,et al.  Fast Deep Neural Network Training on Distributed Systems and Cloud TPUs , 2019, IEEE Transactions on Parallel and Distributed Systems.

[32]  Akshunna S. Dogra,et al.  Hamiltonian Neural Networks for solving differential equations , 2020, ArXiv.

[33]  George Em Karniadakis,et al.  fPINNs: Fractional Physics-Informed Neural Networks , 2018, SIAM J. Sci. Comput..

[34]  Marios Mattheakis,et al.  Physical Symmetries Embedded in Neural Networks , 2019, ArXiv.

[35]  George Em Karniadakis,et al.  A composite neural network that learns from multi-fidelity data: Application to function approximation and inverse PDE problems , 2019, J. Comput. Phys..

[36]  Upmc-Cnrs,et al.  Parareal in time 3D numerical solver for the LWR Benchmark neutron diffusion transient model , 2014 .

[37]  Martin J. Gander,et al.  50 Years of Time Parallel Time Integration , 2015 .

[38]  Panos Stinis,et al.  Doing the Impossible: Why Neural Networks Can Be Trained at All , 2018, Front. Psychol..

[39]  Prabhat,et al.  Exascale Deep Learning for Climate Analytics , 2018, SC18: International Conference for High Performance Computing, Networking, Storage and Analysis.

[40]  Terry Haut,et al.  An Asymptotic Parallel-in-Time Method for Highly Oscillatory PDEs , 2013, SIAM J. Sci. Comput..

[41]  Jie Cheng,et al.  CUDA by Example: An Introduction to General-Purpose GPU Programming , 2010, Scalable Comput. Pract. Exp..

[42]  Y. Maday,et al.  A “Parareal” Time Discretization for Non-Linear PDE’s with Application to the Pricing of an American Put , 2002 .

[43]  Zhaoli Guo,et al.  Localized lattice Boltzmann equation model for simulating miscible viscous displacement in porous media , 2016 .

[44]  Panos Stinis,et al.  Enforcing Constraints for Time Series Prediction in Supervised, Unsupervised and Reinforcement Learning , 2019, AAAI Spring Symposium: MLPS.

[45]  Karthik Ramani,et al.  ConvPDE-UQ: Convolutional neural networks with quantified uncertainty for heterogeneous elliptic partial differential equations on varied domains , 2019, J. Comput. Phys..

[46]  Andrew J. Christlieb,et al.  A Parallel Space-Time Algorithm , 2012, SIAM J. Sci. Comput..

[47]  Charbel Farhat,et al.  Time‐decomposed parallel time‐integrators: theory and feasibility studies for fluid, structure, and fluid–structure applications , 2003 .