Estimates on the generalization error of physics-informed neural networks for approximating a class of inverse problems for PDEs

Physics informed neural networks (PINNs) have recently been very successfully applied for efficiently approximating inverse problems for PDEs. We focus on a particular class of inverse problems, the so-called data assimilation or unique continuation problems, and prove rigorous estimates on the generalization error of PINNs approximating them. An abstract framework is presented and conditional stability estimates for the underlying inverse problem are employed to derive the estimate on the PINN generalization error, providing rigorous justification for the use of PINNs in this context. The abstract framework is illustrated with examples of four prototypical linear PDEs. Numerical experiments, validating the proposed theory, are also presented.

[1]  Fabian Laakmann,et al.  Efficient approximation of solutions of parametric linear transport equations by ReLU DNNs , 2020, Advances in Computational Mathematics.

[2]  Geoffrey E. Hinton,et al.  Deep Learning , 2015, Nature.

[3]  Arnold Neumaier,et al.  Introduction to Numerical Analysis , 2001 .

[4]  R. Molinaro,et al.  A multi-level procedure for enhancing accuracy of machine learning algorithms , 2019, European Journal of Applied Mathematics.

[5]  Erik Burman,et al.  A finite element data assimilation method for the wave equation , 2018, Math. Comput..

[6]  Deep Ray,et al.  Deep learning observables in computational fluid dynamics , 2019, J. Comput. Phys..

[7]  George Em Karniadakis,et al.  fPINNs: Fractional Physics-Informed Neural Networks , 2018, SIAM J. Sci. Comput..

[8]  Siddhartha Mishra,et al.  A machine learning framework for data driven acceleration of computations of differential equations , 2018, ArXiv.

[9]  Erik Burman,et al.  Fully discrete finite element data assimilation method for the heat equation , 2018, ESAIM: Mathematical Modelling and Numerical Analysis.

[10]  Peter Hansbo,et al.  Stabilized nonconforming finite element methods for data assimilation in incompressible flows , 2018, Math. Comput..

[11]  H. Bungartz,et al.  Sparse grids , 2004, Acta Numerica.

[12]  Sebastian Becker,et al.  Solving stochastic differential equations and Kolmogorov equations by means of deep learning , 2018, ArXiv.

[13]  George Em Karniadakis,et al.  On the Convergence and generalization of Physics Informed Neural Networks , 2020, ArXiv.

[14]  Dmitry Yarotsky,et al.  Error bounds for approximations with deep ReLU networks , 2016, Neural Networks.

[15]  Deep Ray,et al.  Iterative Surrogate Model Optimization (ISMO): An active learning algorithm for PDE constrained optimization with deep neural networks , 2020, ArXiv.

[16]  Erik Burman,et al.  Error estimates for stabilized finite element methods applied to ill-posed problems , 2014, 1406.4371.

[17]  L. Sifre,et al.  A 7 D De novo structure prediction with deep learning based scoring , 2022 .

[18]  Jenn-Nan Wang,et al.  Optimal three-ball inequalities and quantitative uniqueness for the Stokes system , 2008, 0812.3730.

[19]  Shai Ben-David,et al.  Understanding Machine Learning: From Theory to Algorithms , 2014 .

[20]  J. Stoer,et al.  Introduction to Numerical Analysis , 2002 .

[21]  R. Caflisch.  Monte Carlo and quasi-Monte Carlo methods , 1998, Acta Numerica.

[22]  Ameya D. Jagtap,et al.  Extended Physics-informed Neural Networks (XPINNs): A Generalized Space-Time Domain Decomposition based Deep Learning Framework for Nonlinear Partial Differential Equations , 2020, AAAI Spring Symposium: MLPS.

[23]  Peter Hansbo,et al.  Solving ill-posed control problems by stabilized finite element methods: an alternative to Tikhonov regularization , 2016, 1609.05101.

[24]  Marius Tucsnak,et al.  Recovering the initial state of an infinite-dimensional system using observers , 2010, Autom..

[25]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[26]  Oleg Yu. Imanuvilov,et al.  Controllability of Evolution equations , 1996 .

[27]  E Weinan,et al.  Deep Learning-Based Numerical Methods for High-Dimensional Parabolic Partial Differential Equations and Backward Stochastic Differential Equations , 2017, Communications in Mathematics and Statistics.

[28]  George Em Karniadakis,et al.  Physics-Informed Neural Network for Ultrasound Nondestructive Quantification of Surface Breaking Cracks , 2020, Journal of Nondestructive Evaluation.

[29]  Dimitris G. Papageorgiou,et al.  Neural-network methods for boundary value problems with irregular boundaries , 2000, IEEE Trans. Neural Networks Learn. Syst..

[30]  L. Dal Negro,et al.  Physics-informed neural networks for inverse problems in nano-optics and metamaterials. , 2019, Optics express.

[31]  E. Burman,et al.  Weakly Consistent Regularisation Methods for Ill-Posed Problems , 2018 .

[32]  G. Karniadakis,et al.  Physics-informed neural networks for high-speed flows , 2020, Computer Methods in Applied Mechanics and Engineering.

[33]  Dimitrios I. Fotiadis,et al.  Artificial neural networks for solving ordinary and partial differential equations , 1997, IEEE Trans. Neural Networks.

[34]  Ameet Talwalkar,et al.  Foundations of Machine Learning , 2012, Adaptive computation and machine learning.

[35]  R. Molinaro,et al.  Estimates on the generalization error of Physics Informed Neural Networks (PINNs) for approximating PDEs , 2020, ArXiv.

[36]  Paris Perdikaris,et al.  Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations , 2019, J. Comput. Phys..

[37]  Gregory Seregin,et al.  Lecture Notes On Regularity Theory For The Navier-stokes Equations , 2014 .

[38]  Lucie Baudouin,et al.  Global Carleman Estimates for Waves and Applications , 2011, 1110.4085.

[39]  Luc Miller,et al.  Escape Function Conditions for the Observation, Control, and Stabilization of the Wave Equation , 2002, SIAM J. Control. Optim..

[40]  Kurt Hornik,et al.  Multilayer feedforward networks are universal approximators , 1989, Neural Networks.

[41]  T. Konstantin Rusch,et al.  Enhancing accuracy of deep learning algorithms by training with low-discrepancy sequences , 2020, SIAM J. Numer. Anal..

[42]  Peter Kuchment,et al.  Mathematics of thermoacoustic tomography , 2007, European Journal of Applied Mathematics.

[43]  Jan S. Hesthaven,et al.  An artificial neural network as a troubled-cell indicator , 2018, J. Comput. Phys..

[44]  Liu Yang,et al.  B-PINNs: Bayesian Physics-Informed Neural Networks for Forward and Inverse PDE Problems with Noisy Data , 2020, J. Comput. Phys..

[45]  P. Cochat,et al.  Et al , 2008, Archives de pediatrie : organe officiel de la Societe francaise de pediatrie.

[46]  Alfio Quarteroni,et al.  Machine learning for fast and reliable solution of time-dependent differential equations , 2019, J. Comput. Phys..

[47]  Fabien Caubet,et al.  Stability estimates for Navier-Stokes equations and application to inverse problems , 2016, 1609.03819.

[48]  O Yu Emanuilov,et al.  Controllability of parabolic equations , 1995 .

[49]  A. Owen.  Multidimensional variation for quasi-Monte Carlo , 2004 .

[50]  George Em Karniadakis,et al.  Hidden fluid mechanics: Learning velocity and pressure fields from flow visualizations , 2020, Science.

[51]  Zhiping Mao,et al.  DeepXDE: A Deep Learning Library for Solving Differential Equations , 2019, AAAI Spring Symposium: MLPS.

[52]  George E. Karniadakis,et al.  Hidden Fluid Mechanics: A Navier-Stokes Informed Deep Learning Framework for Assimilating Flow Visualization Data , 2018, ArXiv.

[53]  George Em Karniadakis,et al.  Adaptive activation functions accelerate convergence in deep and physics-informed neural networks , 2019, J. Comput. Phys..

[54]  George E. Karniadakis,et al.  Hidden physics models: Machine learning of nonlinear partial differential equations , 2017, J. Comput. Phys..

[55]  Robert Lattès,et al.  Méthode de quasi-réversibilbilité et applications , 1967 .

[56]  Luca Rondi,et al.  The stability for the Cauchy problem for elliptic equations , 2009, 0907.2882.

[57]  G. Karniadakis,et al.  Conservative physics-informed neural networks on discrete domains for conservation laws: Applications to forward and inverse problems , 2020 .

[58]  Arnulf Jentzen,et al.  Solving high-dimensional partial differential equations using deep learning , 2017, Proceedings of the National Academy of Sciences.

[59]  Emmanuel Tr'elat,et al.  Geometric control condition for the wave equation with a time-dependent observation domain , 2016, 1607.01527.

[60]  Andrew R. Barron,et al.  Universal approximation bounds for superpositions of a sigmoidal function , 1993, IEEE Trans. Inf. Theory.

[61]  R. Fletcher.  Practical Methods of Optimization , 1988 .

[62]  Oleg Yu. Imanuvilov,et al.  Remarks on exact controllability for the Navier-Stokes equations , 2001 .

[63]  Laurent Bourgeois,et al.  A mixed formulation of quasi-reversibility to solve the Cauchy problem for Laplace's equation , 2005 .