The Neural Network Approach to Inverse Problems in Differential Equations

We proposed a framework for solving inverse problems in differential equations based on neural networks and automatic differentiation. Neural networks are used to approximate hidden fields. We analyze the source of errors in the framework and derive an error estimate for a model diffusion equation problem. Besides, we propose a way for sensitivity analysis, utilizing the automatic differentiation mechanism embedded in the framework. It frees people from the tedious and error-prone process of deriving the gradients. Numerical examples exhibit consistency with the convergence analysis and error saturation is noteworthily predicted. We also demonstrate the unique benefits neural networks offer at the same time: universal approximation ability, regularizing the solution, bypassing the curse of dimensionality and leveraging efficient computing frameworks.

[1]  Kagan Tumer,et al.  Bayes Error Rate Estimation Using Classifier Ensembles , 2003 .

[2]  A. Stuart,et al.  The Bayesian Approach to Inverse Problems , 2013, 1302.6989.

[3]  Otmar Scherzer,et al.  Variational Methods in Imaging , 2008, Applied mathematical sciences.

[4]  Eric Darve,et al.  Calibrating Multivariate Lévy Processes with Neural Networks , 2018, MSML.

[5]  Antony Jameson,et al.  Aerodynamic Shape Optimization Using the Adjoint Method , 2003 .

[6]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[7]  I. Elshafiey.  Neural network approach for solving inverse problems , 1991 .

[8]  D. Patella.  Geophysical Tomography In Engineering Geology: An Overview , 2001, physics/0512154.

[9]  O. Scherzer,et al.  Sparsity regularization in inverse problems , 2017 .

[10]  George Em Karniadakis,et al.  fPINNs: Fractional Physics-Informed Neural Networks , 2018, SIAM J. Sci. Comput..

[11]  J. Brown,et al.  Density diagnostics and inhomogeneous plasmas. I: Isothermal plasmas , 1989 .

[12]  On the numerical solution of a variable-coefficient Burgers equation arising in granular segregation , 2017, 1707.00034.

[13]  Eric Darve,et al.  Calibrating Lévy Process from Observations Based on Neural Networks and Automatic Differentiation with Convergence Proofs , 2018, ArXiv.

[14]  Bernhard Pfahringer,et al.  Regularisation of neural networks by enforcing Lipschitz continuity , 2018, Machine Learning.

[15]  Andreas Fichtner,et al.  The adjoint method in seismology—: II. Applications: traveltimes and sensitivity functionals , 2006 .

[16]  Konstantin Khanin,et al.  Burgers Turbulence , 2007, Energy Transfers in Fluid Flows.

[17]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[18]  Allan Pinkus,et al.  Lower bounds for approximation by MLP neural networks , 1999, Neurocomputing.

[19]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[20]  Hao Li,et al.  Visualizing the Loss Landscape of Neural Nets , 2017, NeurIPS.

[21]  Takehiko Ogawa,et al.  Neural network based solution to inverse problems , 1998, 1998 IEEE International Joint Conference on Neural Networks Proceedings. IEEE World Congress on Computational Intelligence (Cat. No.98CH36227).

[22]  Tan Bui-Thanh,et al.  A Gentle Tutorial on Statistical Inversion using the Bayesian Paradigm , 2012 .

[23]  E Weinan,et al.  DeePMD-kit: A deep learning package for many-body potential energy representation and molecular dynamics , 2017, Comput. Phys. Commun..

[24]  Lei Wu,et al.  Understanding and Enhancing the Transferability of Adversarial Examples , 2018, ArXiv.

[25]  R. Plessix.  A review of the adjoint-state method for computing the gradient of a functional with geophysical applications , 2006 .

[26]  Barbara Kaltenbacher,et al.  Regularization Methods in Banach Spaces , 2012, Radon Series on Computational and Applied Mathematics.

[27]  I. Daubechies,et al.  An iterative thresholding algorithm for linear inverse problems with a sparsity constraint , 2003, math/0307152.

[28]  Hongguang Sun,et al.  Anomalous diffusion modeling by fractal and fractional derivatives , 2010, Comput. Math. Appl..

[29]  Xing Liu,et al.  Kolmogorov superposition theorem and its applications , 2015 .

[30]  Luca Antiga,et al.  Automatic differentiation in PyTorch , 2017 .

[31]  Liu Yang,et al.  Neural-net-induced Gaussian process regression for function approximation and PDE solution , 2018, J. Comput. Phys..

[32]  J. Shewchuk.  An Introduction to the Conjugate Gradient Method Without the Agonizing Pain , 1994 .

[33]  Paris Perdikaris,et al.  Physics Informed Deep Learning (Part I): Data-driven Solutions of Nonlinear Partial Differential Equations , 2017, ArXiv.

[34]  George Em Karniadakis,et al.  Quantifying total uncertainty in physics-informed neural networks for solving forward and inverse stochastic problems , 2018, J. Comput. Phys..

[35]  H. N. Mhaskar,et al.  Neural Networks for Optimal Approximation of Smooth and Analytic Functions , 1996, Neural Computation.

[36]  Paris Perdikaris,et al.  Physics Informed Deep Learning (Part II): Data-driven Discovery of Nonlinear Partial Differential Equations , 2017, ArXiv.

[37]  Luis Santos,et al.  Aerodynamic shape optimization using the adjoint method , 2007 .

[38]  Yuanzhi Li,et al.  An Alternative View: When Does SGD Escape Local Minima? , 2018, ICML.

[39]  G. Lorentz,et al.  Constructive approximation : advanced problems , 1996 .

[40]  Dan G. Cacuci,et al.  Sensitivity Analysis of a Radiative-Convective Model by the Adjoint Method , 1982 .

[41]  E Weinan,et al.  Reinforced dynamics for enhanced sampling in large atomic and molecular systems. I. Basic Methodology , 2017, The Journal of chemical physics.

[42]  Niles A. Pierce,et al.  An Introduction to the Adjoint Approach to Design , 2000 .

[43]  Denis Talay,et al.  Probabilistic numerical methods for partial differential equations: Elements of analysis , 1996 .

[44]  Andrew M. Stuart,et al.  Inverse problems: A Bayesian perspective , 2010, Acta Numerica.

[45]  Zheng Zhang,et al.  MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems , 2015, ArXiv.

[46]  E Weinan,et al.  A mean-field optimal control formulation of deep learning , 2018, Research in the Mathematical Sciences.

[47]  S. Arridge.  Optical tomography in medical imaging , 1999 .

[48]  Tobias Glasmachers,et al.  Limits of End-to-End Learning , 2017, ACML.

[49]  I. Daubechies,et al.  Sparsity-enforcing regularisation and ISTA revisited , 2016 .