Adaptive Physics-Informed Neural Networks for Markov-Chain Monte Carlo

In this paper, we propose the Adaptive Physics-Informed Neural Networks (APINNs) for accurate and efficient simulation-free Bayesian parameter estimation via Markov-Chain Monte Carlo (MCMC). We specifically focus on a class of parameter estimation problems for which computing the likelihood function requires solving a PDE. The proposed method consists of: (1) constructing an offline PINN-UQ model as an approximation to the forward model; and (2) refining this approximate model on the fly using samples generated from the MCMC sampler. The proposed APINN method constantly refines this approximate model on the fly and guarantees that the approximation error is always less than a user-defined residual error threshold. We numerically demonstrate the performance of the proposed APINN method in solving a parameter estimation problem for a system governed by the Poisson equation.

[1]  J. Besag,et al.  Bayesian Computation and Stochastic Systems , 1995 .

[2]  Tony Shardlow,et al.  A deep surrogate approach to efficient Bayesian inversion in PDE and integral equation models , 2019, ArXiv.

[3]  Antony M. Overstall,et al.  A Strategy for Bayesian Inference for Computationally Expensive Models with Application to the Estimation of Stem Cell Properties , 2013, Biometrics.

[4]  Habib N. Najm,et al.  Stochastic spectral methods for efficient Bayesian solution of inverse problems , 2005, J. Comput. Phys..

[5]  Andrew Gelman,et al.  Handbook of Markov Chain Monte Carlo , 2011 .

[6]  Paris Perdikaris,et al.  Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations , 2019, J. Comput. Phys..

[7]  Tsuyoshi Murata,et al.  {m , 1934, ACML.

[8]  Lyle H. Ungar,et al.  A hybrid neural network‐first principles approach to process modeling , 1992 .

[9]  George Em Karniadakis,et al.  A composite neural network that learns from multi-fidelity data: Application to function approximation and inverse PDE problems , 2019, J. Comput. Phys..

[10]  Hadi Meidani,et al.  A deep learning solution approach for high-dimensional random differential equations , 2019, Probabilistic Engineering Mechanics.

[11]  Christian P. Robert,et al.  Markov Chain Monte Carlo Methods, Survey with Some Frequent Misunderstandings , 2020, Wiley StatsRef: Statistics Reference Online.

[12]  Michael S. Triantafyllou,et al.  Deep learning of vortex-induced vibrations , 2018, Journal of Fluid Mechanics.

[13]  Geoffrey E. Hinton,et al.  Deep Learning , 2015, Nature.

[14]  Bin Dong,et al.  PDE-Net: Learning PDEs from Data , 2017, ICML.

[15]  Sebastian Ruder,et al.  An overview of gradient descent optimization algorithms , 2016, Vestnik komp'iuternykh i informatsionnykh tekhnologii.

[16]  Sebastian Becker,et al.  Solving stochastic differential equations and Kolmogorov equations by means of deep learning , 2018, ArXiv.

[17]  Paris Perdikaris,et al.  Physics-Constrained Deep Learning for High-dimensional Surrogate Modeling and Uncertainty Quantification without Labeled Data , 2019, J. Comput. Phys..

[18]  W. Michael Conklin,et al.  Monte Carlo Methods in Bayesian Computation , 2001, Technometrics.

[19]  Léon Bottou,et al.  Stochastic Gradient Descent Tricks , 2012, Neural Networks: Tricks of the Trade.

[20]  Yuan Yu,et al.  TensorFlow: A system for large-scale machine learning , 2016, OSDI.

[21]  T. Rabczuk,et al.  A Deep Collocation Method for the Bending Analysis of Kirchhoff Plate , 2021, Computers, Materials & Continua.

[22]  N. Metropolis,et al.  Equation of State Calculations by Fast Computing Machines , 1953, Resonance.

[23]  W. K. Hastings,et al.  Monte Carlo Sampling Methods Using Markov Chains and Their Applications , 1970 .

[24]  Jinglai Li,et al.  Adaptive Construction of Surrogates for the Bayesian Solution of Inverse Problems , 2013, SIAM J. Sci. Comput..

[25]  D. Dunson,et al.  The Hastings algorithm at fifty , 2020 .

[26]  Geoffrey E. Hinton,et al.  On the importance of initialization and momentum in deep learning , 2013, ICML.

[27]  Hadi Meidani,et al.  Physics-Driven Regularization of Deep Neural Networks for Enhanced Engineering Design and Analysis , 2018, J. Comput. Inf. Sci. Eng..

[28]  Paris Perdikaris,et al.  Adversarial Uncertainty Quantification in Physics-Informed Neural Networks , 2018, J. Comput. Phys..

[29]  Nando de Freitas,et al.  An Introduction to MCMC for Machine Learning , 2004, Machine Learning.

[30]  A. P. Dawid,et al.  Gaussian Processes to Speed up Hybrid Monte Carlo for Expensive Bayesian Integrals , 2003 .

[31]  Pavel B. Bochev,et al.  Least-Squares Finite Element Methods , 2009, Applied mathematical sciences.

[32]  Aaron Smith,et al.  Parallel Local Approximation MCMC for Expensive Models , 2016, SIAM/ASA J. Uncertain. Quantification.

[33]  N. Phan-Thien,et al.  Neural-network-based approximations for solving partial differential equations , 1994 .

[34]  Pascal Fua,et al.  Imposing Hard Constraints on Deep Networks: Promises and Limitations , 2017, CVPR 2017.

[35]  Léon Bottou,et al.  Large-Scale Machine Learning with Stochastic Gradient Descent , 2010, COMPSTAT.

[36]  Yoram Singer,et al.  Adaptive Subgradient Methods for Online Learning and Stochastic Optimization , 2011, J. Mach. Learn. Res..

[37]  Paris Perdikaris,et al.  Physics Informed Deep Learning (Part II): Data-driven Discovery of Nonlinear Partial Differential Equations , 2017, ArXiv.

[38]  Liu Yang,et al.  Physics-Informed Generative Adversarial Networks for Stochastic Differential Equations , 2018, SIAM J. Sci. Comput..

[39]  Barak A. Pearlmutter,et al.  Automatic differentiation in machine learning: a survey , 2015, J. Mach. Learn. Res..

[40]  Shie-Yui Liong,et al.  Efficient MCMC Schemes for Computationally Expensive Posterior Distributions , 2011, Technometrics.

[41]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[42]  Matthew D. Zeiler.  ADADELTA: An Adaptive Learning Rate Method , 2012, ArXiv.

[43]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[44]  Adrian F. M. Smith,et al.  Sampling-Based Approaches to Calculating Marginal Densities , 1990 .

[45]  E Weinan,et al.  Deep Learning-Based Numerical Methods for High-Dimensional Parabolic Partial Differential Equations and Backward Stochastic Differential Equations , 2017, Communications in Mathematics and Statistics.

[46]  Klaus-Robert Müller,et al.  Efficient BackProp , 2012, Neural Networks: Tricks of the Trade.

[47]  George Em Karniadakis,et al.  Adaptive activation functions accelerate convergence in deep and physics-informed neural networks , 2019, J. Comput. Phys..

[48]  L. Tierney.  Markov Chains for Exploring Posterior Distributions , 1994 .

[49]  Kaj Nyström,et al.  A unified deep artificial neural network approach to partial differential equations in complex geometries , 2017, Neurocomputing.

[50]  Stefan M. Wild,et al.  Bayesian Calibration and Uncertainty Analysis for Computationally Expensive Models Using Optimization and Radial Basis Function Approximation , 2008 .

[51]  Dimitrios I. Fotiadis,et al.  Artificial neural networks for solving ordinary and partial differential equations , 1997, IEEE Trans. Neural Networks.

[52]  Heikki Haario,et al.  DRAM: Efficient adaptive MCMC , 2006, Stat. Comput..

[53]  E Weinan,et al.  The Deep Ritz Method: A Deep Learning-Based Numerical Algorithm for Solving Variational Problems , 2017, Communications in Mathematics and Statistics.

[54]  Tao Zhou,et al.  An adaptive surrogate modeling based on deep neural networks for large-scale Bayesian inverse problems , 2019, ArXiv.

[55]  Yann LeCun,et al.  Second Order Properties of Error Surfaces: Learning Time and Generalization , 1990, NIPS 1990.