Bayesian neural networks for weak solution of PDEs with uncertainty quantification

Solving partial differential equations (PDEs) is the canonical approach for understanding the behavior of physical systems. However, large scale solutions of PDEs using state of the art discretization techniques remains an expensive proposition. In this work, a new physics-constrained neural network (NN) approach is proposed to solve PDEs without labels, with a view to enabling high-throughput solutions in support of design and decision-making. Distinct from existing physicsinformed NN approaches, where the strong form or weak form of PDEs are used to construct the loss function, we write the loss function of NNs based on the discretized residual of PDEs through an efficient, convolutional operator-based, and vectorized implementation. We explore an encoderdecoder NN structure for both deterministic and probabilistic models, with Bayesian NNs (BNNs) for the latter, which allow us to quantify both epistemic uncertainty from model parameters and aleatoric uncertainty from noise in the data. For BNNs, the discretized residual is used to construct the likelihood function. In our approach, both deterministic and probabilistic convolutional layers are used to learn the applied boundary conditions (BCs) and to detect the problem domain. As both Dirichlet and Neumann BCs are specified as inputs to NNs, a single NN can solve for similar physics, but with different BCs and on a number of problem domains. The trained surrogate PDE solvers can also make interpolating and extrapolating (to a certain extent) predictions for BCs that they were not exposed to during training. Such surrogate models are of particular importance for problems, where similar types of PDEs need to be repeatedly solved for many times with slight variations. We demonstrate the capability and performance of the proposed framework by applying it to different steady-state and equilibrium boundary value problems with physics that spans diffusion, linear elasticity, and nonlinear elasticity.

[1]  Karthik Ramani,et al.  ConvPDE-UQ: Convolutional neural networks with quantified uncertainty for heterogeneous elliptic partial differential equations on varied domains , 2019, J. Comput. Phys..

[2]  Max Welling,et al.  Auto-Encoding Variational Bayes , 2013, ICLR.

[3]  Julien Yvonnet,et al.  A data-driven computational homogenization method based on neural networks for the nonlinear anisotropic electrical response of graphene/polymer nanocomposites , 2018, Computational Mechanics.

[4]  Kaj Nyström,et al.  A unified deep artificial neural network approach to partial differential equations in complex geometries , 2017, Neurocomputing.

[5]  Sergey Ioffe,et al.  Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift , 2015, ICML.

[6]  Julien Cornebise,et al.  Weight Uncertainty in Neural Networks , 2015, ArXiv.

[7]  R. Asahi,et al.  Microstructure recognition using convolutional neural networks for prediction of ionic conductivity in ceramics , 2017 .

[8]  Yuksel C. Yabansu,et al.  Material structure-property linkages using three-dimensional convolutional neural networks , 2018 .

[9]  S. Brunton,et al.  Discovering governing equations from data by sparse identification of nonlinear dynamical systems , 2015, Proceedings of the National Academy of Sciences.

[10]  A. Kiureghian,et al.  Aleatory or epistemic? Does it matter? , 2009 .

[11]  Dimitrios I. Fotiadis,et al.  Artificial neural networks for solving ordinary and partial differential equations , 1997, IEEE Trans. Neural Networks.

[12]  Wei-keng Liao,et al.  Deep learning approaches for mining structure-property linkages in high contrast composites from simulation datasets , 2018 .

[13]  George Em Karniadakis,et al.  PPINN: Parareal Physics-Informed Neural Network for time-dependent PDEs , 2019, Computer Methods in Applied Mechanics and Engineering.

[14]  Paris Perdikaris,et al.  Physics-Constrained Deep Learning for High-dimensional Surrogate Modeling and Uncertainty Quantification without Labeled Data , 2019, J. Comput. Phys..

[15]  John K Kruschke,et al.  Bayesian data analysis. , 2010, Wiley interdisciplinary reviews. Cognitive science.

[16]  WaiChing Sun,et al.  Meta-modeling game for deriving theoretical-consistent, micro-structural-based traction-separation laws via deep reinforcement learning , 2018, Computer Methods in Applied Mechanics and Engineering.

[17]  Zoubin Ghahramani,et al.  Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning , 2015, ICML.

[18]  R. Jones,et al.  Predicting the mechanical response of oligocrystals with deep learning , 2019, Computational Materials Science.

[19]  Han Gao,et al.  PhyGeoNet: Physics-Informed Geometry-Adaptive Convolutional Neural Networks for Solving Parametric PDEs on Irregular Domain , 2020 .

[20]  Nicholas Zabaras,et al.  Bayesian Deep Convolutional Encoder-Decoder Networks for Surrogate Modeling and Uncertainty Quantification , 2018, J. Comput. Phys..

[21]  Dustin Tran,et al.  Flipout: Efficient Pseudo-Independent Weight Perturbations on Mini-Batches , 2018, ICLR.

[22]  Luning Sun,et al.  Surrogate modeling for fluid flows based on physics-constrained deep learning without simulation data , 2019, Computer Methods in Applied Mechanics and Engineering.

[23]  Andrew Gordon Wilson,et al.  A Simple Baseline for Bayesian Uncertainty in Deep Learning , 2019, NeurIPS.

[24]  George Em Karniadakis,et al.  Quantifying total uncertainty in physics-informed neural networks for solving forward and inverse stochastic problems , 2018, J. Comput. Phys..

[25]  Nicholas Geneva,et al.  Modeling the Dynamics of PDE Systems with Physics-Constrained Deep Auto-Regressive Networks , 2019, J. Comput. Phys..

[26]  X. Huan,et al.  Variational system identification of the partial differential equations governing microstructure evolution in materials: Inference over sparse and spatially unrelated data , 2020 .

[27]  G. Karniadakis,et al.  Conservative physics-informed neural networks on discrete domains for conservation laws: Applications to forward and inverse problems , 2020 .

[28]  David Wells,et al.  The deal.II library, Version 9.0 , 2018, J. Num. Math..

[29]  Michael M. Zavlanos,et al.  VarNet: Variational Neural Networks for the Solution of Partial Differential Equations , 2019, L4DC.

[30]  Youssef M A Hashash,et al.  Numerical implementation of a neural network based material model in finite element analysis , 2004 .

[31]  Alok Choudhary,et al.  Combinatorial screening for new materials in unconstrained composition space with machine learning , 2014 .

[32]  Paris Perdikaris,et al.  Adversarial Uncertainty Quantification in Physics-Informed Neural Networks , 2018, J. Comput. Phys..

[33]  Xun Huan,et al.  Variational system identification of the partial differential equations governing the physics of pattern-formation: Inference under varying fidelity and noise , 2018, Computer Methods in Applied Mechanics and Engineering.

[34]  Haizhao Yang,et al.  Friedrichs Learning: Weak Solutions of Partial Differential Equations via Deep Learning , 2021, SSRN Electronic Journal.

[35]  Alex Kendall,et al.  What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision? , 2017, NIPS.

[36]  Paris Perdikaris,et al.  Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations , 2019, J. Comput. Phys..

[37]  Kun Wang,et al.  A multiscale multi-permeability poroplasticity model linked by recursive homogenizations and deep learning , 2018, Computer Methods in Applied Mechanics and Engineering.

[38]  Timon Rabczuk,et al.  An Energy Approach to the Solution of Partial Differential Equations in Computational Mechanics via Machine Learning: Concepts, Implementation and Applications , 2019, Computer Methods in Applied Mechanics and Engineering.

[39]  Ridha Hambli,et al.  Multiscale methodology for bone remodelling simulation using coupled finite element and neural network computation , 2011, Biomechanics and modeling in mechanobiology.

[40]  Xihaier Luo,et al.  Bayesian deep learning with hierarchical prior: Predictions from limited and noisy data , 2019, Structural Safety.

[41]  B. A. Le,et al.  Computational homogenization of nonlinear elastic materials using neural networks , 2015 .

[42]  Liu Yang,et al.  B-PINNs: Bayesian Physics-Informed Neural Networks for Forward and Inverse PDE Problems with Noisy Data , 2020, J. Comput. Phys..

[43]  Xiang Li,et al.  Predicting the effective mechanical property of heterogeneous materials by image based modeling and deep learning , 2019, Computer Methods in Applied Mechanics and Engineering.

[44]  Ke Li,et al.  D3M: A Deep Domain Decomposition Method for Partial Differential Equations , 2020, IEEE Access.

[45]  Francisco Chinesta,et al.  Hybrid constitutive modeling: data-driven learning of corrections to plasticity models , 2018, International Journal of Material Forming.

[46]  Christian Linder,et al.  A marching cubes based failure surface propagation concept for three‐dimensional finite elements with non‐planar embedded strong discontinuities of higher‐order kinematics , 2013 .

[47]  Charbel Farhat,et al.  Learning constitutive relations from indirect observations using deep neural networks , 2020, J. Comput. Phys..

[48]  George Em Karniadakis,et al.  NSFnets (Navier-Stokes flow nets): Physics-informed neural networks for the incompressible Navier-Stokes equations , 2020, J. Comput. Phys..

[49]  Wei Chen,et al.  A framework for data-driven analysis of materials under uncertainty: Countering the curse of dimensionality , 2017 .

[50]  Alex Graves,et al.  Practical Variational Inference for Neural Networks , 2011, NIPS.

[51]  Alok Choudhary,et al.  A General-Purpose Machine Learning Framework for Predicting Properties of Inorganic Materials , 2016 .

[52]  David M. Blei,et al.  Variational Inference: A Review for Statisticians , 2016, ArXiv.

[53]  Justin A. Sirignano,et al.  DGM: A deep learning algorithm for solving partial differential equations , 2017, J. Comput. Phys..

[54]  Xun Huan,et al.  A perspective on regression and Bayesian approaches for system identification of pattern formation dynamics , 2020, 2001.05646.

[55]  Li Li,et al.  Bypassing the Kohn-Sham equations with machine learning , 2016, Nature Communications.

[56]  Dilin Wang,et al.  Stein Variational Gradient Descent: A General Purpose Bayesian Inference Algorithm , 2016, NIPS.

[57]  Chiho Kim,et al.  Machine learning in materials informatics: recent applications and prospects , 2017, npj Computational Materials.

[58]  Krishna Garikipati,et al.  Machine learning materials physics: Surrogate optimization and multi-fidelity algorithms predict precipitate morphology in an alternative to phase field dynamics , 2018, Computer Methods in Applied Mechanics and Engineering.

[59]  Y. Zhang,et al.  Reaction diffusion system prediction based on convolutional neural network , 2020, Scientific Reports.

[60]  Krishna Garikipati,et al.  Scale bridging materials physics: Active learning workflows and integrable deep neural networks for free energy function representations in alloys , 2020 .

[61]  Anton Van der Ven,et al.  Machine learning materials physics: Integrable deep neural networks enable scale bridging by learning free energy functions , 2019, Computer Methods in Applied Mechanics and Engineering.

[62]  Arnulf Jentzen,et al.  Solving high-dimensional partial differential equations using deep learning , 2017, Proceedings of the National Academy of Sciences.

[63]  Frederic E. Bock,et al.  A Review of the Application of Machine Learning and Data Mining Approaches in Continuum Materials Mechanics , 2019, Front. Mater..

[64]  George Em Karniadakis,et al.  hp-VPINNs: Variational Physics-Informed Neural Networks With Domain Decomposition , 2020, Computer Methods in Applied Mechanics and Engineering.

[65]  George Em Karniadakis,et al.  fPINNs: Fractional Physics-Informed Neural Networks , 2018, SIAM J. Sci. Comput..

[66]  Gang Bao,et al.  Weak Adversarial Networks for High-dimensional Partial Differential Equations , 2019, J. Comput. Phys..