Hybrid FEM-NN models: Combining artificial neural networks with the finite element method

We present a methodology combining neural networks with physical principle constraints in the form of partial differential equations (PDEs). The approach allows to train neural networks while respecting the PDEs as a strong constraint in the optimisation as apposed to making them part of the loss function. The resulting models are discretised in space by the finite element method (FEM). The methodology applies to both stationary and transient as well as linear/nonlinear PDEs. We describe how the methodology can be implemented as an extension of the existing FEM framework FEniCS and its algorithmic differentiation tool dolfin-adjoint. Through series of examples we demonstrate capabilities of the approach to recover coefficients and missing PDE operators from observations. Further, the proposed method is compared with alternative methodologies, namely, physics informed neural networks and standard PDE-constrained optimisation. Finally, we demonstrate the method on a complex cardiac cell model problem using deep neural networks.

[1]  Joakim Sundnes,et al.  Computing the electrical activity in the heart , 2006 .

[2]  Anders Logg,et al.  Unified form language: A domain-specific language for weak formulations of partial differential equations , 2012, TOMS.

[3]  Jorge Nocedal,et al.  Algorithm 778: L-BFGS-B: Fortran subroutines for large-scale bound-constrained optimization , 1997, TOMS.

[4]  S. Nash.  Newton-Type Minimization via the Lanczos Method , 1984 .

[5]  Paris Perdikaris,et al.  Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations , 2019, J. Comput. Phys..

[6]  Ameya D. Jagtap,et al.  Extended Physics-informed Neural Networks (XPINNs): A Generalized Space-Time Domain Decomposition based Deep Learning Framework for Nonlinear Partial Differential Equations , 2020, AAAI Spring Symposium: MLPS.

[7]  Apparent diffusion coefficient estimates based on 24 hours tracer movement support glymphatic transport in human cerebral cortex , 2020, Scientific Reports.

[8]  Bin Dong,et al.  PDE-Net 2.0: Learning PDEs from Data with A Numeric-Symbolic Hybrid Deep Network , 2018, J. Comput. Phys..

[9]  Bin Dong,et al.  PDE-Net: Learning PDEs from Data , 2017, ICML.

[10]  Charbel Farhat,et al.  Learning constitutive relations from indirect observations using deep neural networks , 2020, J. Comput. Phys..

[11]  David Duvenaud,et al.  Neural Ordinary Differential Equations , 2018, NeurIPS.

[12]  George Em Karniadakis,et al.  On the Convergence and generalization of Physics Informed Neural Networks , 2020, ArXiv.

[13]  A. Kværnø,et al.  Norges Teknisk-naturvitenskapelige Universitet Singly Diagonally Implicit Runge-kutta Methods with an Explicit First Stage Singly Diagonally Implicit Runge-kutta Methods with an Explicit First Stage , 2022 .

[14]  Eldad Haber,et al.  Deep Neural Networks Motivated by Partial Differential Equations , 2018, Journal of Mathematical Imaging and Vision.

[15]  Eric Darve,et al.  Solving Inverse Problems in Steady State Navier-Stokes Equations using Deep Neural Networks , 2020, ArXiv.

[16]  G. E. Karniadakis,et al.  Variational Physics-Informed Neural Networks For Solving Partial Differential Equations , 2019, ArXiv.

[17]  Miss A.O. Penney.  (b) , 1974, The New Yale Book of Quotations.

[18]  Anders Logg,et al.  DOLFIN: Automated finite element computing , 2010, TOMS.

[19]  Natalia Gimelshein,et al.  PyTorch: An Imperative Style, High-Performance Deep Learning Library , 2019, NeurIPS.

[20]  David A. Ham,et al.  Exposing and exploiting structure: optimal code generation for high-order finite element methods , 2017, ArXiv.

[21]  Karthik Duraisamy,et al.  Turbulence Modeling in the Age of Data , 2018, Annual Review of Fluid Mechanics.

[22]  K. Skare.  Gryphon - a Module for Time Integration of Partial Differential Equations in FEniCS , 2012 .

[23]  S. Brunton,et al.  Discovering governing equations from data by sparse identification of nonlinear dynamical systems , 2015, Proceedings of the National Academy of Sciences.

[24]  George Em Karniadakis,et al.  Adaptive activation functions accelerate convergence in deep and physics-informed neural networks , 2019, J. Comput. Phys..

[25]  G. Karniadakis,et al.  Conservative physics-informed neural networks on discrete domains for conservation laws: Applications to forward and inverse problems , 2020 .

[26]  Jorge Nocedal,et al.  On the limited memory BFGS method for large scale optimization , 1989, Math. Program..

[27]  Marie E. Rognes,et al.  Automated Adjoints of Coupled PDE-ODE Systems , 2017, SIAM J. Sci. Comput..

[28]  G. W. Beeler,et al.  Reconstruction of the action potential of ventricular myocardial fibres , 1977, The Journal of physiology.

[29]  D. Noble.  A modification of the Hodgkin—Huxley equations applicable to Purkinje fibre action and pacemaker potentials , 1962, The Journal of physiology.

[30]  R. Glowinski.  Finite element methods for incompressible viscous flow , 2003 .

[31]  Shih-Yu Shen,et al.  A numerical study of inverse heat conduction problems , 1999 .

[32]  G. G. Stokes.  "J." , 1890, The New Yale Book of Quotations.

[33]  R. Molinaro,et al.  Estimates on the generalization error of Physics Informed Neural Networks (PINNs) for approximating PDEs , 2020, ArXiv.

[34]  Nicolas Cedilnik,et al.  EP-Net: Learning Cardiac Electrophysiology Models for Physiology-Based Constraints in Data-Driven Predictions , 2019, FIMH.

[35]  Simon W. Funke,et al.  dolfin-adjoint 2018.1: automated adjoints for FEniCS and Firedrake , 2019, J. Open Source Softw..

[36]  Stefan Ulbrich,et al.  Analytical Background and Optimality Theory , 2009 .

[37]  Maziar Raissi,et al.  Deep Hidden Physics Models: Deep Learning of Nonlinear Partial Differential Equations , 2018, J. Mach. Learn. Res..

[38]  A. Hodgkin,et al.  A quantitative description of membrane current and its application to conduction and excitation in nerve , 1952, The Journal of physiology.

[39]  Uwe Naumann,et al.  The Art of Differentiating Computer Programs - An Introduction to Algorithmic Differentiation , 2012, Software, environments, tools.

[40]  Ali Ramadhan,et al.  Universal Differential Equations for Scientific Machine Learning , 2020, ArXiv.

[41]  Lars Ruthotto,et al.  Learning Across Scales - Multiscale Methods for Convolution Neural Networks , 2018, AAAI.