Scientific Machine Learning through Physics-Informed Neural Networks: Where we are and What's next

Physic-Informed Neural Networks (PINN) are neural networks (NNs) that encode model equations, like Partial Differential Equations (PDE), as a component of the neural network itself. PINNs are nowadays used to solve PDEs, fractional equations, and integral-differential equations. This novel methodology has arisen as a multi-task learning framework in which a NN must fit observed data while reducing a PDE residual. This article provides a comprehensive review of the literature on PINNs: while the primary goal of the study was to characterize these networks and their related advantages and disadvantages, the 1 ar X iv :2 20 1. 05 62 4v 1 [ cs .L G ] 1 4 Ja n 20 22 2 Scientific Machine Learning through PINNs review also attempts to incorporate publications on a larger variety of issues, including physics-constrained neural networks (PCNN), where the initial or boundary conditions are directly embedded in the NN structure rather than in the loss functions. The study indicates that most research has focused on customizing the PINN through different activation functions, gradient optimization techniques, neural network structures, and loss function structures. Despite the wide range of applications for which PINNs have been used, by demonstrating their ability to be more feasible in some contexts than classical numerical techniques like Finite Element Method (FEM), advancements are still possible, most notably theoretical issues that remain unresolved.

[1]  Spyros Chatzivasileiadis,et al.  Physics-Informed Neural Networks for Power Systems , 2020, 2020 IEEE Power & Energy Society General Meeting (PESGM).

[2]  Paris Perdikaris,et al.  Machine learning of linear differential equations using Gaussian processes , 2017, J. Comput. Phys..

[3]  Michael Spannowsky,et al.  Elvet - a neural network-based differential equation and variational problem solver , 2021, ArXiv.

[4]  Dmitry Yarotsky,et al.  Error bounds for approximations with deep ReLU networks , 2016, Neural Networks.

[5]  George Em Karniadakis,et al.  Hidden fluid mechanics: Learning velocity and pressure fields from flow visualizations , 2020, Science.

[6]  B. Rogers,et al.  Uncovering turbulent plasma dynamics via deep learning from partial observations. , 2020, Physical review. E.

[7]  S. Barnett,et al.  Philosophical Transactions of the Royal Society A : Mathematical , 2017 .

[8]  Nicholas Geneva,et al.  Modeling the Dynamics of PDE Systems with Physics-Constrained Deep Auto-Regressive Networks , 2019, J. Comput. Phys..

[9]  Moataz O. Abu-Al-Saud,et al.  Prediction of porous media fluid flow using physics informed neural networks , 2022 .

[10]  George Em Karniadakis,et al.  fPINNs: Fractional Physics-Informed Neural Networks , 2018, SIAM J. Sci. Comput..

[11]  Zhiwei Fang,et al.  Deep Physical Informed Neural Networks for Metamaterial Design , 2020, IEEE Access.

[12]  Simon See,et al.  Recent advance in machine learning for partial differential equation , 2021, CCF Transactions on High Performance Computing.

[13]  George Em Karniadakis,et al.  DeepM&Mnet: Inferring the electroconvection multiphysics fields based on operator approximation by neural networks , 2021, J. Comput. Phys..

[14]  Dong Eui Chang,et al.  Deep Neural Networks in a Mathematical Framework , 2018, SpringerBriefs in Computer Science.

[15]  Alexander Koryagin,et al.  PyDEns: a Python Framework for Solving Differential Equations with Neural Networks , 2019, ArXiv.

[16]  A. Tartakovsky,et al.  Physics‐Informed Neural Network Method for Forward and Backward Advection‐Dispersion Equations , 2020, Water Resources Research.

[17]  Felipe Antonio Chegury Viana,et al.  Hybrid physics-informed neural networks for main bearing fatigue prognosis with visual grease inspection , 2021, Comput. Ind..

[18]  Murray Shanahan,et al.  Reconciling deep learning with symbolic artificial intelligence: representing objects and relations , 2019, Current Opinion in Behavioral Sciences.

[19]  Wei Peng,et al.  IDRLnet: A Physics-Informed Neural Network Library , 2021, ArXiv.

[20]  Marc Peter Deisenroth,et al.  Deep Reinforcement Learning: A Brief Survey , 2017, IEEE Signal Processing Magazine.

[21]  Yigit A. Yucesan,et al.  Estimating model inadequacy in ordinary differential equations with physics-informed neural networks , 2021, Computers & Structures.

[22]  Daniel S. Berman,et al.  A Survey of Deep Learning Methods for Cyber Security , 2019, Inf..

[23]  Satish Nagarajaiah,et al.  Structural identification with physics-informed neural ordinary differential equations , 2021 .

[24]  G. Karniadakis,et al.  Physics-Informed Neural Networks for Heat Transfer Problems , 2021, Journal of Heat Transfer.

[25]  Oliver Hennigh,et al.  NVIDIA SimNet^{TM}: an AI-accelerated multi-physics simulation framework , 2020, ArXiv.

[26]  George E. Karniadakis,et al.  Hidden physics models: Machine learning of nonlinear partial differential equations , 2017, J. Comput. Phys..

[27]  Baskar Ganapathysubramanian,et al.  Distributed Multigrid Neural Solvers on Megavoxel Domains , 2021, SC21: International Conference for High Performance Computing, Networking, Storage and Analysis.

[28]  Paris Perdikaris,et al.  Physics-Constrained Deep Learning for High-dimensional Surrogate Modeling and Uncertainty Quantification without Labeled Data , 2019, J. Comput. Phys..

[29]  A. Tartakovsky,et al.  Physics-Informed Neural Networks for Multiphysics Data Assimilation with Application to Subsurface Transport , 2019, Advances in Water Resources.

[30]  Timon Rabczuk,et al.  Transfer learning enhanced physics informed neural network for phase-field modeling of fracture , 2019, Theoretical and Applied Fracture Mechanics.

[31]  Devesh Upadhyay,et al.  NeuralPDE: Automating Physics-Informed Neural Networks (PINNs) with Error Approximations , 2021, ArXiv.

[32]  Mustafa Man,et al.  ANNs-Based Method for Solving Partial Differential Equations : A Survey , 2021 .

[33]  Paris Perdikaris,et al.  Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations , 2019, J. Comput. Phys..

[34]  George Em Karniadakis,et al.  Learning nonlinear operators via DeepONet based on the universal approximation theorem of operators , 2019, Nature Machine Intelligence.

[35]  Aleksandar Ignjatovic,et al.  Machine learning for accelerating 2D flood models: Potential and challenges , 2021, Hydrological Processes.

[36]  Maziar Raissi,et al.  Deep Hidden Physics Models: Deep Learning of Nonlinear Partial Differential Equations , 2018, J. Mach. Learn. Res..

[37]  R. G. Endres,et al.  Physics-informed deep learning characterizes morphodynamics of Asian soybean rust disease , 2021, Nature Communications.

[38]  Seungchul Lee,et al.  Knowledge Integration into deep learning in dynamical systems: an overview and taxonomy , 2021, Journal of Mechanical Science and Technology.

[39]  Tariq Alkhalifah,et al.  PINNeik: Eikonal solution using physics-informed neural networks , 2020, Comput. Geosci..

[40]  Paris Perdikaris,et al.  Learning atrial fiber orientations and conductivity tensors from intracardiac maps using physics-informed neural networks , 2021, FIMH.

[41]  Ali Ramadhan,et al.  Universal Differential Equations for Scientific Machine Learning , 2020, ArXiv.

[42]  Kurt Hornik,et al.  Multilayer feedforward networks are universal approximators , 1989, Neural Networks.

[43]  L. Dal Negro,et al.  Physics-informed neural networks for inverse problems in nano-optics and metamaterials. , 2019, Optics express.

[44]  Eric C. Cyr,et al.  Thermodynamically consistent physics-informed neural networks for hyperbolic systems , 2020, J. Comput. Phys..

[45]  Paris Perdikaris,et al.  Adversarial Uncertainty Quantification in Physics-Informed Neural Networks , 2018, J. Comput. Phys..

[46]  P. Ofner,et al.  Mesh-Free Surrogate Models for Structural Mechanic FEM Simulation: A Comparative Study of Approaches , 2021, Applied Sciences.

[47]  S. Kollmannsberger,et al.  Physics-Informed Neural Networks , 2021, Deep Learning in Computational Mechanics.

[48]  Siddhartha Mishra,et al.  Estimates on the generalization error of physics-informed neural networks for approximating a class of inverse problems for PDEs , 2020, IMA Journal of Numerical Analysis.

[49]  S. Mojumder,et al.  Extraction of material properties through multi-fidelity deep learning from molecular dynamics simulation , 2020, 2009.13106.

[50]  S. Scheel,et al.  Reconstruction of nanoscale particles from single-shot wide-angle free-electron-laser diffraction patterns with physics-informed neural networks. , 2021, Physical review. E.

[51]  Zhiping Mao,et al.  DeepXDE: A Deep Learning Library for Solving Differential Equations , 2019, AAAI Spring Symposium: MLPS.

[52]  Shiliang Sun,et al.  A Survey of Optimization Methods From a Machine Learning Perspective , 2019, IEEE Transactions on Cybernetics.

[53]  Houman Owhadi,et al.  Bayesian Numerical Homogenization , 2014, Multiscale Model. Simul..

[54]  Z. Fang A High-Efficient Hybrid Physics-Informed Neural Networks Based on Convolutional Neural Network , 2021, IEEE Transactions on Neural Networks and Learning Systems.

[55]  George Em Karniadakis,et al.  NSFnets (Navier-Stokes flow nets): Physics-informed neural networks for the incompressible Navier-Stokes equations , 2020, J. Comput. Phys..

[56]  Liu Yang,et al.  B-PINNs: Bayesian Physics-Informed Neural Networks for Forward and Inverse PDE Problems with Noisy Data , 2020, J. Comput. Phys..

[57]  Joanna Staneva,et al.  Towards neural Earth system modelling by integrating artificial intelligence in Earth system science , 2021, Nature Machine Intelligence.

[58]  Carl Leake,et al.  Extreme theory of functional connections: A fast physics-informed neural network method for solving ordinary and partial differential equations , 2021, Neurocomputing.

[59]  George Em Karniadakis,et al.  Extraction of mechanical properties of materials through deep learning from instrumented indentation , 2020, Proceedings of the National Academy of Sciences.

[60]  Yan Wang,et al.  A Dual-Dimer Method for Training Physics-Constrained Neural Networks with Minimax Architecture , 2020, Neural Networks.

[61]  Eric Darve,et al.  ADCME: Learning Spatially-varying Physical Fields using Deep Neural Networks , 2020, ArXiv.

[62]  Spyros Chatzivasileiadis,et al.  Physics-Informed Neural Networks for Non-linear System Identification for Power System Dynamics , 2021, 2021 IEEE Madrid PowerTech.

[63]  Jaehoon Lee,et al.  Neural Tangents: Fast and Easy Infinite Neural Networks in Python , 2019, ICLR.

[64]  Habib Shah,et al.  Deep learning application in smart cities: recent development, taxonomy, challenges and research prospects , 2020, Neural Computing and Applications.

[65]  Lyle H. Ungar,et al.  A hybrid neural network‐first principles approach to process modeling , 1992 .

[66]  Paris Perdikaris,et al.  Physics‐Informed Deep Neural Networks for Learning Parameters and Constitutive Relationships in Subsurface Flow Problems , 2020, Water Resources Research.

[67]  Paris Perdikaris,et al.  Numerical Gaussian Processes for Time-Dependent and Nonlinear Partial Differential Equations , 2017, SIAM J. Sci. Comput..

[68]  G. Karniadakis,et al.  Physics-informed neural networks for high-speed flows , 2020, Computer Methods in Applied Mechanics and Engineering.

[69]  Andrew P King,et al.  Active training of physics-informed neural networks to aggregate and interpolate parametric solutions to the Navier-Stokes equations , 2021, J. Comput. Phys..

[70]  Dimitrios I. Fotiadis,et al.  Artificial neural networks for solving ordinary and partial differential equations , 1997, IEEE Trans. Neural Networks.

[71]  Jinhui Yan,et al.  Machine learning for metal additive manufacturing: predicting temperature and melt pool fluid dynamics using physics-informed neural networks , 2020, Computational Mechanics.

[72]  M. Raissi,et al.  A physics-informed deep learning framework for inversion and surrogate modeling in solid mechanics , 2021, Computer Methods in Applied Mechanics and Engineering.

[73]  Eric Darve,et al.  Solving inverse problems in stochastic models using deep neural networks and adversarial training , 2021 .

[74]  Paris Perdikaris,et al.  When and why PINNs fail to train: A neural tangent kernel perspective , 2020, J. Comput. Phys..

[75]  Yang Liu,et al.  Physics-Informed Multi-LSTM Networks for Metamodeling of Nonlinear Structures , 2020, Computer Methods in Applied Mechanics and Engineering.

[76]  Yew-Soon Ong,et al.  Can Transfer Neuroevolution Tractably Solve Your Differential Equations? , 2021, IEEE Computational Intelligence Magazine.

[77]  Thomas Blaschke,et al.  The rise of deep learning in drug discovery. , 2018, Drug discovery today.

[78]  A. Wills,et al.  Physics-informed machine learning , 2021, Nature Reviews Physics.

[79]  M. Stein Large sample properties of simulations using latin hypercube sampling , 1987 .

[80]  Luka Grubisic,et al.  Deep Neural Network Model for Approximating Eigenmodes Localized by a Confining Potential , 2021, Entropy.

[81]  R. Juanes,et al.  SciANN: A Keras/TensorFlow wrapper for scientific computations and physics-informed deep learning using artificial neural networks , 2020, Computer Methods in Applied Mechanics and Engineering.

[82]  Peter R. Wiecha,et al.  Deep learning in nano-photonics: inverse design and beyond , 2020, Photonics Research.

[83]  Ahearn Driving Scientific and Engineering Discoveries Through the Convergence of HPC, Big Data and AI: 17th Smoky Mountains Computational Sciences and Engineering Conference, SMC 2020, Oak Ridge, TN, USA, August 26-28, 2020, Revised Selected Papers , 2020, SMC.

[84]  G. Karniadakis,et al.  A seamless multiscale operator neural network for inferring bubble dynamics , 2021, Journal of Fluid Mechanics.

[85]  Kaj Nyström,et al.  A unified deep artificial neural network approach to partial differential equations in complex geometries , 2017, Neurocomputing.

[86]  Eduardo Abreu,et al.  A study on a feedforward neural network to solve partial differential equations in hyperbolic-transport problems , 2021, ICCS.

[87]  T. Rabczuk,et al.  A surrogate model for computational homogenization of elastostatics at finite strain using high‐dimensional model representation‐based neural network , 2020, International Journal for Numerical Methods in Engineering.

[88]  Markus Gusenbauer,et al.  Conditional physics informed neural networks , 2021, Commun. Nonlinear Sci. Numer. Simul..

[89]  George Cybenko,et al.  Approximation by superpositions of a sigmoidal function , 1989, Math. Control. Signals Syst..

[90]  Chris Yakopcic,et al.  A State-of-the-Art Survey on Deep Learning Theory and Architectures , 2019, Electronics.

[91]  Stéphane Mallat,et al.  Understanding deep convolutional networks , 2016, Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences.

[92]  Richard Alan Peters,et al.  A Review of Deep Learning with Special Emphasis on Architectures, Applications and Recent Trends , 2019, Knowl. Based Syst..

[93]  Luning Sun,et al.  Surrogate modeling for fluid flows based on physics-constrained deep learning without simulation data , 2019, Computer Methods in Applied Mechanics and Engineering.

[94]  P. Perdikaris,et al.  Machine learning in cardiovascular flows modeling: Predicting arterial blood pressure from non-invasive 4D flow MRI data using physics-informed neural networks , 2019 .

[95]  Balaji Srinivasan,et al.  Physics Informed Extreme Learning Machine (PIELM) - A rapid method for the numerical solution of partial differential equations , 2019, Neurocomputing.

[96]  Paris Perdikaris,et al.  Deep learning of free boundary and Stefan problems , 2020, J. Comput. Phys..

[97]  Bo Zhu,et al.  Symplectic Neural Networks in Taylor Series Form for Hamiltonian Systems , 2020, J. Comput. Phys..

[98]  Liming Ling,et al.  Data-driven vector soliton solutions of coupled nonlinear Schrödinger equation using a deep learning algorithm , 2021, Physics Letters A.

[99]  George Em Karniadakis,et al.  PPINN: Parareal Physics-Informed Neural Network for time-dependent PDEs , 2019, Computer Methods in Applied Mechanics and Engineering.

[100]  Kamyar Azizzadenesheli,et al.  EikoNet: Solving the Eikonal Equation With Deep Neural Networks , 2021, IEEE Transactions on Geoscience and Remote Sensing.

[101]  Yibo Yang,et al.  Physics-Informed Neural Networks for Cardiac Activation Mapping , 2020, Frontiers in Physics.

[102]  Jan Blechschmidt,et al.  Three Ways to Solve Partial Differential Equations with Neural Networks - A Review , 2021, ArXiv.

[103]  Dianhui Wang,et al.  Extreme learning machines: a survey , 2011, Int. J. Mach. Learn. Cybern..

[104]  Heng Xiao,et al.  Flows over periodic hills of parameterized geometries: A dataset for data-driven turbulence modeling from direct simulations , 2019, Computers & Fluids.

[105]  Hyuk Lee,et al.  Neural algorithm for solving differential equations , 1990 .

[106]  Liu Yang,et al.  Physics-Informed Generative Adversarial Networks for Stochastic Differential Equations , 2018, SIAM J. Sci. Comput..

[107]  P. Schrimpf,et al.  Dynamic Programming , 2011 .

[108]  Ausif Mahmood,et al.  Review of Deep Learning Algorithms and Architectures , 2019, IEEE Access.

[109]  Renato Giorgiani do Nascimento,et al.  A tutorial on solving ordinary differential equations using Python and hybrid physics-informed neural network , 2020, Eng. Appl. Artif. Intell..

[110]  R. Vaziri,et al.  Physics-Informed Neural Network for Modelling the Thermochemical Curing Process of Composite-Tool Systems During Manufacture , 2020, ArXiv.

[111]  Paris Perdikaris,et al.  Inferring solutions of differential equations using noisy multi-fidelity data , 2016, J. Comput. Phys..

[112]  Noseong Park,et al.  DPM: A Novel Training Method for Physics-Informed Neural Networks in Extrapolation , 2020, AAAI.

[113]  Ruben Juanes,et al.  A nonlocal physics-informed deep learning framework using the peridynamic differential operator , 2020, ArXiv.

[114]  Chen Cheng,et al.  Deep Learning Method Based on Physics Informed Neural Network with Resnet Block for Solving Fluid Flow Problems , 2021, Water.

[115]  J. B. Muir,et al.  HypoSVI: Hypocentre inversion with Stein variational inference and physics informed neural networks , 2021 .

[116]  Andrew Gordon Wilson,et al.  GPyTorch: Blackbox Matrix-Matrix Gaussian Process Inference with GPU Acceleration , 2018, NeurIPS.

[117]  M. Bazant,et al.  A Physics-Guided Neural Network Framework for Elastic Plates: Comparison of Governing Equations-Based and Energy-Based Approaches , 2020, ArXiv.

[118]  Abdelouahid Derhab,et al.  Deep learning approaches for anomaly-based intrusion detection systems: A survey, taxonomy, and open issues , 2020, Knowl. Based Syst..

[119]  George Em Karniadakis,et al.  hp-VPINNs: Variational Physics-Informed Neural Networks With Domain Decomposition , 2020, Computer Methods in Applied Mechanics and Engineering.

[120]  George E. Karniadakis,et al.  DeepM&Mnet for hypersonics: Predicting the coupled flow and finite-rate chemistry behind a normal shock using neural-network approximation of operators , 2020, J. Comput. Phys..

[121]  A. Quarteroni Numerical Models for Differential Problems , 2009 .

[122]  Pavlos Protopapas,et al.  NeuroDiffEq: A Python package for solving differential equations with neural networks , 2020, J. Open Source Softw..

[123]  Sunna Torge,et al.  Large-scale Neural Solvers for Partial Differential Equations , 2020, SMC.

[124]  Li Wang,et al.  Data-driven rogue waves and parameter discovery in the defocusing NLS equation with a potential using the PINN deep learning , 2020, Physics Letters A.

[125]  Haibin Chang,et al.  Theory-guided Auto-Encoder for Surrogate Construction and Inverse Modeling , 2020, ArXiv.

[126]  Houman Owhadi,et al.  Kernel Flows: from learning kernels from data into the abyss , 2018, J. Comput. Phys..

[127]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[128]  Prabhu Ramachandran,et al.  SPINN: Sparse, Physics-based, and partially Interpretable Neural Networks for PDEs , 2021, J. Comput. Phys..

[129]  Risi Kondor,et al.  On the Generalization of Equivariance and Convolution in Neural Networks to the Action of Compact Groups , 2018, ICML.

[130]  Luning Sun,et al.  PhyGeoNet: Physics-informed geometry-adaptive convolutional neural networks for solving parameterized steady-state PDEs on irregular domain , 2021, J. Comput. Phys..

[131]  Dong Eui Chang,et al.  Generic Representation of Neural Networks , 2018 .

[132]  Zhiwei Fang,et al.  A Physics-Informed Neural Network Framework for PDEs on 3D Surfaces: Time Independent Problems , 2020, IEEE Access.