SMAI-JCM SMAI Journal of Computational Mathematics Model Reduction And Neural Networks For Parametric PDEs

We develop a general framework for data-driven approximation of input-output maps between infinite-dimensional spaces. The proposed approach is motivated by the recent successes of neural networks and deep learning, in combination with ideas from model reduction. This combination results in a neural network approximation which, in principle, is defined on infinite-dimensional spaces and, in practice, is robust to the dimension of finite-dimensional approximations of these spaces required for computation. For a class of input-output maps, and suitably chosen probability measures on the inputs, we prove convergence of the proposed approximation methodology. Numerically we demonstrate the effectiveness of the method on a class of parametric elliptic PDE problems, showing convergence and robustness of the approximation scheme with respect to the size of the discretization, and compare our method with existing algorithms from the literature.

[1]  Ioannis G. Kevrekidis,et al.  Identification of distributed parameter systems: A neural net based approach , 1998 .

[2]  Nello Cristianini,et al.  On the eigenspectrum of the gram matrix and the generalization error of kernel-PCA , 2005, IEEE Transactions on Information Theory.

[3]  George E. Karniadakis,et al.  DeepM&Mnet for hypersonics: Predicting the coupled flow and finite-rate chemistry behind a normal shock using neural-network approximation of operators , 2020, J. Comput. Phys..

[4]  J. Hesthaven,et al.  Non-intrusive reduced order modeling of nonlinear problems using neural networks , 2018, J. Comput. Phys..

[5]  E Weinan,et al.  The Deep Ritz Method: A Deep Learning-Based Numerical Algorithm for Solving Variational Problems , 2017, Communications in Mathematics and Statistics.

[6]  George Em Karniadakis,et al.  DeepM&Mnet: Inferring the electroconvection multiphysics fields based on operator approximation by neural networks , 2021, J. Comput. Phys..

[7]  Andrea Manzoni,et al.  A Comprehensive Deep Learning-Based Approach to Reduced Order Modeling of Nonlinear Time-Dependent Parametrized PDEs , 2020, Journal of Scientific Computing.

[8]  Y. Maday,et al.  Reduced Basis Techniques for Stochastic Problems , 2010, 1004.0357.

[9]  Kevin P. Murphy,et al.  Machine learning - a probabilistic perspective , 2012, Adaptive computation and machine learning series.

[10]  K. Fan.  On a Theorem of Weyl Concerning Eigenvalues of Linear Transformations: II. , 1949, Proceedings of the National Academy of Sciences of the United States of America.

[11]  P. Stern,et al.  Automatic choice of global shape functions in structural analysis , 1978 .

[12]  Bernhard Schölkopf,et al.  Nonlinear Component Analysis as a Kernel Eigenvalue Problem , 1998, Neural Computation.

[13]  Kamyar Azizzadenesheli,et al.  Neural Operator: Graph Kernel Network for Partial Differential Equations , 2020, ICLR 2020.

[14]  A. Quarteroni,et al.  Reduced Basis Methods for Partial Differential Equations: An Introduction , 2015 .

[15]  Mikhail Belkin,et al.  Laplacian Eigenmaps for Dimensionality Reduction and Data Representation , 2003, Neural Computation.

[16]  Paris Perdikaris,et al.  Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations , 2019, J. Comput. Phys..

[17]  Colin J. Cotter,et al.  Probabilistic Forecasting and Bayesian Data Assimilation , 2015 .

[18]  George Em Karniadakis,et al.  Learning nonlinear operators via DeepONet based on the universal approximation theorem of operators , 2019, Nature Machine Intelligence.

[19]  S. Saitoh,et al.  Refined generalizations of the triangle inequality on Banach spaces , 2010 .

[20]  Cheng Huang,et al.  Data-driven reduced-order models via regularised Operator Inference for a single-injector combustion process , 2021, Journal of the Royal Society of New Zealand.

[21]  L. Evans,et al.  Partial Differential Equations , 1941 .

[22]  Wolfgang Dahmen,et al.  State Estimation - The Role of Reduced Models , 2020, ArXiv.

[23]  Gilles Blanchard,et al.  Statistical properties of Kernel Prinicipal Component Analysis , 2019 .

[24]  Catherine E. Powell,et al.  Preface to "An Introduction to Computational Stochastic PDEs" , 2014 .

[25]  A. Stuart,et al.  Data Assimilation: A Mathematical Introduction , 2015, 1506.07825.

[26]  Michael L. Overton,et al.  On the Sum of the Largest Eigenvalues of a Symmetric Matrix , 1992, SIAM J. Matrix Anal. Appl..

[27]  Gilles Blanchard,et al.  Statistical properties of kernel principal component analysis , 2007, Machine Learning.

[28]  Catherine E. Powell,et al.  An Introduction to Computational Stochastic PDEs , 2014 .

[29]  Albert Cohen,et al.  Convergence Rates of Best N-term Galerkin Approximations for a Class of Elliptic sPDEs , 2010, Found. Comput. Math..

[30]  Allan Pinkus,et al.  Lower bounds for approximation by MLP neural networks , 1999, Neurocomputing.

[31]  Benjamin Peherstorfer,et al.  Lift & Learn: Physics-informed machine learning for large-scale nonlinear dynamical systems , 2019, Physica D: Nonlinear Phenomena.

[32]  Samuel S. Schoenholz,et al.  Neural Message Passing for Quantum Chemistry , 2017, ICML.

[33]  Fabian Laakmann,et al.  Efficient approximation of solutions of parametric linear transport equations by ReLU DNNs , 2020, Advances in Computational Mathematics.

[34]  George Em Karniadakis,et al.  DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators , 2019, ArXiv.

[35]  J. Hesthaven,et al.  Certified Reduced Basis Methods for Parametrized Partial Differential Equations , 2015 .

[36]  Geoffrey E. Hinton,et al.  Deep Learning , 2015, Nature.

[37]  Albert Cohen,et al.  Approximation of high-dimensional parametric PDEs * , 2015, Acta Numerica.

[38]  R. DeVore,et al.  ANALYTIC REGULARITY AND POLYNOMIAL APPROXIMATION OF PARAMETRIC AND STOCHASTIC ELLIPTIC PDE'S , 2011 .

[39]  Jan S. Hesthaven,et al.  Non-intrusive reduced order modeling of nonlinear problems using neural networks , 2018, J. Comput. Phys..

[40]  Benjamin Peherstorfer,et al.  Operator inference for non-intrusive model reduction of systems with non-polynomial nonlinear terms , 2020, Computer Methods in Applied Mechanics and Engineering.

[41]  Stefano Ermon,et al.  Learning Neural PDE Solvers with Convergence Guarantees , 2019, ICLR.

[42]  Ioannis G. Kevrekidis,et al.  Model identification of a spatiotemporally varying catalytic reaction , 1993 .

[43]  Eberhard Zeidler,et al.  Applied Functional Analysis: Applications to Mathematical Physics , 1995 .

[44]  Francisco J. Gonzalez,et al.  Deep convolutional recurrent autoencoders for learning low-dimensional feature dynamics of fluid systems , 2018, ArXiv.

[45]  Hong Chen,et al.  Universal approximation to nonlinear operators by neural networks with arbitrary activation functions and its application to dynamical systems , 1995, IEEE Trans. Neural Networks.

[46]  Nello Cristianini,et al.  On the Eigenspectrum of the Gram Matrix and Its Relationship to the Operator Eigenspectrum , 2002, ALT.

[47]  Eldad Haber,et al.  Stable architectures for deep neural networks , 2017, ArXiv.

[48]  Kookjin Lee,et al.  Model reduction of dynamical systems on nonlinear manifolds using deep convolutional autoencoders , 2018, J. Comput. Phys..

[49]  Benjamin Peherstorfer,et al.  Sampling low-dimensional Markovian dynamics for pre-asymptotically recovering reduced models from data with operator inference , 2019, SIAM J. Sci. Comput..

[50]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[51]  W. E.  A Proposal on Machine Learning via Dynamical Systems , 2017 .

[52]  D. Nychka.  Data Assimilation” , 2006 .

[53]  Albert Cohen,et al.  Sparse adaptive Taylor approximation algorithms for parametric and stochastic elliptic PDEs , 2011 .

[54]  Ann B. Lee,et al.  Geometric diffusions as a tool for harmonic analysis and structure definition of data: diffusion maps. , 2005, Proceedings of the National Academy of Sciences of the United States of America.

[55]  Karthik Duraisamy,et al.  Prediction of aerodynamic flow fields using convolutional neural networks , 2019, Computational Mechanics.

[56]  Sepp Hochreiter,et al.  Self-Normalizing Neural Networks , 2017, NIPS.

[57]  Jonathan R. Holland,et al.  Field Inversion and Machine Learning With Embedded Neural Networks: Physics-Consistent Neural Network Training , 2019, AIAA Aviation 2019 Forum.

[58]  G. Petrova,et al.  Nonlinear Approximation and (Deep) ReLU\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathrm {ReLU}$$\end{document} , 2019, Constructive Approximation.

[59]  Christoph Schwab,et al.  Deep ReLU Neural Network Expression Rates for Data-to-QoI Maps in Bayesian PDE Inversion , 2020 .

[60]  Qian Wang,et al.  Non-intrusive reduced order modeling of unsteady flows using artificial neural networks with application to a combustion problem , 2019, J. Comput. Phys..

[61]  R. DeVore,et al.  Nonlinear approximation , 1998, Acta Numerica.

[62]  Gitta Kutyniok,et al.  A Theoretical Analysis of Deep Neural Networks and Parametric PDEs , 2019, Constructive Approximation.

[63]  Nicholas Zabaras,et al.  Bayesian Deep Convolutional Encoder-Decoder Networks for Surrogate Modeling and Uncertainty Quantification , 2018, J. Comput. Phys..

[64]  Lexing Ying,et al.  Solving parametric PDE problems with artificial neural networks , 2017, European Journal of Applied Mathematics.

[65]  Marco A. Iglesias,et al.  Well-posed Bayesian geometric inverse problems arising in subsurface flow , 2014, 1401.5571.

[66]  Dimitrios I. Fotiadis,et al.  Artificial neural networks for solving ordinary and partial differential equations , 1997, IEEE Trans. Neural Networks.

[67]  P. Baxendale.  Gaussian Measures on Function Spaces , 1976 .

[68]  Geoffrey E. Hinton,et al.  Reducing the Dimensionality of Data with Neural Networks , 2006, Science.

[69]  Anthony T. Patera,et al.  A parameterized‐background data‐weak approach to variational data assimilation: formulation, analysis, and application to acoustics , 2015 .

[70]  Gitta Kutyniok,et al.  Numerical Solution of the Parametric Diffusion Equation by Deep Neural Networks , 2020, Journal of Scientific Computing.

[71]  Jonas Adler,et al.  Solving ill-posed inverse problems using iterative deep neural networks , 2017, ArXiv.

[72]  Thomas F. Miller,et al.  Regression-clustering for Improved Accuracy and Training Cost with Molecular-Orbital-Based Machine Learning , 2019, Journal of chemical theory and computation.

[73]  Tim Dockhorn,et al.  A Discussion on Solving Partial Differential Equations using Neural Networks , 2019, ArXiv.

[74]  R. Temam.  Infinite Dimensional Dynamical Systems in Mechanics and Physics Springer Verlag , 1993 .

[75]  Dennis A. Nagy,et al.  Modal representation of geometrically nonlinear behavior by the finite element method , 1979 .

[76]  E Weinan,et al.  A Proposal on Machine Learning via Dynamical Systems , 2017, Communications in Mathematics and Statistics.

[77]  Wolfgang Dahmen,et al.  Data Assimilation in Reduced Modeling , 2015, SIAM/ASA J. Uncertain. Quantification.

[78]  Dmitry Yarotsky,et al.  Error bounds for approximations with deep ReLU networks , 2016, Neural Networks.

[79]  Eldad Haber,et al.  Deep Neural Networks Motivated by Partial Differential Equations , 2018, Journal of Mathematical Imaging and Vision.

[80]  Christoph Schwab,et al.  Deep learning in high dimension: Neural network expression rates for generalized polynomial chaos expansions in UQ , 2018, Analysis and Applications.

[81]  A. Stuart,et al.  Besov priors for Bayesian inverse problems , 2011, 1105.0889.

[82]  George Em Karniadakis,et al.  On the Convergence and generalization of Physics Informed Neural Networks , 2020, ArXiv.

[83]  Karl Pearson F.R.S..  LIII. On lines and planes of closest fit to systems of points in space , 1901 .

[84]  K. Willcox,et al.  Data-driven operator inference for nonintrusive projection-based model reduction , 2016 .

[85]  N. Nguyen,et al.  An ‘empirical interpolation’ method: application to efficient reduced-basis discretization of partial differential equations , 2004 .

[86]  R. DeVore,et al.  Analytic regularity and polynomial approximation of parametric and stochastic elliptic PDEs , 2010 .

[87]  Kamyar Azizzadenesheli,et al.  EikoNet: Solving the Eikonal Equation With Deep Neural Networks , 2021, IEEE Transactions on Geoscience and Remote Sensing.

[88]  Ronald DeVore,et al.  The Theoretical Foundation of Reduced Basis Methods , 2014 .