Neural Operator: Graph Kernel Network for Partial Differential Equations

The classical development of neural networks has been primarily for mappings between a finite-dimensional Euclidean space and a set of classes, or between two finite-dimensional Euclidean spaces. The purpose of this work is to generalize neural networks so that they can learn mappings between infinite-dimensional spaces (operators). The key innovation in our work is that a single set of network parameters, within a carefully designed network architecture, may be used to describe mappings between infinite-dimensional spaces and between different finite-dimensional approximations of those spaces. We formulate approximation of the infinite-dimensional mapping by composing nonlinear activation functions and a class of integral operators. The kernel integration is computed by message passing on graph networks. This approach has substantial practical consequences which we will illustrate in the context of mappings between input data to partial differential equations (PDEs) and their solutions. In this context, such learned networks can generalize among different approximation methods for the PDE (such as finite difference or finite element methods) and among approximations corresponding to different underlying levels of resolution and discretization. Experiments confirm that the proposed graph kernel network does have the desired properties and show competitive performance compared to the state of the art solvers.

[1]  E. Nyström.  Über Die Praktische Auflösung von Integralgleichungen mit Anwendungen auf Randwertaufgaben , 1930 .

[2]  D. Gilbarg,et al.  Elliptic Partial Differential Equa-tions of Second Order , 1977 .

[3]  M. Gurtin,et al.  An introduction to continuum mechanics , 1981 .

[4]  Jacob Bear,et al.  Fundamentals of transport phenomena in porous media , 1984 .

[5]  Claes Johnson.  Numerical solution of partial differential equations by the finite element method , 1988 .

[6]  Geoffrey E. Hinton,et al.  Bayesian Learning for Neural Networks , 1995 .

[7]  Christopher K. I. Williams.  Computing with Infinite Networks , 1996, NIPS.

[8]  P. Bassanini,et al.  Elliptic Partial Differential Equations of Second Order , 1997 .

[9]  Vladimir Vapnik,et al.  Statistical learning theory , 1998 .

[10]  Jitendra Malik,et al.  Spectral Partitioning with Indefinite Kernels Using the Nyström Extension , 2002, ECCV.

[11]  Christopher K. I. Williams,et al.  Gaussian Processes for Machine Learning (Adaptive Computation and Machine Learning) , 2005 .

[12]  Nicolas Le Roux,et al.  Continuous Neural Networks , 2007, AISTATS.

[13]  Mikhail Belkin,et al.  On Learning with Integral Operators , 2010, J. Mach. Learn. Res..

[14]  Neil D. Lawrence,et al.  Deep Gaussian Processes , 2012, AISTATS.

[15]  Catherine E. Powell,et al.  An Introduction to Computational Stochastic PDEs , 2014 .

[16]  Albert Cohen,et al.  Approximation of high-dimensional parametric PDEs * , 2015, Acta Numerica.

[17]  Wei Li,et al.  Convolutional Neural Networks for Steady Flow Approximation , 2016, KDD.

[18]  Hari Sundar,et al.  FFT, FMM, or Multigrid? A comparative Study of State-Of-the-Art Poisson Solvers for Uniform and Nonuniform Grids in the Unit Cube , 2014, SIAM J. Sci. Comput..

[19]  Roi Livni,et al.  Learning Infinite-Layer Networks: Beyond the Kernel Trick , 2016, ArXiv.

[20]  William H. Guss.  Deep Function Machines: Generalized Neural Networks for Topological Layer Expression , 2016, ArXiv.

[21]  E Weinan,et al.  The Deep Ritz Method: A Deep Learning-Based Numerical Algorithm for Solving Variational Problems , 2017, Communications in Mathematics and Statistics.

[22]  Jonas Adler,et al.  Solving ill-posed inverse problems using iterative deep neural networks , 2017, ArXiv.

[23]  Max Welling,et al.  Semi-Supervised Classification with Graph Convolutional Networks , 2016, ICLR.

[24]  Jure Leskovec,et al.  Inductive Representation Learning on Large Graphs , 2017, NIPS.

[25]  Ronald A. DeVore,et al.  Chapter 3: The Theoretical Foundation of Reduced Basis Methods , 2017 .

[26]  Samuel S. Schoenholz,et al.  Neural Message Passing for Quantum Chemistry , 2017, ICML.

[27]  Razvan Pascanu,et al.  Relational inductive biases, deep learning, and graph networks , 2018, ArXiv.

[28]  Pietro Liò,et al.  Graph Attention Networks , 2017, ICLR.

[29]  Nicholas Zabaras,et al.  Bayesian Deep Convolutional Encoder-Decoder Networks for Surrogate Modeling and Uncertainty Quantification , 2018, J. Comput. Phys..

[30]  Andrew M. Stuart,et al.  How Deep Are Deep Gaussian Processes? , 2017, J. Mach. Learn. Res..

[31]  Laurence Aitchison,et al.  Deep Convolutional Networks as shallow Gaussian Processes , 2018, ICLR.

[32]  Jan Eric Lenssen,et al.  Fast Graph Representation Learning with PyTorch Geometric , 2019, ArXiv.

[33]  Ryan L. Murphy,et al.  Janossy Pooling: Learning Deep Permutation-Invariant Functions for Variable-Size Inputs , 2018, ICLR.

[34]  Paris Perdikaris,et al.  Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations , 2019, J. Comput. Phys..

[35]  Chi Chen,et al.  Graph Networks as a Universal Machine Learning Framework for Molecules and Crystals , 2018, Chemistry of Materials.

[36]  Leah Bar,et al.  Unsupervised Deep Learning Algorithm for PDE-based Forward and Inverse Problems , 2019, ArXiv.

[37]  Karthik Duraisamy,et al.  Prediction of aerodynamic flow fields using convolutional neural networks , 2019, Computational Mechanics.

[38]  Leslie Pack Kaelbling,et al.  Graph Element Networks: adaptive, structured computation and memory , 2019, ICML.

[39]  Christoph Schwab,et al.  Deep ReLU Neural Network Expression Rates for Data-to-QoI Maps in Bayesian PDE Inversion , 2020 .

[40]  Nikola B. Kovachki,et al.  Model Reduction and Neural Networks for Parametric PDEs , 2020, The SMAI journal of computational mathematics.