Tensor Neural Network and Its Numerical Integration

In this paper, we introduce a type of tensor neural network. For the first time, we propose its numerical integration scheme and prove the computational complexity to be the polynomial scale of the dimension. Based on the tensor product structure, we develop an efficient numerical integration method by using fixed quadrature points for the functions of the tensor neural network. The corresponding machine learning method is also introduced for solving high-dimensional problems. Some numerical examples are also provided to validate the theoretical results and the numerical algorithm.

[1]  Lu Lu,et al.  MIONet: Learning multiple-input operators via tensor product , 2022, SIAM J. Sci. Comput..

[2]  Haizhao Yang,et al.  Structure Probing Neural Network Deflation , 2020, J. Comput. Phys..

[3]  Minxin Chen,et al.  MIM: A deep mixed residual method for solving high-order partial differential equations , 2020, J. Comput. Phys..

[4]  W. E,et al.  Machine Learning and Computational Mathematics , 2020, Communications in Computational Physics.

[5]  Albert Y. Zomaya,et al.  Partial Differential Equations , 2007, Explorations in Numerical Analysis.

[6]  Gang Bao,et al.  Weak Adversarial Networks for High-dimensional Partial Differential Equations , 2019, J. Comput. Phys..

[7]  Jinchao Xu,et al.  Approximation rates for neural networks with general activation functions , 2019, Neural Networks.

[8]  Bo Li,et al.  Better Approximations of High Dimensional Smooth Functions by Deep Neural Networks with Rectified Power Units , 2019, Communications in Computational Physics.

[9]  Stephen J. Wright,et al.  Numerical Optimization , 2018, Fundamental Statistical Inference.

[10]  Tamara G. Kolda,et al.  Generalized Canonical Polyadic Tensor Decomposition , 2018, SIAM Rev..

[11]  E Weinan,et al.  Solving many-electron Schrödinger equation using deep neural networks , 2018, J. Comput. Phys..

[12]  Jinchao Xu,et al.  Relu Deep Neural Networks and Linear Finite Elements , 2018, Journal of Computational Mathematics.

[13]  Paris Perdikaris,et al.  Physics Informed Deep Learning (Part I): Data-driven Solutions of Nonlinear Partial Differential Equations , 2017, ArXiv.

[14]  E Weinan,et al.  The Deep Ritz Method: A Deep Learning-Based Numerical Algorithm for Solving Variational Problems , 2017, Communications in Mathematics and Statistics.

[15]  Justin A. Sirignano,et al.  DGM: A deep learning algorithm for solving partial differential equations , 2017, J. Comput. Phys..

[16]  E Weinan,et al.  Overcoming the curse of dimensionality: Solving high-dimensional partial differential equations using deep learning , 2017, ArXiv.

[17]  Gregory Beylkin,et al.  Randomized Alternating Least Squares for Canonical Tensor Decompositions: Application to A PDE With Random Data , 2015, SIAM J. Sci. Comput..

[18]  Geoffrey E. Hinton,et al.  Deep Learning , 2015, Nature.

[19]  Ivan V. Oseledets,et al.  Fast low‐rank approximations of multidimensional integrals in ion‐atomic collisions modelling , 2015, Numer. Linear Algebra Appl..

[20]  Carl M. Bender,et al.  Coupled Oscillator Systems Having Partial PT Symmetry , 2015, 1503.05725.

[21]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[22]  Sohrab Effati,et al.  Artificial neural network method for solving the Navier–Stokes equations , 2014, Neural Computing and Applications.

[23]  Jie Shen,et al.  Efficient Spectral Sparse Grid Methods and Applications to High-Dimensional Elliptic Equations II. Unbounded Domains , 2012, SIAM J. Sci. Comput..

[24]  Jie Shen,et al.  Efficient Spectral Sparse Grid Methods and Applications to High-Dimensional Elliptic Problems , 2010, SIAM J. Sci. Comput..

[25]  Xavier Glorot,et al.  Understanding the difficulty of training deep feedforward neural networks , 2010, AISTATS.

[26]  Tamara G. Kolda,et al.  Tensor Decompositions and Applications , 2009, SIAM Rev..

[27]  Hiroshi Nakatsuji,et al.  Solving the Schrodinger equation for helium atom and its isoelectronic ions with the free iterative complement interaction (ICI) method. , 2007, The Journal of chemical physics.

[28]  M. Griebel,et al.  Sparse grids for the Schrödinger equation , 2007 .

[29]  Martin J. Mohlenkamp,et al.  Algorithms for Numerical Analysis in High Dimensions , 2005, SIAM J. Sci. Comput..

[30]  Martin J. Mohlenkamp,et al.  Numerical operator calculus in higher dimensions , 2002, Proceedings of the National Academy of Sciences of the United States of America.

[31]  R. Ryan.  Introduction to Tensor Products of Banach Spaces , 2002 .

[32]  Dimitris G. Papageorgiou,et al.  Neural-network methods for boundary value problems with irregular boundaries , 2000, IEEE Trans. Neural Networks Learn. Syst..

[33]  Dimitrios I. Fotiadis,et al.  Artificial neural networks for solving ordinary and partial differential equations , 1997, IEEE Trans. Neural Networks.

[34]  S. W. Ellacott,et al.  Aspects of the numerical analysis of neural networks , 1994, Acta Numerica.

[35]  A. Pinkus,et al.  Original Contribution: Multilayer feedforward networks with a nonpolynomial activation function can approximate any function , 1993 .

[36]  Kurt Hornik,et al.  Universal approximation of an unknown mapping and its derivatives using multilayer feedforward networks , 1990, Neural Networks.

[37]  Kurt Hornik,et al.  Multilayer feedforward networks are universal approximators , 1989, Neural Networks.

[38]  J. Halton.  On the efficiency of certain quasi-random sequences of points in evaluating multi-dimensional integrals , 1960 .

[39]  E. Hylleraas.  Über den Grundzustand des Heliumatoms , 1928 .

[40]  G. Burton.  Sobolev Spaces , 2013 .

[41]  Boris N. Khoromskij,et al.  Mathematik in den Naturwissenschaften Leipzig Tensor-Product Approximation to Operators and Functions in High Dimensions , 2007 .

[42]  L. Evans,et al.  Partial Differential Equations , 2000 .

[43]  S. Knapek.  Hyperbolic Cross Approximation Of Integral Operators With Smooth Kernel , 2000 .

[44]  M. Griebel,et al.  Optimized Tensor-Product Approximation Spaces , 2000 .

[45]  Din' Zung.  The approximation of classes of periodic functions of many variables , 1983 .