A multiscale neural network based on hierarchical nested bases

In recent years, deep learning has led to impressive results in many fields. In this paper, we introduce a multiscale artificial neural network for high-dimensional nonlinear maps based on the idea of hierarchical nested bases in the fast multipole method and the $$\mathcal {H}^2$$H2-matrices. This approach allows us to efficiently approximate discretized nonlinear maps arising from partial differential equations or integral equations. It also naturally extends our recent work based on the generalization of hierarchical matrices (Fan et al. arXiv:1807.01883), but with a reduced number of parameters. In particular, the number of parameters of the neural network grows linearly with the dimension of the parameter space of the discretized PDE. We demonstrate the properties of the architecture by approximating the solution maps of nonlinear Schrödinger equation, the radiative transfer equation and the Kohn–Sham map.

[1]  G. G. Stokes.  "J." , 1890, The New Yale Book of Quotations.

[2]  P. Hohenberg,et al.  Inhomogeneous Electron Gas , 1964 .

[3]  W. Kohn,et al.  Self-Consistent Equations Including Exchange and Correlation Effects , 1965 .

[4]  Leslie Greengard,et al.  A fast algorithm for particle simulations , 1987 .

[5]  Kurt Hornik,et al.  Approximation capabilities of multilayer feedforward networks , 1991, Neural Networks.

[6]  E. Tyrtyshnikov.  Mosaic-Skeleton approximations , 1996 .

[7]  W. Hackbusch.  A Sparse Matrix Arithmetic Based on $\Cal H$-Matrices. Part I: Introduction to ${\Cal H}$-Matrices , 1999, Computing.

[8]  Wolfgang Hackbusch,et al.  A Sparse Matrix Arithmetic Based on H-Matrices. Part I: Introduction to H-Matrices , 1999, Computing.

[9]  W. Hackbusch,et al.  On H2-Matrices , 2000 .

[10]  W. Hackbusch,et al.  A sparse H -matrix arithmetic: general complexity estimates , 2000 .

[11]  L. Trefethen.  Spectral Methods in MATLAB , 2000 .

[12]  Wolfgang Ketterle,et al.  Bose–Einstein condensation of atomic gases , 2002, Nature.

[13]  A. Klose,et al.  Optical tomography using the time-independent equation of radiative transfer-Part 1: Forward model , 2002 .

[14]  W. Hackbusch,et al.  Introduction to Hierarchical Matrices with Applications , 2003 .

[15]  N. Giokaris,et al.  Tomographic image reconstruction using Artificial Neural Networks , 2004 .

[16]  Rainer Koch,et al.  Evaluation of quadrature schemes for the discrete ordinates method , 2004 .

[17]  Qiang Du,et al.  Computing the Ground State Solution of Bose-Einstein Condensates by a Normalized Gradient Flow , 2003, SIAM J. Sci. Comput..

[18]  Anthony B. Davis,et al.  3D Radiative Transfer in Cloudy Atmospheres , 2005 .

[19]  G. C. Pomraning.  The Equations of Radiation Hydrodynamics , 2005 .

[20]  Lexing Ying,et al.  Fast construction of hierarchical matrix representation from matrix-vector multiplication , 2009, J. Comput. Phys..

[21]  P. Cochat,et al.  Et al , 2008, Archives de pediatrie : organe officiel de la Societe francaise de pediatrie.

[22]  Tao Wang,et al.  End-to-end text recognition with convolutional neural networks , 2012, Proceedings of the 21st International Conference on Pattern Recognition (ICPR2012).

[23]  Tara N. Sainath,et al.  Deep Neural Networks for Acoustic Modeling in Speech Recognition: The Shared Views of Four Research Groups , 2012, IEEE Signal Processing Magazine.

[24]  Yoshua Bengio,et al.  Deep Learning for NLP (without Magic) , 2012, ACL.

[25]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[26]  Stéphane Mallat,et al.  Invariant Scattering Convolution Networks , 2012, IEEE transactions on pattern analysis and machine intelligence.

[27]  Rob Fergus,et al.  Visualizing and Understanding Convolutional Networks , 2013, ECCV.

[28]  Geoffrey E. Hinton,et al.  Application of Deep Belief Networks for Natural Language Understanding , 2014, IEEE/ACM Transactions on Audio, Speech, and Language Processing.

[29]  Brendan J. Frey,et al.  Deep learning of the tissue-regulated splicing code , 2014, Bioinform..

[30]  Quoc V. Le,et al.  Sequence to Sequence Learning with Neural Networks , 2014, NIPS.

[31]  Silvia Ferrari,et al.  A Constrained Backpropagation Approach for the Adaptive Solution of Partial Differential Equations , 2014, IEEE Transactions on Neural Networks and Learning Systems.

[32]  Jürgen Schmidhuber,et al.  Deep learning in neural networks: An overview , 2014, Neural Networks.

[33]  Nadav Cohen,et al.  On the Expressive Power of Deep Learning: A Tensor Analysis , 2015, COLT 2016.

[34]  B. Frey,et al.  The human splicing code reveals new insights into the genetic determinants of disease , 2015, Science.

[35]  Robert P. Sheridan,et al.  Deep Neural Nets as a Method for Quantitative Structure-Activity Relationships , 2015, J. Chem. Inf. Model..

[36]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[37]  Geoffrey E. Hinton,et al.  Deep Learning , 2015, Nature.

[38]  Dumitru Erhan,et al.  Going deeper with convolutions , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[39]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[40]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[41]  Yuan Yu,et al.  TensorFlow: A system for large-scale machine learning , 2016, OSDI.

[42]  Demis Hassabis,et al.  Mastering the game of Go with deep neural networks and tree search , 2016, Nature.

[43]  Timothy Dozat,et al.  Incorporating Nesterov Momentum into Adam , 2016 .

[44]  Tomaso Poggio,et al.  Learning Functions: When Is Deep Better Than Shallow , 2016, 1603.00988.

[45]  Bram van Ginneken,et al.  A survey on deep learning in medical image analysis , 2017, Medical Image Anal..

[46]  E Weinan,et al.  Deep Learning-Based Numerical Methods for High-Dimensional Parabolic Partial Differential Equations and Backward Stochastic Differential Equations , 2017, Communications in Mathematics and Statistics.

[47]  Stefano Soatto,et al.  Partial differential equations for training deep neural networks , 2017, 2017 51st Asilomar Conference on Signals, Systems, and Computers.

[48]  Roberto Cipolla,et al.  SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[49]  George E. Karniadakis,et al.  Hidden physics models: Machine learning of nonlinear partial differential equations , 2017, J. Comput. Phys..

[50]  Amir Adler,et al.  Deep-learning tomography , 2018 .

[51]  Ivan Oseledets,et al.  Expressive power of recurrent neural networks , 2017, ICLR.

[52]  Ahmed H. Elsheikh,et al.  A machine learning approach for efficient uncertainty quantification using multiscale methods , 2017, J. Comput. Phys..

[53]  Kaj Nyström,et al.  A unified deep artificial neural network approach to partial differential equations in complex geometries , 2017, Neurocomputing.

[54]  Iasonas Kokkinos,et al.  DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[55]  Justin A. Sirignano,et al.  DGM: A deep learning algorithm for solving partial differential equations , 2017, J. Comput. Phys..

[56]  Andrea Vedaldi,et al.  Deep Image Prior , 2017, International Journal of Computer Vision.

[57]  E Weinan,et al.  Machine Learning Approximation Algorithms for High-Dimensional Fully Nonlinear Partial Differential Equations and Second-order Backward Stochastic Differential Equations , 2017, J. Nonlinear Sci..

[58]  Lexing Ying,et al.  Fast algorithms for integral formulations of steady-state radiative transfer equation , 2018, J. Comput. Phys..

[59]  Rongting Zhang,et al.  A fast algorithm for radiative transport in isotropic media , 2016, J. Comput. Phys..

[60]  Lexing Ying,et al.  A Multiscale Neural Network Based on Hierarchical Matrices , 2018, Multiscale Model. Simul..

[61]  Yalchin Efendiev,et al.  Deep Multiscale Model Learning , 2018, J. Comput. Phys..

[62]  Lexing Ying,et al.  Solving parametric PDE problems with artificial neural networks , 2017, European Journal of Applied Mathematics.

[63]  Yingzhou Li,et al.  Butterfly-Net: Optimal Function Representation Based on Convolutional Neural Networks , 2018, Communications in Computational Physics.

[64]  P. Alam.  ‘W’ , 2021, Composites Engineering.

[65]  P. Alam.  ‘A’ , 2021, Composites Engineering: An A–Z Guide.