A Multiscale Neural Network Based on Hierarchical Matrices

In this work we introduce a new multiscale artificial neural network based on the structure of $\mathcal{H}$-matrices. This network generalizes the latter to the nonlinear case by introducing a local deep neural network at each spatial scale. Numerical results indicate that the network is able to efficiently approximate discrete nonlinear maps obtained from discretized nonlinear partial differential equations, such as those arising from nonlinear Schr\"odinger equations and the Kohn-Sham density functional theory.

[1]  W. Hackbusch,et al.  An introduction to hierarchical matrices , 2001 .

[2]  D. Brandt,et al.  Multi-level adaptive solutions to boundary-value problems math comptr , 1977 .

[3]  Kaj Nyström,et al.  A unified deep artificial neural network approach to partial differential equations in complex geometries , 2017, Neurocomputing.

[4]  Stéphane Mallat,et al.  Invariant Scattering Convolution Networks , 2012, IEEE transactions on pattern analysis and machine intelligence.

[5]  Wolfgang Ketterle,et al.  Bose–Einstein condensation of atomic gases , 2002, Nature.

[6]  Leslie Greengard,et al.  A fast algorithm for particle simulations , 1987 .

[7]  George E. Karniadakis,et al.  Hidden physics models: Machine learning of nonlinear partial differential equations , 2017, J. Comput. Phys..

[8]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[9]  Ivan Oseledets,et al.  Expressive power of recurrent neural networks , 2017, ICLR.

[10]  Nadav Cohen,et al.  On the Expressive Power of Deep Learning: A Tensor Analysis , 2015, COLT 2016.

[11]  Tara N. Sainath,et al.  FUNDAMENTAL TECHNOLOGIES IN MODERN SPEECH RECOGNITION Digital Object Identifier 10.1109/MSP.2012.2205597 , 2012 .

[12]  Joan Bruna.  Scattering Representations for Recognition , 2013 .

[13]  N. Nguyen,et al.  An ‘empirical interpolation’ method: application to efficient reduced-basis discretization of partial differential equations , 2004 .

[14]  Yvon Maday,et al.  Convergence analysis of the Generalized Empirical Interpolation Method , 2016, SIAM J. Numer. Anal..

[15]  Amir Adler,et al.  Deep-learning tomography , 2018 .

[16]  Tomaso Poggio,et al.  Learning Functions: When Is Deep Better Than Shallow , 2016, 1603.00988.

[17]  N. Kishore Kumar,et al.  Literature survey on low rank approximation of matrices , 2016, ArXiv.

[18]  Wolfgang Hackbusch,et al.  A Sparse Matrix Arithmetic Based on H-Matrices. Part I: Introduction to H-Matrices , 1999, Computing.

[19]  Lexing Ying,et al.  A multiscale neural network based on hierarchical nested bases , 2018, Research in the Mathematical Sciences.

[20]  Linfeng Zhang,et al.  Adaptive coupling of a deep neural network potential to a classical force field. , 2018, The Journal of chemical physics.

[21]  E Weinan,et al.  Deep Learning-Based Numerical Methods for High-Dimensional Parabolic Partial Differential Equations and Backward Stochastic Differential Equations , 2017, Communications in Mathematics and Statistics.

[22]  Lexing Ying,et al.  Fast construction of hierarchical matrix representation from matrix-vector multiplication , 2009, J. Comput. Phys..

[23]  Lei Zhang,et al.  Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising , 2016, IEEE Transactions on Image Processing.

[24]  Justin A. Sirignano,et al.  DGM: A deep learning algorithm for solving partial differential equations , 2017, J. Comput. Phys..

[25]  Y. Efendiev,et al.  Generalized Multiscale Finite Element Methods. Nonlinear Elliptic Equations , 2013, 1304.5188.

[26]  Qiang Du,et al.  Computing the Ground State Solution of Bose-Einstein Condensates by a Normalized Gradient Flow , 2003, SIAM J. Sci. Comput..

[27]  Robert P. Sheridan,et al.  Deep Neural Nets as a Method for Quantitative Structure-Activity Relationships , 2015, J. Chem. Inf. Model..

[28]  Daniël M Pelt,et al.  A mixed-scale dense convolutional neural network for image analysis , 2017, Proceedings of the National Academy of Sciences.

[29]  Iasonas Kokkinos,et al.  DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[30]  E Weinan,et al.  Machine Learning Approximation Algorithms for High-Dimensional Fully Nonlinear Partial Differential Equations and Second-order Backward Stochastic Differential Equations , 2017, J. Nonlinear Sci..

[31]  J. Schneider,et al.  Literature survey on low rank approximation of matrices , 2017 .

[32]  Stéphane Mallat,et al.  A Wavelet Tour of Signal Processing - The Sparse Way, 3rd Edition , 2008 .

[33]  Andrea Vedaldi,et al.  Deep Image Prior , 2017, International Journal of Computer Vision.

[34]  N. Giokaris,et al.  Tomographic image reconstruction using Artificial Neural Networks , 2004 .

[35]  N. Nguyen,et al.  EFFICIENT REDUCED-BASIS TREATMENT OF NONAFFINE AND NONLINEAR PARTIAL DIFFERENTIAL EQUATIONS , 2007 .

[36]  Gabriele Steidl,et al.  FMM and H-matrices: A Short Introduction to the Basic Idea , 2002 .

[37]  G. G. Stokes.  "J." , 1890, The New Yale Book of Quotations.

[38]  Maziar Raissi,et al.  Forward-Backward Stochastic Neural Networks: Deep Learning of High-dimensional Partial Differential Equations , 2018, ArXiv.

[39]  Quoc V. Le,et al.  Sequence to Sequence Learning with Neural Networks , 2014, NIPS.

[40]  W. Hackbusch,et al.  A sparse H -matrix arithmetic: general complexity estimates , 2000 .

[41]  Lexing Ying,et al.  Solving parametric PDE problems with artificial neural networks , 2017, European Journal of Applied Mathematics.

[42]  Linfeng Zhang,et al.  DeePCG: Constructing coarse-grained models via deep neural networks. , 2018, The Journal of chemical physics.

[43]  Stefano Soatto,et al.  Partial differential equations for training deep neural networks , 2017, 2017 51st Asilomar Conference on Signals, Systems, and Computers.

[44]  P. Cochat,et al.  Et al , 2008, Archives de pediatrie : organe officiel de la Societe francaise de pediatrie.

[45]  Yuan Yu,et al.  TensorFlow: A system for large-scale machine learning , 2016, OSDI.

[46]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[47]  Silvia Ferrari,et al.  A Constrained Backpropagation Approach for the Adaptive Solution of Partial Differential Equations , 2014, IEEE Transactions on Neural Networks and Learning Systems.

[48]  Jürgen Schmidhuber,et al.  Deep learning in neural networks: An overview , 2014, Neural Networks.

[49]  Bram van Ginneken,et al.  A survey on deep learning in medical image analysis , 2017, Medical Image Anal..

[50]  Tsuyoshi Murata,et al.  {m , 1934, ACML.

[51]  Yalchin Efendiev,et al.  Deep Multiscale Model Learning , 2018, J. Comput. Phys..

[52]  Yann LeCun,et al.  Deep multi-scale video prediction beyond mean square error , 2015, ICLR.

[53]  Brendan J. Frey,et al.  Deep learning of the tissue-regulated splicing code , 2014, Bioinform..

[54]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[55]  W. Kohn,et al.  Self-Consistent Equations Including Exchange and Correlation Effects , 1965 .

[56]  Yingzhou Li,et al.  Butterfly-Net: Optimal Function Representation Based on Convolutional Neural Networks , 2018, Communications in Computational Physics.

[57]  L. Trefethen.  Spectral Methods in MATLAB , 2000 .

[58]  Kurt Hornik,et al.  Approximation capabilities of multilayer feedforward networks , 1991, Neural Networks.

[59]  Yalchin Efendiev,et al.  Generalized multiscale finite element methods (GMsFEM) , 2013, J. Comput. Phys..

[60]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[61]  Ahmed H. Elsheikh,et al.  A machine learning approach for efficient uncertainty quantification using multiscale methods , 2017, J. Comput. Phys..

[62]  Timothy Dozat,et al.  Incorporating Nesterov Momentum into Adam , 2016 .

[63]  W. Hackbusch,et al.  Introduction to Hierarchical Matrices with Applications , 2003 .

[64]  P. Hohenberg,et al.  Inhomogeneous Electron Gas , 1964 .

[65]  Yalchin Efendiev,et al.  Multiscale finite element methods for porous media flows and their applications , 2007 .

[66]  Roberto Cipolla,et al.  SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[67]  Jian Sun,et al.  Convolutional neural networks at constrained time cost , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[68]  Samy Bengio,et al.  Understanding deep learning requires rethinking generalization , 2016, ICLR.