Multi-scale Deep Neural Networks for Solving High Dimensional PDEs

In this paper, we propose the idea of radial scaling in frequency domain and activation functions with compact support to produce a multi-scale DNN (MscaleDNN), which will have the multi-scale capability in approximating high frequency and high dimensional functions and speeding up the solution of high dimensional PDEs. Numerical results on high dimensional function fitting and solutions of high dimensional PDEs, using loss functions with either Ritz energy or least squared PDE residuals, have validated the increased power of multi-scale resolution and high frequency capturing of the proposed MscaleDNN.

[1]  Jinchao Xu,et al.  Relu Deep Neural Networks and Linear Finite Elements , 2018, Journal of Computational Mathematics.

[2]  E Weinan,et al.  Deep Learning-Based Numerical Methods for High-Dimensional Parabolic Partial Differential Equations and Backward Stochastic Differential Equations , 2017, Communications in Mathematics and Statistics.

[3]  E Weinan,et al.  Deep Learning Approximation for Stochastic Control Problems , 2016, ArXiv.

[4]  Haizhao Yang,et al.  Int-Deep: A Deep Learning Initialized Iterative Method for Nonlinear Problems , 2020, J. Comput. Phys..

[5]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[6]  Zheng Ma,et al.  Explicitizing an Implicit Bias of the Frequency Principle in Two-layer Neural Networks , 2019, ArXiv.

[7]  E. Weinan,et al.  Deep Potential: a general representation of a many-body potential energy surface , 2017, 1707.01478.

[8]  Yoshua Bengio,et al.  On the Spectral Bias of Neural Networks , 2018, ICML.

[9]  Zheng Ma,et al.  Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networks , 2019, Communications in Computational Physics.

[10]  Grant M. Rotskoff,et al.  Parameters as interacting particles: long time convergence and asymptotic error scaling of neural networks , 2018, NeurIPS.

[11]  Zheng Ma,et al.  Theory of the Frequency Principle for General Deep Neural Networks , 2019, CSIAM Transactions on Applied Mathematics.

[12]  Yoshua Bengio,et al.  Understanding the difficulty of training deep feedforward neural networks , 2010, AISTATS.

[13]  Zhi-Qin John Xu,et al.  Understanding training and generalization in deep learning by Fourier analysis , 2018, ArXiv.

[14]  Xiaoguang Li,et al.  A Phase Shift Deep Neural Network for High Frequency Wave Equations in Inhomogeneous Media , 2019, ArXiv.

[15]  Lexing Ying,et al.  Solving parametric PDE problems with artificial neural networks , 2017, European Journal of Applied Mathematics.

[16]  E Weinan,et al.  The Deep Ritz Method: A Deep Learning-Based Numerical Algorithm for Solving Variational Problems , 2017, Communications in Mathematics and Statistics.

[17]  Ingrid Daubechies,et al.  Ten Lectures on Wavelets , 1992 .

[18]  Xiaoguang Li,et al.  A Phase Shift Deep Neural Network for High Frequency Approximation and Wave Problems , 2020, SIAM J. Sci. Comput..

[19]  Lexing Ying,et al.  A Multiscale Neural Network Based on Hierarchical Matrices , 2018, Multiscale Model. Simul..

[20]  Yoshua Bengio,et al.  On the Spectral Bias of Deep Neural Networks , 2018, ArXiv.

[21]  Arthur Jacot,et al.  Neural tangent kernel: convergence and generalization in neural networks (invited paper) , 2018, NeurIPS.

[22]  Zhi-Qin John Xu,et al.  Training behavior of deep neural network in frequency domain , 2018, ICONIP.