Numerical solution for high-dimensional partial differential equations based on deep learning with residual learning and data-driven learning

Solving high-dimensional partial differential equations (PDEs) is a long-term computational challenge due to the fundamental obstacle known as the curse of dimensionality. This paper develops a novel method ( DL4HPDE ) based on residual neural network learning with data-driven learning elliptic PDEs on a box-shaped domain. However, to combine a strong mechanism with a weak mechanism, we reconstruct a trial solution to the equations in two parts: the first part satisfies the initial and boundary conditions, while the second part is the residual neural network algorithm, which is used to train the other part. In our proposed method, residual learning is adopted to make our model easier to optimize. Moreover, we propose a data-driven algorithm that can increase the training spatial points according to the regional error and improve the accuracy of the model. Finally, the numerical experiments show the efficiency of our proposed model.

[1]  Zhou Tao,et al.  Forecasting stock index with multi-objective optimization model based on optimized neural network architecture avoiding overfitting , 2018, Comput. Sci. Inf. Syst..

[2]  Keke Wu,et al.  A Comparison Study of Deep Galerkin Method and Deep Ritz Method for Elliptic Problems with Different Boundary Conditions , 2020 .

[3]  Jie Zhang,et al.  Residual compensation extreme learning machine for regression , 2018, Neurocomputing.

[4]  Dimitrios I. Fotiadis,et al.  Artificial neural networks for solving ordinary and partial differential equations , 1997, IEEE Trans. Neural Networks.

[5]  Gang Bao,et al.  Weak Adversarial Networks for High-dimensional Partial Differential Equations , 2019, J. Comput. Phys..

[6]  Arnulf Jentzen,et al.  Analysis of the generalization error: Empirical risk minimization over deep artificial neural networks overcomes the curse of dimensionality in the numerical approximation of Black-Scholes partial differential equations , 2018, SIAM J. Math. Data Sci..

[7]  Samy Bengio,et al.  Understanding deep learning requires rethinking generalization , 2016, ICLR.

[8]  Tao Zhou,et al.  Numerical solution for ruin probability of continuous time model based on neural network algorithm , 2019, Neurocomputing.

[9]  Christopher M. Bishop,et al.  Neural networks for pattern recognition , 1995 .

[10]  R. Bellman.  Dynamic programming. , 1957, Science.

[11]  Feng Han,et al.  Solving Partial Differential Equation Based on Bernstein Neural Network and Extreme Learning Machine Algorithm , 2018, Neural Processing Letters.

[12]  Anastasia Borovykh,et al.  Optimally weighted loss functions for solving PDEs with Neural Networks , 2020, ArXiv.

[13]  Kailiang Wu,et al.  Data-Driven Deep Learning of Partial Differential Equations in Modal Space , 2020, J. Comput. Phys..

[14]  Dumitru Erhan,et al.  Going deeper with convolutions , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[15]  Snehashish Chakraverty,et al.  Single Layer Chebyshev Neural Network Model for Solving Elliptic Partial Differential Equations , 2016, Neural Processing Letters.

[16]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[17]  Arnulf Jentzen,et al.  Numerical simulations for full history recursive multilevel Picard approximations for systems of high-dimensional partial differential equations , 2020, ArXiv.

[18]  Jianshu Luo,et al.  Neural network method for lossless two-conductor transmission line equations based on the IELM algorithm , 2018, AIP Advances.

[19]  Muzhou Hou,et al.  A Fast Implicit Finite Difference Method for Fractional Advection-Dispersion Equations with Fractional Derivative Boundary Conditions , 2017 .

[20]  Xiaohan Zhang.  Actor-Critic Algorithm for High-dimensional Partial Differential Equations , 2020, ArXiv.

[21]  Snehashish Chakraverty,et al.  Applied Soft Computing , 2016 .

[22]  Xiaoqun Cao,et al.  Solving Partial Differential Equations Using Deep Learning and Physical Constraints , 2020, Applied Sciences.

[23]  Richard Bellman,et al.  Adaptive Control Processes: A Guided Tour , 1961, The Mathematical Gazette.

[24]  Paris Perdikaris,et al.  Physics Informed Deep Learning (Part II): Data-driven Discovery of Nonlinear Partial Differential Equations , 2017, ArXiv.

[25]  Vladimir Vapnik,et al.  Statistical learning theory , 1998 .

[26]  Qi Zhang,et al.  Multilevel sparse grids collocation for linear partial differential equations, with tensor product smooth basis functions , 2017, Comput. Math. Appl..

[27]  Yu Meng,et al.  An Effective CNN Method for Fully Automated Segmenting Subcutaneous and Visceral Adipose Tissue on CT Scans , 2019, Annals of Biomedical Engineering.

[28]  Justin A. Sirignano,et al.  DGM: A deep learning algorithm for solving partial differential equations , 2017, J. Comput. Phys..

[29]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[30]  Xiaowei Liu,et al.  Automatically discriminating and localizing COVID-19 from community-acquired pneumonia on chest X-rays , 2020, Pattern Recognition.

[31]  William N. Venables,et al.  Modern Applied Statistics with S-Plus. , 1996 .

[32]  Maziar Raissi,et al.  Deep Hidden Physics Models: Deep Learning of Nonlinear Partial Differential Equations , 2018, J. Mach. Learn. Res..

[33]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[34]  Arnulf Jentzen,et al.  Space-time deep neural network approximations for high-dimensional partial differential equations , 2020, ArXiv.

[35]  Bolei Zhou,et al.  Learning Deep Features for Discriminative Localization , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[36]  Haizhao Yang,et al.  SelectNet: Self-paced Learning for High-dimensional Partial Differential Equations , 2020, J. Comput. Phys..

[37]  Brian D. Ripley,et al.  Pattern Recognition and Neural Networks , 1996 .

[38]  Lawrence D. Jackel,et al.  Backpropagation Applied to Handwritten Zip Code Recognition , 1989, Neural Computation.

[39]  Muzhou Hou,et al.  A novel improved extreme learning machine algorithm in solving ordinary differential equations by Legendre neural network methods , 2018, Advances in Difference Equations.

[40]  Wei Guo,et al.  Sparse grid discontinuous Galerkin methods for high-dimensional elliptic equations , 2015, J. Comput. Phys..

[41]  Geoffrey E. Hinton,et al.  Dynamic Routing Between Capsules , 2017, NIPS.

[42]  F. An,et al.  A Study on the Efficient Method for the Solution of the Saha Equation in the Numerical Simulation of Capillary Discharge , 2017 .

[43]  Geoffrey E. Hinton,et al.  Deep Learning , 2015, Nature.

[44]  Allan Pinkus,et al.  Approximation theory of the MLP model in neural networks , 1999, Acta Numerica.

[45]  Geoffrey E. Hinton,et al.  Rectified Linear Units Improve Restricted Boltzmann Machines , 2010, ICML.

[46]  Demis Hassabis,et al.  Mastering the game of Go with deep neural networks and tree search , 2016, Nature.

[47]  Tara N. Sainath,et al.  FUNDAMENTAL TECHNOLOGIES IN MODERN SPEECH RECOGNITION Digital Object Identifier 10.1109/MSP.2012.2205597 , 2012 .

[48]  Paris Perdikaris,et al.  Physics-Constrained Deep Learning for High-dimensional Surrogate Modeling and Uncertainty Quantification without Labeled Data , 2019, J. Comput. Phys..

[49]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[50]  Arnulf Jentzen,et al.  Solving high-dimensional partial differential equations using deep learning , 2017, Proceedings of the National Academy of Sciences.

[51]  M. Nica,et al.  A Derivative-Free Method for Solving Elliptic Partial Differential Equations with Deep Neural Networks , 2020, J. Comput. Phys..

[52]  Maziar Raissi,et al.  Forward-Backward Stochastic Neural Networks: Deep Learning of High-dimensional Partial Differential Equations , 2018, ArXiv.

[53]  R. Bellman.  Dynamic Programming , 1957, Science.