A unified deep artificial neural network approach to partial differential equations in complex geometries

Abstract In this paper, we use deep feedforward artificial neural networks to approximate solutions to partial differential equations in complex geometries. We show how to modify the backpropagation algorithm to compute the partial derivatives of the network output with respect to the space variables which is needed to approximate the differential operator. The method is based on an ansatz for the solution which requires nothing but feedforward neural networks and an unconstrained gradient based optimization method such as gradient descent or a quasi-Newton method. We show an example where classical mesh based methods cannot be used and neural networks can be seen as an attractive alternative. Finally, we highlight the benefits of deep compared to shallow neural networks and device some other convergence enhancing techniques.

[1]  Kurt Hornik,et al.  Multilayer feedforward networks are universal approximators , 1989, Neural Networks.

[2]  I. Sloan,et al.  Low discrepancy sequences in high dimensions: How well are their projections distributed? , 2008 .

[3]  Frank Cuypers.  Tools for Computational Finance , 2003 .

[4]  Quoc V. Le,et al.  Measuring Invariances in Deep Networks , 2009, NIPS.

[5]  Silvia Ferrari,et al.  A constrained integration (CINT) approach to solving partial differential equations using artificial neural networks , 2015, Neurocomputing.

[6]  Jon Louis Bentley,et al.  Multidimensional binary search trees used for associative searching , 1975, CACM.

[7]  Eric Jones,et al.  SciPy: Open Source Scientific Tools for Python , 2001 .

[8]  Xin Li,et al.  Simultaneous approximations of multivariate functions and their derivatives by neural networks with one hidden layer , 1996, Neurocomputing.

[9]  Jian Cheng,et al.  Computational Investigation of Low-Discrepancy Sequences in Simulation Algorithms for Bayesian Networks , 2000, UAI.

[10]  I. Sobol.  On the distribution of points in a cube and the approximate evaluation of integrals , 1967 .

[11]  Jukka Saranen,et al.  Streamline diffusion methods for the incompressible Euler and Navier-Stokes equations , 1986 .

[12]  Frances Y. Kuo,et al.  Constructing Sobol Sequences with Better Two-Dimensional Projections , 2008, SIAM J. Sci. Comput..

[13]  E Weinan,et al.  Overcoming the curse of dimensionality: Solving high-dimensional partial differential equations using deep learning , 2017, ArXiv.

[14]  Gunnar Rätsch,et al.  An introduction to kernel-based learning algorithms , 2001, IEEE Trans. Neural Networks.

[15]  Anders Logg,et al.  Automated Solution of Differential Equations by the Finite Element Method: The FEniCS Book , 2012 .

[16]  Manoj Kumar,et al.  Multilayer perceptrons and radial basis function neural network methods for the solution of differential equations: A survey , 2011, Comput. Math. Appl..

[17]  Justin A. Sirignano,et al.  DGM: A deep learning algorithm for solving partial differential equations , 2017, J. Comput. Phys..

[18]  Silvia Ferrari,et al.  A Constrained Backpropagation Approach for the Adaptive Solution of Partial Differential Equations , 2014, IEEE Transactions on Neural Networks and Learning Systems.

[19]  Frances Y. Kuo,et al.  Remark on algorithm 659: Implementing Sobol's quasirandom sequence generator , 2003, TOMS.

[20]  Klaus-Robert Müller,et al.  Kernel Analysis of Deep Networks , 2011, J. Mach. Learn. Res..

[21]  Luca Antiga,et al.  Automatic differentiation in PyTorch , 2017 .

[22]  H. Kreiss.  Initial boundary value problems for hyperbolic systems , 1970 .

[23]  R. Fletcher.  Practical Methods of Optimization , 1988 .

[24]  Manoj Kumar,et al.  An Introduction to Neural Network Methods for Differential Equations , 2015 .

[25]  Yoshua Bengio,et al.  Understanding the difficulty of training deep feedforward neural networks , 2010, AISTATS.

[26]  Stefano Soatto,et al.  Partial differential equations for training deep neural networks , 2017, 2017 51st Asilomar Conference on Signals, Systems, and Computers.

[27]  Dimitris G. Papageorgiou,et al.  Neural-network methods for boundary value problems with irregular boundaries , 2000, IEEE Trans. Neural Networks Learn. Syst..

[28]  Yee Whye Teh,et al.  A Fast Learning Algorithm for Deep Belief Nets , 2006, Neural Computation.

[29]  Kurt Hornik,et al.  Universal approximation of an unknown mapping and its derivatives using multilayer feedforward networks , 1990, Neural Networks.

[30]  Stephen M. Omohundro,et al.  Five Balltree Construction Algorithms , 2009 .

[31]  E Weinan,et al.  Deep Learning-Based Numerical Methods for High-Dimensional Parabolic Partial Differential Equations and Backward Stochastic Differential Equations , 2017, Communications in Mathematics and Statistics.

[32]  Neil E. Cotter,et al.  The Stone-Weierstrass theorem and its application to neural networks , 1990, IEEE Trans. Neural Networks.

[33]  Stefano Soatto,et al.  Deep relaxation: partial differential equations for optimizing deep neural networks , 2017, Research in the Mathematical Sciences.

[34]  Kevin Stanley McFall,et al.  Artificial Neural Network Method for Solution of Boundary Value Problems With Exact Satisfaction of Arbitrary Boundary Conditions , 2009, IEEE Transactions on Neural Networks.

[35]  E Weinan,et al.  The Deep Ritz Method: A Deep Learning-Based Numerical Algorithm for Solving Variational Problems , 2017, Communications in Mathematics and Statistics.

[36]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[37]  Paul Bratley,et al.  Algorithm 659: Implementing Sobol's quasirandom sequence generator , 1988, TOMS.

[38]  Sepp Hochreiter,et al.  Self-Normalizing Neural Networks , 2017, NIPS.

[39]  Dimitrios I. Fotiadis,et al.  Artificial neural networks for solving ordinary and partial differential equations , 1997, IEEE Trans. Neural Networks.

[40]  W. Pitts,et al.  A Logical Calculus of the Ideas Immanent in Nervous Activity (1943) , 2021, Ideas That Created the Future.

[41]  Arnulf Jentzen,et al.  Solving high-dimensional partial differential equations using deep learning , 2017, Proceedings of the National Academy of Sciences.