Solving Differential Equation with Constrained Multilayer Feedforward Network

In this paper, we present a novel framework to solve differential equations based on multilayer feedforward network. Previous works indicate that solvers based on neural network have low accuracy due to that the boundary conditions are not satisfied accurately. The boundary condition is now inserted directly into the model as boundary term, and the model is a combination of a boundary term and a multilayer feedforward network with its weight function. As the boundary condition becomes predefined constraintion in the model itself, the neural network is trained as an unconstrained optimization problem. This leads to both ease of training and high accuracy. Due to universal convergency of multilayer feedforward networks, the new method is a general approach in solving different types of differential equations. Numerical examples solving ODEs and PDEs with Dirichlet boundary condition are presented and discussed.

[1]  Qingdong Cai,et al.  Neural network as a function approximator and its application in solving differential equations , 2019, Applied Mathematics and Mechanics.

[2]  G. Lewicki,et al.  Approximation by Superpositions of a Sigmoidal Function , 2003 .

[3]  Simon Haykin,et al.  Neural Networks and Learning Machines , 2010 .

[4]  A. A. Mullin,et al.  Principles of neurodynamics , 1962 .

[5]  Dimitrios I. Fotiadis,et al.  Artificial neural networks for solving ordinary and partial differential equations , 1997, IEEE Trans. Neural Networks.

[6]  W. Pitts,et al.  A Logical Calculus of the Ideas Immanent in Nervous Activity (1943) , 2021, Ideas That Created the Future.

[7]  Snehashish Chakraverty,et al.  Chebyshev Neural Network based model for solving Lane-Emden type equations , 2014, Appl. Math. Comput..

[8]  Yoshua Bengio,et al.  Gradient Flow in Recurrent Nets: the Difficulty of Learning Long-Term Dependencies , 2001 .

[9]  L. Prandtl,et al.  Zur Berechnung der Grenzschichten , 1938 .

[10]  P. Werbos,et al.  Beyond Regression : "New Tools for Prediction and Analysis in the Behavioral Sciences , 1974 .

[11]  Paris Perdikaris,et al.  Physics Informed Deep Learning (Part II): Data-driven Discovery of Nonlinear Partial Differential Equations , 2017, ArXiv.

[12]  Paris Perdikaris,et al.  Physics Informed Deep Learning (Part I): Data-driven Solutions of Nonlinear Partial Differential Equations , 2017, ArXiv.

[13]  Marvin Minsky,et al.  Perceptrons: An Introduction to Computational Geometry , 1969 .

[14]  Joshua B. Tenenbaum,et al.  Human-level concept learning through probabilistic program induction , 2015, Science.

[15]  Nam Mai-Duy,et al.  Numerical solution of differential equations using multiquadric radial basis function networks , 2001, Neural Networks.

[16]  Louis B. Rall,et al.  Automatic Differentiation: Techniques and Applications , 1981, Lecture Notes in Computer Science.

[17]  Jorge Nocedal,et al.  On the limited memory BFGS method for large scale optimization , 1989, Math. Program..

[18]  Barak A. Pearlmutter,et al.  Automatic differentiation in machine learning: a survey , 2015, J. Mach. Learn. Res..

[19]  L. Jones.  Constructive approximations for neural networks by sigmoidal functions , 1990, Proc. IEEE.

[20]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[21]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[22]  Ying Jiang,et al.  Machine-learning solver for modified diffusion equations , 2018, Physical Review E.

[23]  F ROSENBLATT,et al.  The perceptron: a probabilistic model for information storage and organization in the brain. , 1958, Psychological review.

[24]  Sepp Hochreiter,et al.  Untersuchungen zu dynamischen neuronalen Netzen , 1991 .

[25]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[26]  B. Frey,et al.  Predicting the sequence specificities of DNA- and RNA-binding proteins by deep learning , 2015, Nature Biotechnology.

[27]  Kurt Hornik,et al.  Multilayer feedforward networks are universal approximators , 1989, Neural Networks.

[28]  Kaj Nyström,et al.  A unified deep artificial neural network approach to partial differential equations in complex geometries , 2017, Neurocomputing.

[29]  S. M. Carroll,et al.  Construction of neural nets using the radon transform , 1989, International 1989 Joint Conference on Neural Networks.