Can Transfer Neuroevolution Tractably Solve Your Differential Equations?

This paper introduces neuroevolution for solving differential equations. The solution is obtained through optimizing a deep neural network whose loss function is defined by the residual terms from the differential equations. Recent studies have focused on learning such physics-informed neural networks through stochastic gradient descent (SGD) variants, yet they face the difficulty of obtaining an accurate solution due to optimization challenges. In the context of solving differential equations, we are faced with the problem of finding globally optimum parameters of the network, instead of being concerned with out-of-sample generalization. SGD, which searches along a single gradient direction, is prone to become trapped in local optima, so it may not be the best approach here. In contrast, neuroevolution carries out a parallel exploration of diverse solutions with the goal of circumventing local optima. It could potentially find more accurate solutions with better optimized neural networks. However, neuroevolution can be slow, raising tractability issues in practice. With that in mind, a novel and computationally efficient transfer neuroevolution algorithm is proposed in this paper. Our method is capable of exploiting relevant experiential priors when solving a new problem, with adaptation to protect against the risk of negative transfer. The algorithm is applied on a variety of differential equations to empirically demonstrate that transfer neuroevolution can indeed achieve better accuracy and faster convergence than SGD. The experimental outcomes thus establish transfer neuroevolution as a noteworthy approach for solving differential equations, one that has never been studied in the past. Our work expands the resource of available algorithms for optimizing physics-informed neural networks.

[1]  D. Korteweg,et al.  XLI. On the change of form of long waves advancing in a rectangular canal, and on a new type of long stationary waves , 1895 .

[2]  R. Mattheij,et al.  Partial Differential Equations: Modeling, Analysis, Computation (Siam Monographs on Mathematical Modeling and Computation) (Saim Models on Mathematical Modeling and Computation) , 2005 .

[3]  Hyuk Lee,et al.  Neural algorithm for solving differential equations , 1990 .

[4]  Andrew J. Meade,et al.  The numerical solution of linear ordinary differential equations by feedforward neural networks , 1994 .

[5]  Andrew J. Meade,et al.  Solution of nonlinear ordinary differential equations by feedforward neural networks , 1994 .

[6]  N. Phan-Thien,et al.  Neural-network-based approximations for solving partial differential equations , 1994 .

[7]  Michael T. Heath,et al.  Scientific Computing: An Introductory Survey , 1996 .

[8]  L. Debnath Nonlinear Partial Differential Equations for Scientists and Engineers , 1997 .

[9]  D. Fotiadis,et al.  Artificial neural network methods in quantum mechanics , 1997, quant-ph/9705029.

[10]  Dimitrios I. Fotiadis,et al.  Artificial neural networks for solving ordinary and partial differential equations , 1997, IEEE Trans. Neural Networks.

[11]  Dimitris G. Papageorgiou,et al.  Neural-network methods for boundary value problems with irregular boundaries , 2000, IEEE Trans. Neural Networks Learn. Syst..

[12]  Nikolaus Hansen,et al.  Completely Derandomized Self-Adaptation in Evolution Strategies , 2001, Evolutionary Computation.

[13]  Risto Miikkulainen,et al.  Evolving Neural Networks through Augmenting Topologies , 2002, Evolutionary Computation.

[14]  Christian Igel,et al.  Neuroevolution for reinforcement learning using evolution strategies , 2003, The 2003 Congress on Evolutionary Computation, 2003. CEC '03..

[15]  S. Sherwin,et al.  Finite Difference, Finite Element and Finite Volume Methods for Partial Differential Equations , 2005 .

[16]  Tom Schaul,et al.  Natural Evolution Strategies , 2008, 2008 IEEE Congress on Evolutionary Computation (IEEE World Congress on Computational Intelligence).

[17]  採編典藏組 Society for Industrial and Applied Mathematics(SIAM) , 2008 .

[18]  Kevin Stanley McFall,et al.  Artificial Neural Network Method for Solution of Boundary Value Problems With Exact Satisfaction of Arbitrary Boundary Conditions , 2009, IEEE Transactions on Neural Networks.

[19]  Tom Schaul,et al.  Exponential natural evolution strategies , 2010, GECCO '10.

[20]  Hiroki Okubo,et al.  Identification of basketball parameters for a simulation model , 2010 .

[21]  Tom Schaul,et al.  High dimensions and heavy tails for natural evolution strategies , 2011, GECCO '11.

[22]  Stéphane Doncieux,et al.  Encouraging Behavioral Diversity in Evolutionary Robotics: An Empirical Study , 2012, Evolutionary Computation.

[23]  Abhishek Gupta Numerical Modelling and Optimization of Non-isothermal, Rigid Tool Liquid Composite Moulding Processes , 2013 .

[24]  E. Süli,et al.  Numerical Solution of Partial Differential Equations , 2014 .

[25]  Silvia Ferrari,et al.  A Constrained Backpropagation Approach for the Adaptive Solution of Partial Differential Equations , 2014, IEEE Transactions on Neural Networks and Learning Systems.

[26]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[27]  Silvia Ferrari,et al.  A constrained integration (CINT) approach to solving partial differential equations using artificial neural networks , 2015, Neurocomputing.

[28]  Marcel Bauer,et al.  Numerical Methods for Partial Differential Equations , 1994 .

[29]  Amiram Moshaiov,et al.  Multi-objective topology and weight evolution of neuro-controllers , 2016, 2016 IEEE Congress on Evolutionary Computation (CEC).

[30]  E Weinan,et al.  The Deep Ritz Method: A Deep Learning-Based Numerical Algorithm for Solving Variational Problems , 2017, Communications in Mathematics and Statistics.

[31]  Chrisantha Fernando,et al.  PathNet: Evolution Channels Gradient Descent in Super Neural Networks , 2017, ArXiv.

[32]  Xi Chen,et al.  Evolution Strategies as a Scalable Alternative to Reinforcement Learning , 2017, ArXiv.

[33]  Kenneth O. Stanley,et al.  On the Relationship Between the OpenAI Evolution Strategy and Stochastic Gradient Descent , 2017, ArXiv.

[34]  Karen Kafadar,et al.  Letter-Value Plots: Boxplots for Large Data , 2017 .

[35]  Barak A. Pearlmutter,et al.  Automatic differentiation in machine learning: a survey , 2015, J. Mach. Learn. Res..

[36]  Michael S. Triantafyllou,et al.  Deep learning of vortex-induced vibrations , 2018, Journal of Fluid Mechanics.

[37]  Liang Feng,et al.  Insights on Transfer Optimization: Because Experience is the Best Teacher , 2018, IEEE Transactions on Emerging Topics in Computational Intelligence.

[38]  Kenneth O. Stanley,et al.  ES is more than just a traditional finite-difference approximator , 2017, GECCO.

[39]  Silja Meyer-Nieberg,et al.  Evolving Artificial Neural Networks for Multi-objective Tasks , 2018, EvoApplications.

[40]  Kaj Nyström,et al.  A unified deep artificial neural network approach to partial differential equations in complex geometries , 2017, Neurocomputing.

[41]  Justin A. Sirignano,et al.  DGM: A deep learning algorithm for solving partial differential equations , 2017, J. Comput. Phys..

[42]  Maziar Raissi,et al.  Deep Hidden Physics Models: Deep Learning of Nonlinear Partial Differential Equations , 2018, J. Mach. Learn. Res..

[43]  Risto Miikkulainen,et al.  Designing neural networks through neuroevolution , 2019, Nat. Mach. Intell..

[44]  Yew-Soon Ong,et al.  AIR5: Five Pillars of Artificial Intelligence Research , 2018, IEEE Transactions on Emerging Topics in Computational Intelligence.

[45]  Liang Li,et al.  Machine Discovery of Partial Differential Equations from Spatiotemporal Data , 2019, ArXiv.

[46]  Naif Alajlan,et al.  Artificial Neural Network Methods for the Solution of Second Order Boundary Value Problems , 2019, Computers, Materials & Continua.

[47]  P. Perdikaris,et al.  Machine learning in cardiovascular flows modeling: Predicting arterial blood pressure from non-invasive 4D flow MRI data using physics-informed neural networks , 2019 .

[48]  Yew-Soon Ong,et al.  Curbing Negative Influences Online for Seamless Transfer Evolutionary Optimization , 2019, IEEE Transactions on Cybernetics.

[49]  Timon Rabczuk,et al.  Transfer learning enhanced physics informed neural network for phase-field modeling of fracture , 2019, Theoretical and Applied Fracture Mechanics.

[50]  Paris Perdikaris,et al.  Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations , 2019, J. Comput. Phys..

[51]  Stefano Nolfi,et al.  Robust optimization through neuroevolution , 2019, PloS one.

[52]  Paris Perdikaris,et al.  Adversarial Uncertainty Quantification in Physics-Informed Neural Networks , 2018, J. Comput. Phys..

[53]  J. Clune,et al.  Scaling MAP-Elites to deep neuroevolution , 2020, GECCO.

[54]  G. Karniadakis,et al.  Physics-informed neural networks for high-speed flows , 2020, Computer Methods in Applied Mechanics and Engineering.

[55]  L. Dal Negro,et al.  Physics-informed neural networks for inverse problems in nano-optics and metamaterials. , 2019, Optics express.

[56]  Huynh Thi Thanh Binh,et al.  Multifactorial evolutionary algorithm for solving clustered tree problems: competition among Cayley codes , 2020, Memetic Computing.

[57]  Nicholas Geneva,et al.  Modeling the Dynamics of PDE Systems with Physics-Constrained Deep Auto-Regressive Networks , 2019, J. Comput. Phys..

[58]  George Em Karniadakis,et al.  Hidden fluid mechanics: Learning velocity and pressure fields from flow visualizations , 2020, Science.

[59]  Haiyan Wang,et al.  Modeling Information Diffusion in Online Social Networks with Partial Differential Equations , 2013, Surveys and Tutorials in the Applied Mathematical Sciences.

[60]  G. Karniadakis,et al.  Conservative physics-informed neural networks on discrete domains for conservation laws: Applications to forward and inverse problems , 2020 .

[61]  Lu Bai,et al.  Multi-task gradient descent for multi-task learning , 2020, Memetic Computing.

[62]  Keju Tang,et al.  D3M: A Deep Domain Decomposition Method for Partial Differential Equations , 2019, IEEE Access.

[63]  George Em Karniadakis,et al.  Adaptive activation functions accelerate convergence in deep and physics-informed neural networks , 2019, J. Comput. Phys..