Accelerating Simulation of Stiff Nonlinear Systems using Continuous-Time Echo State Networks

Modern design, control, and optimization often requires simulation of highly nonlinear models, leading to prohibitive computational costs. These costs can be amortized by evaluating a cheap surrogate of the full model. Here we present a general data-driven method, the continuous-time echo state network (CTESN), for generating surrogates of nonlinear ordinary differential equations with dynamics at widely separated timescales. We empirically demonstrate near-constant time performance using our CTESNs on a physically motivated scalable model of a heating system whose full execution time increases exponentially, while maintaining relative error of within 0.2 %. We also show that our model captures fast transients as well as slow dynamics effectively, while other techniques such as physics informed neural networks have difficulties trying to train and predict the highly nonlinear behavior of these models.

[1]  Lawrence F. Shampine,et al.  Implementation of Rosenbrock Methods , 1982, TOMS.

[2]  S. Schuster,et al.  A generalization of Wegscheider's condition. Implications for properties of steady states and for quasi-steady-state approximation , 1989 .

[3]  S. Schnell,et al.  Use and abuse of the quasi-steady-state approximation. , 2006, Systems biology.

[4]  Lawrence F. Shampine,et al.  Analysis and implementation of TR-BDF2 , 1996 .

[5]  Herbert Jaeger,et al.  Reservoir computing approaches to recurrent neural network training , 2009, Comput. Sci. Rev..

[6]  Manuel Calvo,et al.  Stiffness 1952–2012: Sixty years in search of a definition , 2015 .

[7]  José Carlos Príncipe,et al.  Analysis and Design of Echo State Networks , 2007, Neural Computation.

[8]  Devika Subramanian,et al.  Data-driven prediction of a multi-scale Lorenz 96 chaotic system using a hierarchy of deep learning methods: Reservoir computing, ANN, and RNN-LSTM , 2019, ArXiv.

[9]  Weiqi Ji,et al.  Stiff-PINN: Physics-Informed Neural Network for Stiff Chemical Kinetics , 2020, AAAI Spring Symposium: MLPS.

[10]  Paris Perdikaris,et al.  Understanding and mitigating gradient pathologies in physics-informed neural networks , 2020, ArXiv.

[11]  Alan Edelman,et al.  Julia: A Fresh Approach to Numerical Computing , 2014, SIAM Rev..

[12]  Tamás Turányi,et al.  On the error of the quasi-steady-state approximation , 1993 .

[13]  Benjamin Peherstorfer,et al.  Lift & Learn: Physics-informed machine learning for large-scale nonlinear dynamical systems , 2019, Physica D: Nonlinear Phenomena.

[14]  Devika Subramanian,et al.  Data-driven predictions of a multiscale Lorenz 96 chaotic system using machine-learning methods: reservoir computing, artificial neural network, and long short-term memory network , 2020, Nonlinear Processes in Geophysics.

[15]  Lawrence F. Shampine,et al.  A User’s View of Solving Stiff Ordinary Differential Equations , 1979 .

[16]  Qing Nie,et al.  DifferentialEquations.jl – A Performant and Feature-Rich Ecosystem for Solving Differential Equations in Julia , 2017, Journal of Open Research Software.

[17]  Jaideep Pathak,et al.  Backpropagation algorithms and Reservoir Computing in Recurrent Neural Networks for the forecasting of complex spatiotemporal dynamics , 2019, Neural Networks.

[18]  D. Ellison,et al.  Efficient automatic integration of ordinary differential equations with discontinuities , 1981 .

[19]  Jan G. Verwer,et al.  Gauss-Seidel Iteration for Stiff ODES from Chemical Kinetics , 1994, SIAM J. Sci. Comput..

[20]  Athanasios S. Polydoros,et al.  Advantages and limitations of reservoir computing on model learning for robot control , 2015 .

[21]  Aleksej F. Filippov,et al.  Differential Equations with Discontinuous Righthand Sides , 1988, Mathematics and Its Applications.

[22]  Francesco Casella,et al.  Multi-rate integration algorithms: a path towards efficient simulation of object-oriented models of very large systems , 2014, EOOLT '14.

[23]  Thierry S. Nouidui,et al.  Modelica Buildings library , 2014 .

[24]  Kevin Gimpel,et al.  Gaussian Error Linear Units (GELUs) , 2016 .

[25]  Yoshua Bengio,et al.  On the Spectral Bias of Neural Networks , 2018, ICML.

[26]  O. Martin,et al.  Short relaxation times but long transient times in both simple and complex reaction networks , 2015, Journal of The Royal Society Interface.

[27]  B. Schrauwen,et al.  Reservoir computing and extreme learning machines for non-linear time-series data analysis , 2013, Neural Networks.

[28]  H. H. Robertson,et al.  Numerical Integration of Systems of Stiff Ordinary Differential Equations with Special Structure , 1976 .

[29]  V. B. Nguyena,et al.  Model reduction for reacting flow applications , 2014 .

[30]  Vipin Kumar,et al.  Integrating Physics-Based Modeling with Machine Learning: A Survey , 2020, ArXiv.

[31]  Yoshua Bengio,et al.  Understanding the difficulty of training deep feedforward neural networks , 2010, AISTATS.

[32]  Herbert Jaeger,et al.  Optimization and applications of echo state networks with leaky- integrator neurons , 2007, Neural Networks.

[33]  K. Gobbert.  ROBERTSON'S EXAMPLE FOR STIFF DIFFERENTIAL EQUATIONS , 2007 .

[34]  I. Sobol,et al.  Construction and Comparison of High-Dimensional Sobol' Generators , 2011 .

[35]  Karen Willcox,et al.  A Survey of Projection-Based Model Reduction Methods for Parametric Dynamical Systems , 2015, SIAM Rev..

[36]  Herbert Jaeger,et al.  Adaptive Nonlinear System Identification with Echo State Networks , 2002, NIPS.

[37]  Karen Willcox,et al.  Nonlinear Model Order Reduction via Lifting Transformations and Proper Orthogonal Decomposition , 2018, AIAA Journal.

[38]  Risto Miikkulainen,et al.  Reservoir Computing , 2017, Encyclopedia of Machine Learning and Data Mining.

[40]  Michelle Girvan,et al.  Hybrid Forecasting of Chaotic Processes: Using Machine Learning in Conjunction with a Knowledge-Based Model , 2018, Chaos.

[41]  J. M. Watt.  Numerical Initial Value Problems in Ordinary Differential Equations , 1972 .

[42]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[43]  Cory Merkel,et al.  Design and Analysis of a Neuromemristive Reservoir Computing Architecture for Biosignal Processing , 2016, Front. Neurosci..

[44]  H. Robertson,et al.  Some Properties of Algorithms for Stiff Differential Equations , 1975 .

[45]  Francesco Casella,et al.  Simulation of Large-Scale Models in Modelica: State of the Art and Future Perspectives , 2015 .

[46]  E. Hairer,et al.  Stiff differential equations solved by Radau methods , 1999 .

[47]  E. Hairer,et al.  Solving Ordinary Differential Equations II , 2010 .

[48]  Dimitris A. Goussis,et al.  Asymptotic Solution of Stiff PDEs with the CSP Method: The Reaction Diffusion Equation , 1998, SIAM J. Sci. Comput..

[49]  Hilding Elmqvist,et al.  Physical system modeling with Modelica , 1998 .

[50]  Karen Willcox,et al.  Surrogate and reduced-order modeling: a comparison of approaches for large-scale statistical inverse problems [Chapter 7] , 2010 .

[51]  Wolfgang Polifke,et al.  Physics-Informed Echo State Networks for Chaotic Systems Forecasting , 2019, ICCS.

[52]  Paris Perdikaris,et al.  Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations , 2019, J. Comput. Phys..

[53]  Philipp Thomas,et al.  Communication: limitations of the stochastic quasi-steady-state approximation in open biochemical reaction networks. , 2011, The Journal of chemical physics.

[54]  Youngsoo Choi,et al.  A fast and accurate physics-informed neural network reduced order model with shallow masked autoencoder , 2020, J. Comput. Phys..

[55]  Stéphane Bressan,et al.  Hydrological Process Surrogate Modelling and Simulation with Neural Networks , 2020, PAKDD.

[56]  Carol S. Woodward,et al.  Enabling New Flexibility in the SUNDIALS Suite of Nonlinear and Differential/Algebraic Equation Solvers , 2020, ACM Trans. Math. Softw..