Data Driven Governing Equations Approximation Using Deep Neural Networks

Abstract We present a numerical framework for approximating unknown governing equations using observation data and deep neural networks (DNN). In particular, we propose to use residual network (ResNet) as the basic building block for equation approximation. We demonstrate that the ResNet block can be considered as a one-step method that is exact in temporal integration. We then present two multi-step methods, recurrent ResNet (RT-ResNet) method and recursive ReNet (RS-ResNet) method. The RT-ResNet is a multi-step method on uniform time steps, whereas the RS-ResNet is an adaptive multi-step method using variable time steps. All three methods presented here are based on integral form of the underlying dynamical system. As a result, they do not require time derivative data for equation recovery and can cope with relatively coarsely distributed trajectory data. Several numerical examples are presented to demonstrate the performance of the methods.

[1]  A. R. Humphries,et al.  Dynamical Systems And Numerical Analysis , 1996 .

[2]  Steven L. Brunton,et al.  Data-driven discovery of partial differential equations , 2016, Science Advances.

[3]  M.N.S. Swamy,et al.  Neural Networks and Statistical Learning , 2013 .

[4]  Hayden Schaeffer,et al.  Sparse model selection via integral terms. , 2017, Physical review. E.

[5]  Yalchin Efendiev,et al.  Deep Multiscale Model Learning , 2018, J. Comput. Phys..

[6]  Razvan Pascanu,et al.  On the Number of Linear Regions of Deep Neural Networks , 2014, NIPS.

[7]  Nicholas Zabaras,et al.  Bayesian Deep Convolutional Encoder-Decoder Networks for Surrogate Modeling and Uncertainty Quantification , 2018, J. Comput. Phys..

[8]  George E. Karniadakis,et al.  Hidden physics models: Machine learning of nonlinear partial differential equations , 2017, J. Comput. Phys..

[9]  Andrew R. Barron,et al.  Universal approximation bounds for superpositions of a sigmoidal function , 1993, IEEE Trans. Inf. Theory.

[10]  H. Schaeffer,et al.  Learning partial differential equations via data discovery and sparse optimization , 2017, Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences.

[11]  Jan S. Hesthaven,et al.  An artificial neural network as a troubled-cell indicator , 2018, J. Comput. Phys..

[12]  Jürgen Schmidhuber,et al.  Deep learning in neural networks: An overview , 2014, Neural Networks.

[13]  Ilias Bilionis,et al.  Deep UQ: Learning deep neural network surrogate models for high dimensional uncertainty quantification , 2018, J. Comput. Phys..

[14]  A. Majda,et al.  Nonlinear Laplacian spectral analysis for time series with intermittency and low-frequency variability , 2012, Proceedings of the National Academy of Sciences.

[15]  Michael D. Schmidt,et al.  Automated refinement and inference of analytical models for metabolic networks , 2011, Physical biology.

[16]  Kailiang Wu,et al.  Numerical Aspects for Approximating Governing Equations Using Data , 2018, J. Comput. Phys..

[17]  J N Kutz,et al.  Model selection for dynamical systems via sparse regression and information criteria , 2017, Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences.

[18]  David Duvenaud,et al.  Neural Ordinary Differential Equations , 2018, NeurIPS.

[19]  R. Tibshirani.  Regression Shrinkage and Selection via the Lasso , 1996 .

[20]  Bryan C. Daniels,et al.  Automated adaptive inference of phenomenological dynamical models , 2015, Nature Communications.

[21]  Franco Scarselli,et al.  On the Complexity of Neural Network Classifiers: A Comparison Between Shallow and Deep Architectures , 2014, IEEE Transactions on Neural Networks and Learning Systems.

[22]  Maziar Raissi,et al.  Deep Hidden Physics Models: Deep Learning of Nonlinear Partial Differential Equations , 2018, J. Mach. Learn. Res..

[23]  Paris Perdikaris,et al.  Machine learning of linear differential equations using Gaussian processes , 2017, J. Comput. Phys..

[24]  Ohad Shamir,et al.  The Power of Depth for Feedforward Neural Networks , 2015, COLT.

[25]  Steven L. Brunton,et al.  Deep learning of dynamics and signal-noise decomposition with time-stepping constraints , 2018, J. Comput. Phys..

[26]  George Sugihara,et al.  Detecting Causality in Complex Ecosystems , 2012, Science.

[27]  Frederick Tung,et al.  Multi-level Residual Networks from Dynamical Systems View , 2017, ICLR.

[28]  Hod Lipson,et al.  Distilling Free-Form Natural Laws from Experimental Data , 2009, Science.

[29]  Lorenzo Rosasco,et al.  Why and when can deep-but not shallow-networks avoid the curse of dimensionality: A review , 2016, International Journal of Automation and Computing.

[30]  Steven L. Brunton,et al.  Chaos as an intermittently forced linear system , 2016, Nature Communications.

[31]  C. W. Gear,et al.  Equation-Free, Coarse-Grained Multiscale Computation: Enabling Mocroscopic Simulators to Perform System-Level Analysis , 2003 .

[32]  Lexing Ying,et al.  Solving parametric PDE problems with artificial neural networks , 2017, European Journal of Applied Mathematics.

[33]  Hao Wu,et al.  VAMPnets for deep learning of molecular kinetics , 2017, Nature Communications.

[34]  J. Collins,et al.  Construction of a genetic toggle switch in Escherichia coli , 2000, Nature.

[35]  Allan Pinkus,et al.  Approximation theory of the MLP model in neural networks , 1999, Acta Numerica.

[36]  Jan S. Hesthaven,et al.  Non-intrusive reduced order modeling of nonlinear problems using neural networks , 2018, J. Comput. Phys..

[37]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[38]  Kurt Hornik,et al.  Approximation capabilities of multilayer feedforward networks , 1991, Neural Networks.

[39]  Bryan C. Daniels,et al.  Efficient Inference of Parsimonious Phenomenological Models of Cellular Dynamics Using S-Systems and Alternating Regression , 2014, PloS one.

[40]  Allan Pinkus,et al.  Multilayer Feedforward Networks with a Non-Polynomial Activation Function Can Approximate Any Function , 1991, Neural Networks.

[41]  Giang Tran,et al.  Exact Recovery of Chaotic Systems from Highly Corrupted Data , 2016, Multiscale Model. Simul..

[42]  Ioannis G. Kevrekidis,et al.  Identification of distributed parameter systems: A neural net based approach , 1998 .

[43]  Philip M. Long,et al.  Representing smooth functions as compositions of near-identity functions with implications for deep network optimization , 2018, ArXiv.

[44]  Ahmed H. Elsheikh,et al.  A machine learning approach for efficient uncertainty quantification using multiscale methods , 2017, J. Comput. Phys..

[45]  Geoffrey E. Hinton,et al.  Deep Learning , 2015, Nature.

[46]  J. Kurths,et al.  Amplitude equations from spatiotemporal binary-fluid convection data , 1999 .

[47]  Bin Dong,et al.  PDE-Net: Learning PDEs from Data , 2017, ICML.

[48]  G. Karniadakis,et al.  Multistep Neural Networks for Data-driven Discovery of Nonlinear Dynamical Systems , 2018, 1801.01236.

[49]  Rick Chartrand,et al.  Numerical Differentiation of Noisy, Nonsmooth Data , 2011 .

[50]  George Sugihara,et al.  Equation-free mechanistic ecosystem forecasting using empirical dynamic modeling , 2015, Proceedings of the National Academy of Sciences.

[51]  S. Brunton,et al.  Discovering governing equations from data by sparse identification of nonlinear dynamical systems , 2015, Proceedings of the National Academy of Sciences.

[52]  E Weinan,et al.  Heterogeneous multiscale method: A general methodology for multiscale modeling , 2003 .

[53]  Hod Lipson,et al.  Automated reverse engineering of nonlinear dynamical systems , 2007, Proceedings of the National Academy of Sciences.