Variational inference formulation for a model-free simulation of a dynamical system with unknown parameters by a recurrent neural network

We propose a recurrent neural network for a "model-free" simulation of a dynamical system with unknown parameters without prior knowledge. The deep learning model aims to jointly learn the nonlinear time marching operator and the effects of the unknown parameters from a time series dataset. We assume that the time series data set consists of an ensemble of trajectories for a range of the parameters. The learning task is formulated as a statistical inference problem by considering the unknown parameters as random variables. A variational inference method is employed to train a recurrent neural network jointly with a feedforward neural network for an approximately posterior distribution. The approximate posterior distribution makes an inference on a trajectory to identify the effects of the unknown parameters and a recurrent neural network makes a prediction by using the outcome of the inference. In the numerical experiments, it is shown that the proposed variational inference model makes a more accurate simulation compared to the standard recurrent neural networks. It is found that the proposed deep learning model is capable of correctly identifying the dimensions of the random parameters and learning a representation of complex time series data.

[1]  E. Lorenz.  Deterministic nonperiodic flow , 1963 .

[2]  Miss A.O. Penney.  (b) , 1974, The New Yale Book of Quotations.

[3]  L. Glass,et al.  Oscillation and chaos in physiological control systems. , 1977, Science.

[4]  J. D. Farmer,et al.  Chaotic attractors of an infinite-dimensional dynamical system , 1982 .

[5]  Jonathan D. Cryer,et al.  Time Series Analysis , 1986 .

[6]  Andrew Harvey,et al.  Forecasting, Structural Time Series Models and the Kalman Filter , 1990 .

[7]  Steven H. Strogatz,et al.  Nonlinear Dynamics and Chaos , 2024 .

[8]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[9]  Jürgen Schmidhuber,et al.  Learning to Forget: Continual Prediction with LSTM , 2000, Neural Computation.

[10]  J. Sprott.  Chaos and time-series analysis , 2001 .

[11]  Neil J. Gordon,et al.  A tutorial on particle filters for online nonlinear/non-Gaussian Bayesian tracking , 2002, IEEE Trans. Signal Process..

[12]  Dongbin Xiu,et al.  The Wiener-Askey Polynomial Chaos for Stochastic Differential Equations , 2002, SIAM J. Sci. Comput..

[13]  Geir Evensen,et al.  The Ensemble Kalman Filter: theoretical formulation and practical implementation , 2003 .

[14]  Celso Grebogi,et al.  Chemical and biological activity in open flows: A dynamical system approach , 2005 .

[15]  Silvina Ponce Dawson,et al.  Simple data-driven models of intracellular calcium dynamics with predictive power. , 2006, Physical review. E, Statistical, nonlinear, and soft matter physics.

[16]  Jiguo Cao,et al.  Parameter estimation for differential equations: a generalized smoothing approach , 2007 .

[17]  Nasser M. Nasrabadi,et al.  Pattern Recognition and Machine Learning , 2006, Technometrics.

[18]  Habib N. Najm,et al.  Uncertainty Quantification and Polynomial Chaos Techniques in Computational Fluid Dynamics , 2009 .

[19]  J. Sprott.  Elegant Chaos: Algebraically Simple Chaotic Flows , 2010 .

[20]  C. Furusawa,et al.  A Dynamical-Systems View of Stem Cell Biology , 2012, Science.

[21]  George Sugihara,et al.  Detecting Causality in Complex Ecosystems , 2012, Science.

[22]  David R. Cox,et al.  Time Series Analysis , 2012 .

[23]  Pascal Vincent,et al.  Representation Learning: A Review and New Perspectives , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[24]  Leonidas S. Rompolis,et al.  Forecasting the mean and volatility of stock returns from option prices , 2006 .

[25]  Yoshua Bengio,et al.  Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation , 2014, EMNLP.

[26]  Max Welling,et al.  Auto-Encoding Variational Bayes , 2013, ICLR.

[27]  Michael G Fehlings,et al.  Introduction to focus issue. , 2014, Spine.

[28]  Steven L. Brunton,et al.  On dynamic mode decomposition: Theory and applications , 2013, 1312.0041.

[29]  Yoshua Bengio,et al.  A Recurrent Latent Variable Model for Sequential Data , 2015, NIPS.

[30]  Tyrus Berry,et al.  Predicting chaotic time series with a partial model. , 2015, Physical review. E, Statistical, nonlinear, and soft matter physics.

[31]  Geoffrey E. Hinton,et al.  Deep Learning , 2015, Nature.

[32]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[33]  S. Brunton,et al.  Discovering governing equations from data by sparse identification of nonlinear dynamical systems , 2015, Proceedings of the National Academy of Sciences.

[34]  David M. Blei,et al.  Variational Inference: A Review for Statisticians , 2016, ArXiv.

[35]  Tyrus Berry,et al.  Ensemble Kalman Filtering without a Model , 2016 .

[36]  Thomas Erneux,et al.  Introduction to Focus Issue: Time-delay dynamics. , 2017, Chaos.

[37]  Christopher Burgess,et al.  beta-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework , 2016, ICLR 2016.

[38]  Frank Hutter,et al.  SGDR: Stochastic Gradient Descent with Warm Restarts , 2016, ICLR.

[39]  Yu Zhang,et al.  Unsupervised Learning of Disentangled and Interpretable Representations from Sequential Data , 2017, NIPS.

[40]  Steven L. Brunton,et al.  Deep learning for universal linear embeddings of nonlinear dynamics , 2017, Nature Communications.

[41]  Stephan Mandt,et al.  Disentangled Sequential Autoencoder , 2018, ICML.

[42]  Nicholas Zabaras,et al.  Bayesian Deep Convolutional Encoder-Decoder Networks for Surrogate Modeling and Uncertainty Quantification , 2018, J. Comput. Phys..

[43]  Aaas News,et al.  Book Reviews , 1893, Buffalo Medical and Surgical Journal.

[44]  Steven L. Brunton,et al.  Deep learning of dynamics and signal-noise decomposition with time-stepping constraints , 2018, J. Comput. Phys..

[45]  Igor Melnyk,et al.  Deep learning algorithm for data-driven simulation of noisy dynamical system , 2018, J. Comput. Phys..

[46]  Kyongmin Yeo,et al.  Short note on the behavior of recurrent neural network for noisy dynamical system , 2019, ArXiv.

[47]  Paris Perdikaris,et al.  Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations , 2019, J. Comput. Phys..

[48]  Paris Perdikaris,et al.  Adversarial Uncertainty Quantification in Physics-Informed Neural Networks , 2018, J. Comput. Phys..

[49]  Petros Koumoutsakos,et al.  Machine Learning for Fluid Mechanics , 2019, Annual Review of Fluid Mechanics.

[50]  Allen G Hart,et al.  Embedding and approximation theorems for echo state networks , 2019, Neural Networks.

[51]  Nicholas Geneva,et al.  Modeling the Dynamics of PDE Systems with Physics-Constrained Deep Auto-Regressive Networks , 2019, J. Comput. Phys..

[52]  George Em Karniadakis,et al.  Hidden fluid mechanics: Learning velocity and pressure fields from flow visualizations , 2020, Science.

[53]  Luning Sun,et al.  Surrogate modeling for fluid flows based on physics-constrained deep learning without simulation data , 2019, Computer Methods in Applied Mechanics and Engineering.

[54]  Chryssostomos Chryssostomidis,et al.  Learning functionals via LSTM neural networks for predicting vessel dynamics in extreme sea states , 2019, Proceedings of the Royal Society A.

[55]  P. Alam,et al.  H , 1887, High Explosives, Propellants, Pyrotechnics.

[56]  P. Alam.  ‘A’ , 2021, Composites Engineering: An A–Z Guide.