Variational Deep Learning for the Identification and Reconstruction of Chaotic and Stochastic Dynamical Systems from Noisy and Partial Observations

The data-driven recovery of the unknown governing equations of dynamical systems has recently received an increasing interest. However, the identification of the governing equations remains challenging when dealing with noisy and partial observations. Here, we address this challenge and investigate variational deep learning schemes. Within the proposed framework, we jointly learn an inference model to reconstruct the true states of the system from series of noisy and partial data and the governing equations of these states. In doing so, this framework bridges classical data assimilation and state-of-the-art machine learning techniques and we show that it generalizes state-of-the-art methods. Importantly, both the inference model and the governing equations embed stochastic components to account for stochastic variabilities, model errors and reconstruction uncertainties. Various experiments on chaotic and stochastic dynamical systems support the relevance of our scheme w.r.t. state-of-the-art approaches.

[1]  Kailiang Wu,et al.  Data Driven Governing Equations Approximation Using Deep Neural Networks , 2018, J. Comput. Phys..

[2]  Jaideep Pathak,et al.  Using machine learning to replicate chaotic attractors and calculate Lyapunov exponents from data. , 2017, Chaos.

[3]  Zoubin Ghahramani,et al.  Learning Nonlinear Dynamical Systems Using an EM Algorithm , 1998, NIPS.

[4]  Ronan Fablet,et al.  Joint learning of variational representations and solvers for inverse problems with partially-observed data , 2020, ArXiv.

[5]  Ronan Fablet,et al.  Residual Integration Neural Network , 2019, ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[6]  Ole Winther,et al.  Sequential Neural Models with Stochastic Layers , 2016, NIPS.

[7]  Igor Melnyk,et al.  Deep learning algorithm for data-driven simulation of noisy dynamical system , 2018, J. Comput. Phys..

[8]  E. Lorenz.  Predictability of Weather and Climate: Predictability – a problem partly solved , 2006 .

[9]  David Duvenaud,et al.  Latent Ordinary Differential Equations for Irregularly-Sampled Time Series , 2019, NeurIPS.

[10]  S. Brunton,et al.  Discovering governing equations from data by sparse identification of nonlinear dynamical systems , 2015, Proceedings of the National Academy of Sciences.

[11]  Bertrand Chapron,et al.  Large‐scale flows under location uncertainty: a consistent stochastic framework , 2018 .

[12]  Christopher K. Wikle,et al.  A model‐based approach for analog spatio‐temporal dynamic forecasting , 2015, 1506.06169.

[13]  Jaideep Pathak,et al.  Model-Free Prediction of Large Spatiotemporally Chaotic Systems from Data: A Reservoir Computing Approach. , 2018, Physical review letters.

[14]  Nancy Nichols,et al.  Very large inverse problems in atmosphere and ocean modelling , 2005 .

[15]  J. L. Roux.  An Introduction to the Kalman Filter , 2003 .

[16]  Jeffrey L. Anderson,et al.  An investigation into the application of an ensemble Kalman smoother to high-dimensional geophysical systems , 2008 .

[17]  Yee Whye Teh,et al.  Filtering Variational Objectives , 2017, NIPS.

[18]  Redouane Lguensat,et al.  The Analog Data Assimilation , 2017 .

[19]  Ronan Fablet,et al.  EM-like Learning Chaotic Dynamics from Noisy and Partial Observations , 2019, ArXiv.

[20]  Steven L. Brunton,et al.  Data-Driven Science and Engineering , 2019 .

[21]  A. Doucet,et al.  A Tutorial on Particle Filtering and Smoothing: Fifteen years later , 2008 .

[22]  G. Karniadakis,et al.  Multistep Neural Networks for Data-driven Discovery of Nonlinear Dynamical Systems , 2018, 1801.01236.

[23]  G. Evensen,et al.  An ensemble Kalman smoother for nonlinear dynamics , 2000 .

[24]  Steven Lake Waslander,et al.  Multistep Prediction of Dynamic Systems With Recurrent Neural Networks , 2018, IEEE Transactions on Neural Networks and Learning Systems.

[25]  G. Evensen.  Data Assimilation: The Ensemble Kalman Filter , 2006 .

[26]  Geoffrey E. Hinton,et al.  Parameter estimation for linear dynamical systems , 1996 .

[27]  Patrick Gallinari,et al.  Learning Dynamical Systems from Partial Observations , 2019, ArXiv.

[28]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[29]  Ronan Fablet,et al.  Assimilation-Based Learning of Chaotic Dynamical Systems from Noisy and Partial Data , 2020, ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[30]  Petros Koumoutsakos,et al.  Data-driven forecasting of high-dimensional chaotic systems with long short-term memory networks , 2018, Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences.

[31]  Nasser M. Nasrabadi,et al.  Pattern Recognition and Machine Learning , 2006, Technometrics.

[32]  Zhizhen Zhao,et al.  Analog forecasting with dynamics-adapted kernels , 2014, 1412.3831.

[33]  M. Hoshiya,et al.  Structural Identification by Extended Kalman Filter , 1984 .

[34]  Marc Bocquet,et al.  Combining data assimilation and machine learning to emulate a dynamical model from sparse and noisy observations: a case study with the Lorenz 96 model , 2019, J. Comput. Sci..

[35]  Yoshua Bengio,et al.  A Recurrent Latent Variable Model for Sequential Data , 2015, NIPS.

[36]  M. Hirsch,et al.  Differential Equations, Dynamical Systems, and an Introduction to Chaos , 2003 .

[37]  Bertrand Chapron,et al.  Learning Latent Dynamics for Partially-Observed Chaotic Systems , 2019, Chaos.

[38]  Marc Bocquet,et al.  Bayesian inference of dynamics from partial and noisy observations using data assimilation and machine learning , 2020, ArXiv.

[39]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[40]  Steven L. Brunton,et al.  Data-driven discovery of coordinates and governing equations , 2019, Proceedings of the National Academy of Sciences.

[41]  Luca Delle Monache,et al.  An Evaluation of Analog-Based Postprocessing Methods across Several Variables and Forecast Models , 2015 .

[42]  Cédric Herzet,et al.  Bilinear Residual Neural Network for the Identification and Forecasting of Geophysical Dynamics , 2018, 2018 26th European Signal Processing Conference (EUSIPCO).

[43]  A. Wolf,et al.  Determining Lyapunov exponents from a time series , 1985 .

[44]  David Duvenaud,et al.  Neural Ordinary Differential Equations , 2018, NeurIPS.

[45]  J. Isern‐Fontanet,et al.  Diagnosis of high-resolution upper ocean dynamics from noisy sea surface temperatures , 2014 .

[46]  Marc Bocquet,et al.  Data assimilation as a learning tool to infer ordinary differential equation representations of dynamical models , 2019, Nonlinear Processes in Geophysics.

[47]  Ruslan Salakhutdinov,et al.  Importance Weighted Autoencoders , 2015, ICLR.

[48]  Jürgen Kurths,et al.  Nonlinear Dynamical System Identification from Uncertain and Indirect Measurements , 2004, Int. J. Bifurc. Chaos.

[49]  J. Sprott.  Chaos and time-series analysis , 2001 .

[50]  P. Courtier,et al.  A strategy for operational implementation of 4D‐Var, using an incremental approach , 1994 .

[51]  E. Lorenz.  Deterministic nonperiodic flow , 1963 .

[52]  David W. Pierce,et al.  Distinguishing coupled ocean–atmosphere interactions from background noise in the North Pacific , 2001 .

[53]  Robert C. Hilborn,et al.  Chaos And Nonlinear Dynamics: An Introduction for Scientists and Engineers , 1994 .