Markov Neural Operators for Learning Chaotic Systems

Chaotic systems are notoriously challenging to predict because of their instability. Small errors accumulate in the simulation of each time step, resulting in completely different trajectories. However, the trajectories of many prominent chaotic systems live in a low-dimensional subspace (attractor). If the system is Markovian, the attractor is uniquely determined by the Markov operator that maps the evolution of infinitesimal time steps. This makes it possible to predict the behavior of the chaotic system by learning the Markov operator even if we cannot predict the exact trajectory. Recently, a new framework for learning resolution-invariant solution operators for PDEs was proposed, known as neural operators. In this work, we train a Markov neural operator (MNO) with only the local one-step evolution information. We then compose the learned operator to obtain the global attractor and invariant measure. Such a Markov neural operator forms a discrete semigroup and we empirically observe that does not collapse or blow up. Experiments show neural operators are more accurate and stable compared to previous methods on chaotic systems such as the Kuramoto-Sivashinsky and Navier-Stokes equations.

[1]  Razvan Pascanu,et al.  Sobolev Training for Neural Networks , 2017, NIPS.

[2]  Maziar Raissi,et al.  Deep Hidden Physics Models: Deep Learning of Nonlinear Partial Differential Equations , 2018, J. Mach. Learn. Res..

[3]  Nicholas Geneva,et al.  Modeling the Dynamics of PDE Systems with Physics-Constrained Deep Auto-Regressive Networks , 2019, J. Comput. Phys..

[4]  Chun Zhang,et al.  Long-term prediction of chaotic systems with machine learning , 2020 .

[5]  Jürgen Schmidhuber,et al.  Learning to Forget: Continual Prediction with LSTM , 2000, Neural Computation.

[6]  Ryan P. Adams,et al.  Learning Composable Energy Surrogates for PDE Order Reduction , 2020, NeurIPS.

[7]  Andrew M. Stuart,et al.  Runge-Kutta methods for dissipative and gradient dynamical systems , 1994 .

[8]  Kamyar Azizzadenesheli,et al.  Neural Operator: Learning Maps Between Function Spaces , 2021, arXiv.org.

[9]  Stephan Hoyer,et al.  Learning data-driven discretizations for partial differential equations , 2018, Proceedings of the National Academy of Sciences.

[10]  Stephanie Boehm,et al.  Chaos Making A New Science , 2016 .

[11]  Jaideep Pathak,et al.  Backpropagation algorithms and Reservoir Computing in Recurrent Neural Networks for the forecasting of complex spatiotemporal dynamics , 2019, Neural Networks.

[12]  Catherine E. Powell,et al.  An Introduction to Computational Stochastic PDEs , 2014 .

[13]  Jaideep Pathak,et al.  Model-Free Prediction of Large Spatiotemporally Chaotic Systems from Data: A Reservoir Computing Approach. , 2018, Physical review letters.

[14]  Kamyar Azizzadenesheli,et al.  Fourier Neural Operator for Parametric Partial Differential Equations , 2021, ICLR.

[15]  Lloyd N. Trefethen,et al.  Fourth-Order Time-Stepping for Stiff PDEs , 2005, SIAM J. Sci. Comput..

[16]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[18]  Grigorios A. Pavliotis,et al.  Multiscale Methods: Averaging and Homogenization , 2008 .

[19]  David Duvenaud,et al.  Neural Ordinary Differential Equations , 2018, NeurIPS.

[20]  Dit-Yan Yeung,et al.  Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting , 2015, NIPS.

[21]  Kamyar Azizzadenesheli,et al.  Neural Operator: Graph Kernel Network for Partial Differential Equations , 2020, ICLR 2020.

[22]  Javier Jiménez,et al.  The turbulent cascade in five dimensions , 2017, Science.

[23]  Steven L. Brunton,et al.  Data-Driven Stabilization of Periodic Orbits , 2020, IEEE Access.

[24]  Gary J. Chandler,et al.  Invariant recurrent solutions embedded in a turbulent two-dimensional Kolmogorov flow , 2013, Journal of Fluid Mechanics.

[25]  G. Sviridyuk On the general theory of operator semigroups , 1994 .

[26]  Mark Holland,et al.  Central limit theorems and invariance principles for Lorenz attractors , 2007 .

[27]  Yoshua Bengio,et al.  Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling , 2014, ArXiv.

[28]  Petros Koumoutsakos,et al.  Data-driven forecasting of high-dimensional chaotic systems with long short-term memory networks , 2018, Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences.

[29]  A. Stuart Perturbation Theory for Infinite Dimensional Dynamical Systems , 1995 .

[30]  Alexander Smits,et al.  High–Reynolds Number Wall Turbulence , 2011 .

[31]  R. Temam Infinite Dimensional Dynamical Systems in Mechanics and Physics Springer Verlag , 1993 .

[32]  Kunihiko Taira,et al.  Machine-learning-based spatio-temporal super resolution reconstruction of turbulent flows , 2020, Journal of Fluid Mechanics.