Learning Dynamical Systems from Data: A Simple Cross-Validation Perspective, Part V: Sparse Kernel Flows for 132 Chaotic Dynamical Systems

Regressing the vector field of a dynamical system from a finite number of observed states is a natural way to learn surrogate models for such systems. A simple and interpretable way to learn a dynamical system from data is to interpolate its vector-field with a data-adapted kernel which can be learned by using Kernel Flows. The method of Kernel Flows is a trainable machine learning method that learns the optimal parameters of a kernel based on the premise that a kernel is good if there is no significant loss in accuracy if half of the data is used. The objective function could be a short-term prediction or some other objective for other variants of Kernel Flows). However, this method is limited by the choice of the base kernel. In this paper, we introduce the method of \emph{Sparse Kernel Flows } in order to learn the ``best'' kernel by starting from a large dictionary of kernels. It is based on sparsifying a kernel that is a linear combination of elemental kernels. We apply this approach to a library of 132 chaotic systems.

[1]  B. Hamzi,et al.  Learning dynamical systems from data: A simple cross-validation perspective, part IV: Case with partial observations , 2023, Physica D: Nonlinear Phenomena.

[2]  H. Owhadi,et al.  Learning "best" kernels from data in Gaussian process regression. With application to aerodynamics , 2022, J. Comput. Phys..

[3]  B. Hamzi,et al.  Learning dynamical systems from data: A simple cross-validation perspective, part III: Irregularly-Sampled Time Series , 2021, Physica A: Statistical Mechanics and its Applications.

[4]  H. Owhadi.  Computational graph completion , 2021, Research in the Mathematical Sciences.

[5]  W. Gilpin.  Chaos as an interpretable benchmark for forecasting and data-driven modelling , 2021, NeurIPS Datasets and Benchmarks.

[6]  Houman Owhadi,et al.  Solving and Learning Nonlinear PDEs with Gaussian Processes , 2021, J. Comput. Phys..

[7]  B. Hamzi,et al.  Simple, low-cost and accurate data-driven geophysical forecasting with learned kernels , 2021, Proceedings of the Royal Society A.

[8]  Bernard Haasdonk,et al.  Kernel methods for center manifold approximation and a data-based version of the Center Manifold Theorem , 2020, ArXiv.

[9]  E. Gilli.  Monographs , 2020, Big Karst Chambers.

[10]  Houman Owhadi,et al.  Learning dynamical systems from data: a simple cross-validation perspective , 2020, Physica D: Nonlinear Phenomena.

[11]  B. Hamzi,et al.  Kernel-Based Approximation of the Koopman Generator and Schrödinger Operator , 2020, Entropy.

[12]  H. Owhadi,et al.  Deep regularization and direct training of the inner layers of Neural Networks with Kernel Flows , 2020, ArXiv.

[13]  C. Clementi,et al.  Data-driven approximation of the Koopman generator: Model reduction, system identification, and control , 2019, Physica D: Nonlinear Phenomena.

[14]  G. Santin,et al.  Kernel Methods for Surrogate Modeling , 2019, System- and Data-Driven Methods and Algorithms.

[15]  Devika Subramanian,et al.  Data-driven prediction of a multi-scale Lorenz 96 chaotic system using a hierarchy of deep learning methods: Reservoir computing, ANN, and RNN-LSTM , 2019, ArXiv.

[16]  Fritz Colonius,et al.  Kernel methods for the approximation of discrete-time linear autonomous and control systems , 2019, SN Applied Sciences.

[17]  D. Giannakis,et al.  Operator-theoretic framework for forecasting nonlinear time series with kernel analog techniques , 2019, Physica D: Nonlinear Phenomena.

[18]  B. Hamzi,et al.  Dimensionality Reduction of Complex Metastable Systems via Kernel Embeddings of Transition Manifolds , 2019, Journal of Nonlinear Science.

[19]  Bernard Haasdonk,et al.  Greedy Kernel Methods for Center Manifold Approximation , 2018, 1810.11329.

[20]  Houman Owhadi,et al.  Kernel Flows: from learning kernels from data into the abyss , 2018, J. Comput. Phys..

[21]  Christian Kuehn,et al.  A note on kernel methods for multiscale systems with critical transitions , 2018, Mathematical Methods in the Applied Sciences.

[22]  Yoonsuh Jung.  Multiple predicting K-fold cross-validation for model selection , 2018 .

[23]  Jaideep Pathak,et al.  Using machine learning to replicate chaotic attractors and calculate Lyapunov exponents from data. , 2017, Chaos.

[24]  P. Giesl,et al.  Approximation of Lyapunov functions from noisy data , 2016, Journal of Computational Dynamics.

[25]  S. Brunton,et al.  Discovering governing equations from data by sparse identification of nonlinear dynamical systems , 2015, Proceedings of the National Academy of Sciences.

[26]  J. A. Stewart,et al.  Nonlinear Time Series Analysis , 2015 .

[27]  B. Hamzi,et al.  Empirical estimators for stochastically forced nonlinear systems: Observability, controllability and the invariant measure , 2012, 2012 American Control Conference (ACC).

[28]  Jake V. Bouvrie,et al.  Kernel Methods for the Approximation of Some Key Quantities of Nonlinear Systems , 2012, 1204.0563.

[29]  JAKE BOUVRIE,et al.  Kernel Methods for the Approximation of Nonlinear Systems , 2011, SIAM J. Control. Optim..

[30]  P. Groenen,et al.  Nonlinear Forecasting with Many Predictors Using Kernel Ridge Regression , 2011 .

[31]  Felipe Cucker,et al.  On the mathematical foundations of learning , 2001 .

[32]  Ioannis G. Kevrekidis,et al.  Identification of distributed parameter systems: A neural net based approach , 1998 .

[33]  A. B. Bulsari,et al.  Neural Networks for Chemical Engineers , 1995 .

[34]  Ioannis G. Kevrekidis,et al.  DISCRETE- vs. CONTINUOUS-TIME NONLINEAR SIGNAL PROCESSING OF Cu ELECTRODISSOLUTION DATA , 1992 .

[35]  Martin Casdagli,et al.  Nonlinear prediction of chaotic time series , 1989 .

[36]  N. Aronszajn.  Theory of Reproducing Kernels. , 1950 .

[37]  B. Hamzi,et al.  One-Shot Learning of Stochastic Differential Equations with Computational Graph Completion , 2022, ArXiv.

[38]  B. Hamzi,et al.  Learning dynamical systems from data: a simple cross-validation perspective, part , 2021 .

[39]  Houman Owhadi,et al.  A Game Theoretic Approach to Numerical Approximation and Algorithm Design , 2018 .

[40]  S. Wakeman.  One Shot , 2013, Journal of General Internal Medicine.

[41]  Léon Bottou,et al.  Stochastic Gradient Descent Tricks , 2012, Neural Networks: Tricks of the Trade.

[42]  Mehmet Emre Çek,et al.  Analysis of observed chaotic data , 2004 .

[43]  R. Tibshirani.  Regression Shrinkage and Selection via the Lasso , 1996 .

[44]  Ioannis G. Kevrekidis,et al.  Nonlinear signal processing and system identification: applications to time series from electrochemical reactions , 1990 .

[45]  Laurence B. Milstein,et al.  and Control Systems , 1984 .