Learning Dynamical Systems Requires Rethinking Generalization

The ability to generalize to unseen data is at the core of machine learning. A traditional view of generalization refers to unseen data from the same distribution. Dynamical systems challenge the conventional wisdom of generalization in learning systems due to distribution shifts from non-stationarity and chaos. In this paper, we investigate the generalization ability of dynamical systems in the forecasting setting. Through systematic experiments, we show deep learning models fail to generalize to shifted distributions in the data and parameter domains of dynamical systems. We find a sharp contrast between the performance of deep learning models on interpolation (same distribution) and extrapolation (shifted distribution). Our findings can help explain the inferior performance of deep learning models compared to physics-based models on the COVID-19 forecasting task.

[1]  S. Yoshizawa,et al.  An Active Pulse Transmission Line Simulating Nerve Axon , 1962, Proceedings of the IRE.

[2]  Jan Van Impe,et al.  A Tutorial on Numerical Methods for State and Parameter Estimation in Nonlinear Dynamic Systems , 2012 .

[3]  John Schulman,et al.  Concrete Problems in AI Safety , 2016, ArXiv.

[4]  Lukasz Kaiser,et al.  Attention is All you Need , 2017, NIPS.

[5]  Valentin Flunkert,et al.  DeepAR: Probabilistic Forecasting with Autoregressive Recurrent Networks , 2017, International Journal of Forecasting.

[6]  Ali Al-Aradi,et al.  Solving Nonlinear and High-Dimensional Partial Differential Equations via Deep Learning , 2018, 1811.08782.

[7]  George E. Karniadakis,et al.  Hidden physics models: Machine learning of nonlinear partial differential equations , 2017, J. Comput. Phys..

[8]  David Duvenaud,et al.  Neural Ordinary Differential Equations , 2018, NeurIPS.

[9]  Patrick Gallinari,et al.  Learning Partially Observed PDE Dynamics with Neural Networks , 2018 .

[10]  Matthias W. Seeger,et al.  Deep State Space Models for Time Series Forecasting , 2018, NeurIPS.

[11]  Wouter M. Kouw.  An introduction to domain adaptation and transfer learning , 2018, ArXiv.

[12]  Justin A. Sirignano,et al.  DGM: A deep learning algorithm for solving partial differential equations , 2017, J. Comput. Phys..

[13]  Wenhu Chen,et al.  Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecasting , 2019, NeurIPS.

[14]  Julio Delgado,et al.  Elastic Machine Learning Algorithms in Amazon SageMaker , 2020, SIGMOD Conference.

[15]  Ahmet Murat Ozbayoglu,et al.  Financial Time Series Forecasting with Deep Learning : A Systematic Literature Review: 2005-2019 , 2019, Appl. Soft Comput..

[16]  Rui Wang,et al.  Towards Physics-informed Deep Learning for Turbulent Flow Prediction , 2019, KDD.

[17]  Syama Sundar Rangapuram,et al.  Neural forecasting: Introduction and literature overview , 2020, ArXiv.

[18]  E. Dong,et al.  An interactive web-based dashboard to track COVID-19 in real time , 2020, The Lancet Infectious Diseases.

[19]  L. Wang,et al.  Epidemic Model Guided Machine Learning for COVID-19 Forecasts in the United States , 2020, medRxiv.

[20]  Xue Ben,et al.  Deep Transformer Models for Time Series Forecasting: The Influenza Prevalence Case , 2020, ArXiv.