Bridging Physics-based and Data-driven modeling for Learning Dynamical Systems

How can we learn a dynamical system to make forecasts, when some variables are unobserved? For instance, in COVID-19, we want to forecast the number of infected and death cases but we do not know the count of susceptible and exposed people. While mechanics compartment models are widely-used in epidemic modeling, data-driven models are emerging for disease forecasting. As a case study, we compare these two types of models for COVID-19 forecasting and notice that physics-based models significantly outperform deep learning models. We present a hybrid approach, AutoODE-COVID, which combines a novel compartmental model with automatic differentiation. Our method obtains a 57.4% reduction in mean absolute errors for 7-day ahead COVID-19 forecasting compared with the best deep learning competitor. To understand the inferior performance of deep learning, we investigate the generalization problem in forecasting. Through systematic experiments, we found that deep learning models fail to forecast under shifted distributions either in the data domain or the parameter domain. This calls attention to rethink generalization especially for learning dynamical systems.

[1]  Karthik Kashinath,et al.  Towards Physics-informed Deep Learning for Turbulent Flow Prediction , 2020, KDD.

[2]  Luca Antiga,et al.  Automatic differentiation in PyTorch , 2017 .

[3]  Ali Ramadhan,et al.  Universal Differential Equations for Scientific Machine Learning , 2020, ArXiv.

[4]  Ali Al-Aradi,et al.  Solving Nonlinear and High-Dimensional Partial Differential Equations via Deep Learning , 2018, 1811.08782.

[5]  Wouter M. Kouw.  An introduction to domain adaptation and transfer learning , 2018, ArXiv.

[6]  Jan Van Impe,et al.  A Tutorial on Numerical Methods for State and Parameter Estimation in Nonlinear Dynamic Systems , 2012 .

[7]  Shair Ahmad,et al.  On the nonautonomous Volterra-Lotka competition equations , 1993 .

[8]  Julio Delgado,et al.  Elastic Machine Learning Algorithms in Amazon SageMaker , 2020, SIGMOD Conference.

[9]  Patrick Gallinari,et al.  Learning Dynamical Systems from Partial Observations , 2019, ArXiv.

[10]  Bryan Lim,et al.  Time-series forecasting with deep learning: a survey , 2020, Philosophical Transactions of the Royal Society A.

[11]  L. Wang,et al.  Epidemic Model Guided Machine Learning for COVID-19 Forecasts in the United States , 2020, medRxiv.

[12]  R. FitzHugh.  Impulses and Physiological States in Theoretical Models of Nerve Membrane. , 1961, Biophysical journal.

[13]  Rose Yu,et al.  Incorporating Symmetry into Deep Dynamics Models for Improved Generalization , 2020, ArXiv.

[14]  S. Yoshizawa,et al.  An Active Pulse Transmission Line Simulating Nerve Axon , 1962, Proceedings of the IRE.

[15]  Xue Ben,et al.  Deep Transformer Models for Time Series Forecasting: The Influenza Prevalence Case , 2020, ArXiv.

[16]  Samuel B. Williams,et al.  ASSOCIATION FOR COMPUTING MACHINERY , 2000 .

[17]  Michael Bohlke-Schneider,et al.  High-Dimensional Multivariate Forecasting with Low-Rank Gaussian Copula Processes , 2019, NeurIPS.

[18]  A. Vespignani,et al.  Estimating the establishment of local transmission and the cryptic phase of the COVID-19 pandemic in the USA , 2020, medRxiv.

[19]  R. May,et al.  Infectious Diseases of Humans: Dynamics and Control , 1991, Annals of Internal Medicine.

[20]  Ahmet Murat Ozbayoglu,et al.  Financial Time Series Forecasting with Deep Learning : A Systematic Literature Review: 2005-2019 , 2019, Appl. Soft Comput..

[21]  Syama Sundar Rangapuram,et al.  Neural forecasting: Introduction and literature overview , 2020, ArXiv.

[22]  D. Lathrop.  Nonlinear Dynamics and Chaos: With Applications to Physics, Biology, Chemistry, and Engineering , 2015 .

[23]  Lukasz Kaiser,et al.  Attention is All you Need , 2017, NIPS.

[24]  S. Pei,et al.  Initial Simulation of SARS-CoV2 Spread and Intervention Effects in the Continental US , 2020, medRxiv.

[25]  Valentin Flunkert,et al.  DeepAR: Probabilistic Forecasting with Autoregressive Recurrent Networks , 2017, International Journal of Forecasting.

[26]  Cheng-Shang Chang,et al.  A Time-dependent SIR model for COVID-19 , 2020, ArXiv.

[27]  K. Torkkola,et al.  A Multi-Horizon Quantile Recurrent Forecaster , 2017, 1711.11053.

[28]  E. Dong,et al.  An interactive web-based dashboard to track COVID-19 in real time , 2020, The Lancet Infectious Diseases.

[29]  John Schulman,et al.  Concrete Problems in AI Safety , 2016, ArXiv.

[30]  Justin A. Sirignano,et al.  DGM: A deep learning algorithm for solving partial differential equations , 2017, J. Comput. Phys..

[31]  David Duvenaud,et al.  Neural Ordinary Differential Equations , 2018, NeurIPS.

[32]  Wenhu Chen,et al.  Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecasting , 2019, NeurIPS.

[33]  D. Zeng,et al.  Survival-Convolution Models for Predicting COVID-19 Cases and Assessing Effects of Mitigation Strategies , 2020, medRxiv.

[34]  S. Brunton,et al.  Discovering governing equations from data by sparse identification of nonlinear dynamical systems , 2015, Proceedings of the National Academy of Sciences.

[35]  George E. Karniadakis,et al.  Hidden physics models: Machine learning of nonlinear partial differential equations , 2017, J. Comput. Phys..

[36]  P. Kaye.  Infectious diseases of humans: Dynamics and control , 1993 .

[37]  Utkarsh Upadhyay,et al.  Recurrent Marked Temporal Point Processes: Embedding Event History to Vector , 2016, KDD.

[38]  Patrick Gallinari,et al.  Learning Partially Observed PDE Dynamics with Neural Networks , 2018 .

[39]  Matthias W. Seeger,et al.  Deep State Space Models for Time Series Forecasting , 2018, NeurIPS.

[40]  Pietro Liò,et al.  Graph Attention Networks , 2017, ICLR.

[41]  Barak A. Pearlmutter,et al.  Automatic differentiation in machine learning: a survey , 2015, J. Mach. Learn. Res..