High-dimensional time series prediction using kernel-based Koopman mode regression

We propose a novel methodology for high-dimensional time series prediction based on the kernel method extension of data-driven Koopman spectral analysis, via the following methodological advances: (a) a new numerical regularization method, (b) a natural ordering of Koopman modes which provides a fast alternative to the sparsity-promoting procedure, (c) a predictable Koopman modes selection technique which is equivalent to cross-validation in machine learning, (d) an optimization method for selected Koopman modes to improve prediction accuracy, (e) prediction model generation and selection based on historical error measures. The prediction accuracy of this methodology is excellent: for example, when it is used to predict clients’ order flow time series of foreign exchange, which is almost random, it can achieve more than 10% improvement on root-mean-square error over auto-regressive moving average. This methodology also opens up new possibilities for data-driven modeling and forecasting complex systems that generate the high-dimensional time series. We believe that this methodology will be of interest to the community of scientists and engineers working on quantitative finance, econometrics, system biology, neurosciences, meteorology, oceanography, system identification and control, data mining, machine learning, and many other fields involving high-dimensional time series and spatio-temporal data.

[1]  Michael Robinson,et al.  Sheaves are the canonical data structure for sensor integration , 2016, Inf. Fusion.

[2]  Ronald L. Allen,et al.  Signal Analysis: Time, Frequency, Scale and Structure , 2003 .

[3]  Gemunu H. Gunaratne,et al.  Variable diffusion in stock market fluctuations , 2015 .

[4]  Gemunu H. Gunaratne,et al.  Ensemble vs. time averages in financial time series analysis , 2012 .

[5]  Yifan Gong,et al.  Towards better performance with heterogeneous training data in acoustic modeling using deep neural networks , 2014, INTERSPEECH.

[6]  D. Giannakis Data-driven spectral decomposition and forecasting of ergodic dynamical systems , 2015, Applied and Computational Harmonic Analysis.

[7]  E. B. Andersen,et al.  Information Science and Statistics , 1986 .

[8]  Feng Ding,et al.  Decomposition based least squares iterative identification algorithm for multivariate pseudo-linear ARMA systems using the data filtering , 2017, J. Frankl. Inst..

[9]  Feng Ding,et al.  A multi-innovation state and parameter estimation algorithm for a state space system with d-step state-delay , 2017, Signal Process..

[10]  Mehdi Ghommem,et al.  Real-time tumor ablation simulation based on the dynamic mode decomposition method. , 2014, Medical physics.

[11]  Philip Heng Wai Leong,et al.  Dynamic hedging of foreign exchange risk using stochastic model predictive control , 2014, 2014 IEEE Conference on Computational Intelligence for Financial Engineering & Economics (CIFEr).

[12]  Andrzej Banaszuk,et al.  Comparison of systems with complex behavior , 2004 .

[13]  M. Golubitsky,et al.  Singularities and groups in bifurcation theory , 1985 .

[14]  Wei Zhang,et al.  Improved least squares identification algorithm for multivariable Hammerstein systems , 2015, J. Frankl. Inst..

[15]  Meng Joo Er,et al.  Hybrid recursive least squares algorithm for online sequential identification using data chunks , 2016, Neurocomputing.

[16]  Nadine Aubry,et al.  Spatiotemporal analysis of complex signals: Theory and applications , 1991 .

[17]  Steven L. Brunton,et al.  Dynamic Mode Decomposition with Control , 2014, SIAM J. Appl. Dyn. Syst..

[18]  Steven L. Brunton,et al.  Multi-Resolution Dynamic Mode Decomposition , 2015 .

[19]  I. Mezić,et al.  Applied Koopmanism. , 2012, Chaos.

[20]  Dongqing Wang,et al.  Hierarchical parameter estimation for a class of MIMO Hammerstein systems based on the reframed models , 2016, Appl. Math. Lett..

[21]  J. McCauley,et al.  Using dynamic mode decomposition to extract cyclic behavior in the stock market , 2016 .

[22]  Heni Ben Amor,et al.  Estimation of perturbations in robotic behavior using dynamic mode decomposition , 2015, Adv. Robotics.

[23]  Clarence W. Rowley,et al.  A Data–Driven Approximation of the Koopman Operator: Extending Dynamic Mode Decomposition , 2014, Journal of Nonlinear Science.

[24]  I. Mezić Spectral Properties of Dynamical Systems, Model Reduction and Decompositions , 2005 .

[25]  P. Schmid Nonmodal Stability Theory , 2007 .

[26]  P. Schmid,et al.  Dynamic mode decomposition of numerical and experimental data , 2008, Journal of Fluid Mechanics.

[27]  Gemunu H Gunaratne,et al.  Deconvolution of reacting-flow dynamics using proper orthogonal and dynamic mode decompositions. , 2015, Physical review. E, Statistical, nonlinear, and soft matter physics.

[28]  Carl E. Rasmussen,et al.  Gaussian processes for machine learning , 2005, Adaptive computation and machine learning.

[29]  Bingni W. Brunton,et al.  Extracting spatial–temporal coherent patterns in large-scale neural recordings using dynamic mode decomposition , 2014, Journal of Neuroscience Methods.

[30]  M. Cross,et al.  Pattern formation outside of equilibrium , 1993 .

[31]  B. O. Koopman,et al.  Hamiltonian Systems and Transformation in Hilbert Space. , 1931, Proceedings of the National Academy of Sciences of the United States of America.

[32]  Geoffrey Zweig,et al.  Recent advances in deep learning for speech research at Microsoft , 2013, 2013 IEEE International Conference on Acoustics, Speech and Signal Processing.

[33]  Steven L. Brunton,et al.  On dynamic mode decomposition: Theory and applications , 2013, 1312.0041.

[34]  James R. Gord,et al.  Dynamic-mode decomposition based analysis of shear coaxial jets with and without transverse acoustic driving , 2016, Journal of Fluid Mechanics.

[35]  I. Mezić,et al.  Analysis of Fluid Flows via Spectral Properties of the Koopman Operator , 2013 .

[36]  Peter J. Schmid,et al.  Sparsity-promoting dynamic mode decomposition , 2012, 1309.4165.

[37]  Joseph C. Slater,et al.  A numerical method for determining nonlinear normal modes , 1996 .

[38]  Yoav Freund,et al.  Boosting: Foundations and Algorithms , 2012 .

[39]  Rob J Hyndman,et al.  Automatic Time Series Forecasting: The forecast Package for R , 2008 .

[40]  Steven L. Brunton,et al.  Generalizing Koopman Theory to Allow for Inputs and Control , 2016, SIAM J. Appl. Dyn. Syst..

[41]  Zhi-Hua Zhou,et al.  Ensemble Methods: Foundations and Algorithms , 2012 .

[42]  Alexandre Mauroy,et al.  Linear identification of nonlinear systems: A lifting technique based on the Koopman operator , 2016, 2016 IEEE 55th Conference on Decision and Control (CDC).

[43]  Clarence W. Rowley,et al.  Data fusion via intrinsic dynamic variables: An application of data-driven Koopman spectral analysis , 2014, 1411.5424.

[44]  Robert Allen,et al.  Handbook of Medical Imaging—Processing and Analysis , 2001 .

[45]  I. Mezić,et al.  Spectral analysis of nonlinear flows , 2009, Journal of Fluid Mechanics.

[46]  Nello Cristianini,et al.  An Introduction to Support Vector Machines and Other Kernel-based Learning Methods , 2000 .

[47]  Trevor Hastie,et al.  The Elements of Statistical Learning , 2001 .

[48]  J. Nathan Kutz,et al.  Dynamic mode decomposition for financial trading strategies , 2015, 1508.04487.

[49]  Bernhard Schölkopf,et al.  A kernel view of the dimensionality reduction of manifolds , 2004, ICML.

[50]  J.S.Chitode DIGITAL SIGNAL PROCESSING FUNDAMENTALS , 2011 .

[51]  Philip H. W. Leong,et al.  Stochastic Receding Horizon Control for Short-Term Risk Management in Foreign Exchange , 2015 .

[52]  Wei Xing Zheng,et al.  Parameter estimation algorithms for Hammerstein output error systems using Levenberg-Marquardt optimization method with varying interval measurements , 2017, J. Frankl. Inst..

[53]  Joshua Garland,et al.  Model-free quantification of time-series predictability. , 2014, Physical review. E, Statistical, nonlinear, and soft matter physics.

[54]  Meng Joo Er,et al.  Constructive multi-output extreme learning machine with application to large tanker motion dynamics identification , 2014, Neurocomputing.

[55]  Philip H. W. Leong,et al.  On Time Series Forecasting Error Measures for Finite Horizon Control , 2017, IEEE Transactions on Control Systems Technology.

[56]  Julien M. Hendrickx,et al.  Spectral Identification of Networks Using Sparse Measurements , 2016, SIAM J. Appl. Dyn. Syst..

[57]  I. Mezic,et al.  Nonlinear Koopman Modes and a Precursor to Power System Swing Instabilities , 2012, IEEE Transactions on Power Systems.

[58]  Bernhard Schölkopf,et al.  The Kernel Trick for Distances , 2000, NIPS.

[59]  Mohammad Valipour Ability of Box-Jenkins Models to Estimate of Reference Potential Evapotranspiration (A Case Study: Mehrabad Synoptic Station, Tehran, Iran) , 2012 .

[60]  Christophe Pierre,et al.  Non-linear normal modes, invariance, and modal dynamics approximations of non-linear systems , 1995, Nonlinear Dynamics.

[61]  M. Valipour Long‐term runoff study using SARIMA and ARIMA models in the United States , 2015 .

[62]  F. Ding,et al.  Multi-innovation stochastic gradient identification for Hammerstein controlled autoregressive autoregressive systems based on the filtering technique , 2015 .

[63]  Adrian E. Raftery,et al.  Weather Forecasting with Ensemble Methods , 2005, Science.

[64]  Zhizhen Zhao,et al.  Spatiotemporal Feature Extraction with Data-Driven Koopman Operators , 2015, FE@NIPS.

[65]  George Haller,et al.  Nonlinear normal modes and spectral submanifolds: existence, uniqueness and use in model reduction , 2016, 1602.00560.

[66]  Steven L. Brunton,et al.  Compressed Dynamic Mode Decomposition for Real-Time Object Detection , 2015, ArXiv.

[67]  Meng Joo Er,et al.  Generalized Single-Hidden Layer Feedforward Networks for Regression Problems , 2015, IEEE Transactions on Neural Networks and Learning Systems.

[68]  Meng Joo Er,et al.  Parsimonious Extreme Learning Machine Using Recursive Orthogonal Least Squares , 2014, IEEE Transactions on Neural Networks and Learning Systems.

[69]  Joshua L. Proctor,et al.  Discovering dynamic patterns from infectious disease data using dynamic mode decomposition , 2015, International health.

[70]  Steven L. Brunton,et al.  Koopman Invariant Subspaces and Finite Linear Representations of Nonlinear Dynamical Systems for Control , 2015, PloS one.

[71]  Farzad Noorian,et al.  Risk Management using Model Predictive Control , 2015 .

[72]  Christopher M. Bishop,et al.  Pattern Recognition and Machine Learning (Information Science and Statistics) , 2006 .

[73]  Patrick T. Brewick,et al.  An evaluation of data-driven identification strategies for complex nonlinear dynamic systems , 2016 .

[74]  M. Valipour,et al.  Comparison of the ARMA, ARIMA, and the autoregressive artificial neural network models in forecasting the monthly inflow of Dez dam reservoir , 2013 .

[75]  Matthew O. Williams,et al.  A Kernel-Based Approach to Data-Driven Koopman Spectral Analysis , 2014, 1411.2260.