A Regularized Linear Dynamical System Framework for Multivariate Time Series Analysis

Linear Dynamical System (LDS) is an elegant mathematical framework for modeling and learning Multivariate Time Series (MTS). However, in general, it is difficult to set the dimension of an LDS's hidden state space. A small number of hidden states may not be able to model the complexities of a MTS, while a large number of hidden states can lead to overfitting. In this paper, we study learning methods that impose various regularization penalties on the transition matrix of the LDS model and propose a regularized LDS learning framework (rLDS) which aims to (1) automatically shut down LDSs' spurious and unnecessary dimensions, and consequently, address the problem of choosing the optimal number of hidden states; (2) prevent the overfitting problem given a small amount of MTS data; and (3) support accurate MTS forecasting. To learn the regularized LDS from data we incorporate a second order cone program and a generalized gradient descent method into the Maximum a Posteriori framework and use Expectation Maximization to obtain a low-rank transition matrix of the LDS model. We propose two priors for modeling the matrix which lead to two instances of our rLDS. We show that our rLDS is able to recover well the intrinsic dimensionality of the time series dynamics and it improves the predictive performance when compared to baselines on both synthetic and real-world MTS datasets.

[1]  James R. Bence,et al.  Analysis of Short Time Series: Correcting for Autocorrelation , 1995 .

[2]  Alessandro Chiuso,et al.  Learning sparse dynamic linear systems using stable spline kernels and exponential hyperpriors , 2010, NIPS.

[3]  T. Başar,et al.  A New Approach to Linear Filtering and Prediction Problems , 2001 .

[4]  Justin K. Romberg,et al.  Sparsity penalties in dynamical system estimation , 2011, 2011 45th Annual Conference on Information Sciences and Systems.

[5]  Volker Roth,et al.  The Bayesian group-Lasso for analyzing contingency tables , 2009, ICML '09.

[6]  Milos Hauskrecht,et al.  A temporal pattern mining approach for classifying electronic health record data , 2013, ACM Trans. Intell. Syst. Technol..

[7]  Judith Rousseau,et al.  Bayesian matrix completion: prior specification and consistency , 2014 .

[8]  John L. Kling,et al.  A comparison of multivariate forecasting procedures for economic time series , 1985 .

[9]  Julien Mairal,et al.  Convex optimization with sparsity-inducing norms , 2011 .

[10]  Ziv Bar-Joseph,et al.  Clustering short time series gene expression data , 2005, ISMB.

[11]  Bruno A. Olshausen,et al.  Group Sparse Coding with a Laplacian Scale Mixture Prior , 2010, NIPS.

[12]  Gregory F Cooper,et al.  Conditional outlier detection for clinical alerting. , 2010, AMIA ... Annual Symposium proceedings. AMIA Symposium.

[13]  Geoffrey E. Hinton,et al.  Parameter estimation for linear dynamical systems , 1996 .

[14]  Milos Hauskrecht,et al.  Clinical time series prediction: Toward a hierarchical dynamical system framework , 2015, Artif. Intell. Medicine.

[15]  Narendra Ahuja,et al.  Sparse Coding of Linear Dynamical Systems with an Application to Dynamic Texture Recognition , 2010, 2010 20th International Conference on Pattern Recognition.

[16]  Jan Lunze,et al.  Qualitative modelling of linear dynamical systems with quantized state measurements , 1994, Autom..

[17]  Lennart Ljung,et al.  Modeling Of Dynamic Systems , 1994 .

[18]  Pini Gurfil,et al.  Methods for Sparse Signal Recovery Using Kalman Filtering With Embedded Pseudo-Measurement Norms and Quasi-Norms , 2010, IEEE Transactions on Signal Processing.

[19]  Katya Scheinberg,et al.  Noname manuscript No. (will be inserted by the editor) Efficient Block-coordinate Descent Algorithms for the Group Lasso , 2022 .

[20]  G. Reinsel.  Elements of Multivariate Time Series Analysis , 1995 .

[21]  Milos Hauskrecht,et al.  Modeling Clinical Time Series Using Gaussian Process Sequences , 2013, SDM.

[22]  Justin K. Romberg,et al.  Estimation and dynamic updating of time-varying signals with sparse variations , 2011, 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[23]  Bart De Moor,et al.  Subspace Identification for Linear Systems: Theory ― Implementation ― Applications , 2011 .

[24]  Byron Boots,et al.  A Constraint Generation Approach to Learning Stable Linear Dynamical Systems , 2007, NIPS.

[25]  Sach Mukherjee,et al.  Penalized estimation in high-dimensional hidden Markov models with state-specific graphical models , 2012, 1208.4989.

[26]  Milos Hauskrecht,et al.  Clinical Time Series Prediction with a Hierarchical Dynamical System , 2013, AIME.

[27]  Milos Hauskrecht,et al.  Feature importance analysis for patient management decisions , 2010, MedInfo.

[28]  Stergios I. Roumeliotis,et al.  Lasso-Kalman smoother for tracking sparse signals , 2009, 2009 Conference Record of the Forty-Third Asilomar Conference on Signals, Systems and Computers.

[29]  S. F. Witt,et al.  Univariate versus multivariate time series forecasting: an application to international tourism demand , 2003 .

[30]  琢史 大久保.  ズームアップ経済統計 マクロ経済データは真に無料の時代に(アメリカ・セントルイス連銀「Federal Reserve Economic Data」) , 2015 .

[31]  Gilles Clermont,et al.  Outlier detection for patient monitoring and alerting , 2013, J. Biomed. Informatics.

[32]  Jieping Ye,et al.  Efficient Methods for Overlapping Group Lasso , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[33]  Tohru Katayama,et al.  Subspace Methods for System Identification , 2005 .