Linear Dynamics: Clustering without identification

Clustering time series is a delicate task; varying lengths and temporal offsets obscure direct comparisons. A natural strategy is to learn a parametric model foreach time series and to cluster the model parameters rather than the sequences themselves. Linear dynamical systems are a fundamental and powerful parametric model class. However, identifying the parameters of a linear dynamical systems is a venerable task, permitting provably efficient solutions only in special cases. In this work, we show that clustering the parameters of unknown linear dynamical systems is, in fact, easier than identifying them. We analyze a computationally efficient clustering algorithm that enjoys provable convergence guarantees under a natural separation assumption. Although easy to implement, our algorithm is general, handling multi-dimensional data with time offsets and partial sequences. Evaluating our algorithm on both synthetic data and real electrocardiogram (ECG) signals, we see significant improvements in clustering quality over existing baselines.

[1]  J. Durbin EFFICIENT ESTIMATION OF PARAMETERS IN MOVING-AVERAGE MODELS , 1959 .

[2]  J. Durbin Estimation of Parameters in Time‐Series Regression Models , 1960 .

[3]  Clive W. J. Granger,et al.  Time Series Modelling and Interpretation , 1976 .

[4]  S. Chiba,et al.  Dynamic programming algorithm optimization for spoken word recognition , 1978 .

[5]  E. Hannan,et al.  Estimation of vector ARMAX models , 1980 .

[6]  A. Hasman,et al.  Piecewise analysis of EEGs using AR-modeling and clustering. , 1981, Computers and biomedical research, an international journal.

[7]  S. Liberty,et al.  Linear Systems , 2010, Scientific Parallel Computing.

[8]  R. Shumway,et al.  AN APPROACH TO TIME SERIES SMOOTHING AND FORECASTING USING THE EM ALGORITHM , 1982 .

[9]  David Q. Mayne,et al.  Linear identification of ARMA processes , 1982, Autom..

[10]  T. Lai,et al.  Asymptotic properties of general autoregressive models and strong consistency of least-squares estimates of their parameters , 1983 .

[11]  G. C. Tiao,et al.  Consistency Properties of Least Squares Estimates of Autoregressive Parameters in ARMA Models , 1983 .

[12]  G. C. Tiao,et al.  Consistent Estimates of Autoregressive Parameters and Extended Sample Autocorrelation Function for Stationary and Nonstationary ARMA Models , 1984 .

[13]  Karl Johan Åström,et al.  Computer-Controlled Systems: Theory and Design , 1984 .

[14]  G. E. Taylor,et al.  Computer Controlled Systems: Theory and Design , 1985 .

[15]  Lawrence R. Rabiner,et al.  A modified K-means clustering algorithm for use in isolated work recognition , 1985, IEEE Trans. Acoust. Speech Signal Process..

[16]  L. Hubert,et al.  Comparing partitions , 1985 .

[17]  B. Friedlander,et al.  Optimal instrumental variable multistep algorithms for estimation of the AR parameters of an ARMA process , 1985, 1985 24th IEEE Conference on Decision and Control.

[18]  T. Söderström,et al.  A high-order Yule-Walker method for estimation of the AR parameters of an ARMA model , 1988 .

[19]  B. Hanzon Identifiability, recursive identification and spaces of linear dynamical systems: part II , 1989 .

[20]  K. Kosmelj,et al.  Cross-sectional approach for clustering time varying data , 1990 .

[21]  D. Piccolo A DISTANCE MEASURE FOR CLASSIFYING ARIMA MODELS , 1990 .

[22]  Jian Li,et al.  Optimal high-order Yule-Walker estimation of sinusoidal frequencies , 1991, IEEE Trans. Signal Process..

[23]  G. P. King,et al.  Using cluster analysis to classify time series , 1992 .

[24]  ByoungSeon Choi,et al.  Arma Model Identification , 1992 .

[25]  B. Bercu Weighted estimation and tracking for ARMAX models , 1992, [1992] Proceedings of the 31st IEEE Conference on Decision and Control.

[26]  Geoffrey E. Hinton,et al.  Parameter estimation for linear dynamical systems , 1996 .

[27]  Padhraic Smyth,et al.  Clustering Sequences with Hidden Markov Models , 1996, NIPS.

[28]  Lei Guo Self-convergence of weighted least-squares with applications to stochastic adaptive control , 1996, IEEE Trans. Autom. Control..

[29]  P. Boesiger,et al.  A new correlation‐based fuzzy logic clustering algorithm for FMRI , 1998, Magnetic resonance in medicine.

[30]  Jarke J. van Wijk,et al.  Cluster and Calendar Based Visualization of Time Series Data , 1999, INFOVIS.

[31]  L. K. Hansen,et al.  On Clustering fMRI Time Series , 1999, NeuroImage.

[32]  Zoubin Ghahramani,et al.  A Unifying Review of Linear Gaussian Models , 1999, Neural Computation.

[33]  P. Lancaster,et al.  On the perturbation of analytic matrix functions , 1999 .

[34]  B. Beauzamy How the Roots of a Polynomial Vary with its Coefficients: A Local Quantitative Result , 1999, Canadian Mathematical Bulletin.

[35]  Gautam Biswas,et al.  Temporal Pattern Generation Using Hidden Markov Model Based Unsupervised Classification , 1999, IDA.

[36]  Jeffrey M. Hausdorff,et al.  Physionet: Components of a New Research Resource for Complex Physiologic Signals". Circu-lation Vol , 2000 .

[37]  Gérard Govaert,et al.  Assessing a Mixture Model for Clustering with the Integrated Completed Likelihood , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[38]  B. Moor,et al.  Subspace angles and distances between ARMA models , 2000 .

[39]  Elizabeth Ann Maharaj,et al.  Cluster of Time Series , 2000, J. Classif..

[40]  Richard J. Martin A metric for ARMA processes , 2000, IEEE Trans. Signal Process..

[41]  G.B. Moody,et al.  The impact of the MIT-BIH Arrhythmia Database , 2001, IEEE Engineering in Medicine and Biology Magazine.

[42]  Arindam Banerjee,et al.  Clickstream clustering using weighted longest common subsequences , 2001 .

[43]  Konstantinos Kalpakis,et al.  Distance measures for effective clustering of ARIMA time-series , 2001, Proceedings 2001 IEEE International Conference on Data Mining.

[44]  T. Başar,et al.  A New Approach to Linear Filtering and Prediction Problems , 2001 .

[45]  Mahesh Kumar,et al.  Clustering seasonality patterns in the presence of errors , 2002, KDD.

[46]  Manuele Bicego,et al.  A Hidden Markov Model-Based Approach to Sequential Data Clustering , 2002, SSPR/SPR.

[47]  Dit-Yan Yeung,et al.  Mixtures of ARMA models for model-based time series clustering , 2002, 2002 IEEE International Conference on Data Mining, 2002. Proceedings..

[48]  Dat Tran,et al.  Fuzzy C-Means Clustering-Based Speaker Verification , 2002, AFSS.

[49]  D. Ge,et al.  Cardiac arrhythmia classification using autoregressive modeling , 2002, Biomedical engineering online.

[50]  Dimitrios Gunopulos,et al.  Discovering similar multidimensional trajectories , 2002, Proceedings 18th International Conference on Data Engineering.

[51]  Peter Lancaster,et al.  Perturbation Theory for Analytic Matrix Functions: The Semisimple Case , 2003, SIAM J. Matrix Anal. Appl..

[52]  Frank Klawonn,et al.  Fuzzy Clustering of Short Time-Series and Unevenly Distributed Sampling Points , 2003, IDA.

[53]  Mário A. T. Figueiredo,et al.  Similarity-Based Clustering of Sequences Using Hidden Markov Models , 2003, MLDM.

[54]  Moawwad E. A. El-Mikkawy Explicit inverse of a generalized Vandermonde matrix , 2003, Appl. Math. Comput..

[55]  Dimitrios Gunopulos,et al.  A Wavelet-Based Anytime Algorithm for K-Means Clustering of Time Series , 2003 .

[56]  P. Stoica,et al.  Optimal Yule Walker Method for Pole Estimation of ARMA Signals , 2003 .

[57]  Philip de Chazal,et al.  Automatic classification of heartbeats using ECG morphology and heartbeat interval features , 2004, IEEE Transactions on Biomedical Engineering.

[58]  Inderjit S. Dhillon,et al.  Kernel k-means: spectral clustering and normalized cuts , 2004, KDD.

[59]  Lei Chen,et al.  On The Marriage of Lp-norms and Edit Distance , 2004, VLDB.

[60]  George Kollios,et al.  Extraction and clustering of motion trajectories in video , 2004, Proceedings of the 17th International Conference on Pattern Recognition, 2004. ICPR 2004..

[61]  T. Warren Liao,et al.  Clustering of time series data - a survey , 2005, Pattern Recognit..

[62]  Nuno Vasconcelos,et al.  Probabilistic kernels for the classification of auto-regressive visual processes , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[63]  Petre Stoica,et al.  Spectral Analysis of Signals , 2009 .

[64]  Gian Luca Foresti,et al.  Trajectory clustering and its applications for video surveillance , 2005, IEEE Conference on Advanced Video and Signal Based Surveillance, 2005..

[65]  Lei Chen,et al.  Robust and fast similarity search for moving object trajectories , 2005, SIGMOD '05.

[66]  Hans-Peter Kriegel,et al.  Similarity Search on Time Series Based on Threshold Queries , 2006, EDBT.

[67]  Yüksel Özbay,et al.  A fuzzy clustering neural network architecture for classification of ECG arrhythmias , 2006, Comput. Biol. Medicine.

[68]  B. Ray,et al.  An Interweaved HMM/DTW Approach to Robust Time Series Clustering , 2006, 18th International Conference on Pattern Recognition (ICPR'06).

[69]  Alexander J. Smola,et al.  Binet-Cauchy Kernels on Dynamical Systems and its Application to the Analysis of Dynamic Scenes , 2007, International Journal of Computer Vision.

[70]  Anthony K. H. Tung,et al.  SpADe: On Shape-based Pattern Detection in Streaming Time Series , 2007, 2007 IEEE 23rd International Conference on Data Engineering.

[71]  Julia Hirschberg,et al.  V-Measure: A Conditional Entropy-Based External Cluster Evaluation Measure , 2007, EMNLP.

[72]  Zeeshan Syed,et al.  Clustering and Symbolic Analysis of Cardiovascular Signals: Discovery and Visualization of Medically Relevant Patterns in Long-Term Data Using Limited Prior Knowledge , 2007, EURASIP J. Adv. Signal Process..

[73]  Jignesh M. Patel,et al.  An efficient and accurate method for evaluating time series similarity , 2007, SIGMOD '07.

[74]  Philipp Koehn,et al.  Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL) , 2007 .

[75]  Marcella Corduas,et al.  Time series clustering and classification by the autoregressive metric , 2008, Comput. Stat. Data Anal..

[76]  Ilse C. F. Ipsen,et al.  Perturbation Bounds for Determinants and Characteristic Polynomials , 2008, SIAM J. Matrix Anal. Appl..

[77]  Yüksel Özbay,et al.  A novel approach for classification of ECG arrhythmias: Type-2 fuzzy clustering neural network , 2009, Expert Syst. Appl..

[78]  automatic classification of , 2009 .

[79]  James Bailey,et al.  Information Theoretic Measures for Clusterings Comparison: Variants, Properties, Normalization and Correction for Chance , 2010, J. Mach. Learn. Res..

[80]  Mehmet Korürek,et al.  Clustering MIT-BIH arrhythmias with Ant Colony Optimization using time domain and PCA compressed wavelet coefficients , 2010, Digit. Signal Process..

[81]  Skipper Seabold,et al.  Statsmodels: Econometric and Statistical Modeling with Python , 2010, SciPy.

[82]  Gaël Varoquaux,et al.  Scikit-learn: Machine Learning in Python , 2011, J. Mach. Learn. Res..

[83]  Lei Li,et al.  Time Series Clustering: Complex is Simpler! , 2011, ICML.

[84]  Marco Cuturi,et al.  Fast Global Alignment Kernels , 2011, ICML.

[85]  René Vidal,et al.  Group action induced distances for averaging and clustering Linear Dynamical Systems with applications to the analysis of dynamic scenes , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[86]  Che Wun Chiou,et al.  Analyzing ECG for cardiac arrhythmia using cluster analysis , 2012, Expert Syst. Appl..

[87]  Ying Wah Teh,et al.  Time-series clustering - A decade review , 2015, Inf. Syst..

[88]  Luis Gravano,et al.  k-Shape: Efficient and Accurate Clustering of Time Series , 2016, SGMD.

[89]  Tak-Chung Fu,et al.  Pattern discovery from stock time series using self-organizing maps , 2016 .

[90]  Marco Cuturi,et al.  Soft-DTW: a Differentiable Loss Function for Time-Series , 2017, ICML.

[91]  Karan Singh,et al.  Learning Linear Dynamical Systems via Spectral Filtering , 2017, NIPS.

[92]  Michael I. Jordan,et al.  Learning Without Mixing: Towards A Sharp Analysis of Linear System Identification , 2018, COLT.

[93]  Tengyu Ma,et al.  Gradient Descent Learns Linear Dynamical Systems , 2016, J. Mach. Learn. Res..

[94]  Yi Zhang,et al.  Spectral Filtering for General Linear Dynamical Systems , 2018, NeurIPS.

[95]  Masoumeh Haghpanahi,et al.  Cardiologist-level arrhythmia detection and classification in ambulatory electrocardiograms using a deep neural network , 2019, Nature Medicine.

[96]  George J. Pappas,et al.  Finite Sample Analysis of Stochastic System Identification , 2019, 2019 IEEE 58th Conference on Decision and Control (CDC).

[97]  M. Desai,et al.  Rationale and design of a large-scale, app-based study to identify cardiac arrhythmias using a smartwatch: The Apple Heart Study , 2018, American heart journal.

[98]  Andrei Novikov,et al.  PyClustering: Data Mining Library , 2019, J. Open Source Softw..

[99]  Anne Greenbaum,et al.  First-order Perturbation Theory for Eigenvalues and Eigenvectors , 2019, SIAM Rev..

[100]  Nikolai Matni,et al.  On the Sample Complexity of the Linear Quadratic Regulator , 2017, Foundations of Computational Mathematics.