Reservoir Computing Approaches for Representation and Classification of Multivariate Time Series

Classification of multivariate time series (MTS) has been tackled with a large variety of methodologies and applied to a wide range of scenarios. Reservoir computing (RC) provides efficient tools to generate a vectorial, fixed-size representation of the MTS that can be further processed by standard classifiers. Despite their unrivaled training speed, MTS classifiers based on a standard RC architecture fail to achieve the same accuracy of fully trainable neural networks. In this article, we introduce the reservoir model space, an unsupervised approach based on RC to learn vectorial representations of MTS. Each MTS is encoded within the parameters of a linear model trained to predict a low-dimensional embedding of the reservoir dynamics. Compared with other RC methods, our model space yields better representations and attains comparable computational performance due to an intermediate dimensionality reduction procedure. As a second contribution, we propose a modular RC framework for MTS classification, with an associated open-source Python library. The framework provides different modules to seamlessly implement advanced RC architectures. The architectures are compared with other MTS classifiers, including deep learning models and time series kernels. Results obtained on the benchmark and real-world MTS data sets show that RC classifiers are dramatically faster and, when implemented using our proposed representation, also achieve superior classification accuracy.

[1]  Robert Jenssen,et al.  Training Echo State Networks with Regularization Through Dimensionality Reduction , 2016, Cognitive Computation.

[2]  Ali Deihimi,et al.  Application of echo state networks in short-term electric load forecasting , 2012 .

[3]  Jiri Pospichal,et al.  Merging Echo State and Feedforward Neural Networks for Time Series Forecasting , 2006, ICANN.

[4]  Huanhuan Chen,et al.  Learning in the Model Space for Cognitive Fault Diagnosis , 2014, IEEE Transactions on Neural Networks and Learning Systems.

[5]  Yoshua Bengio,et al.  Understanding the difficulty of training deep feedforward neural networks , 2010, AISTATS.

[6]  Klaus Moessner,et al.  Univariate and Multivariate Time Series Manifold Learning , 2017, Knowl. Based Syst..

[7]  Shan Liu,et al.  An effective multivariate time series classification approach using echo state network and adaptive differential evolution algorithm , 2016, Expert Syst. Appl..

[8]  C. Anderson,et al.  Modeling reward functions for incomplete state representations via echo state networks , 2005, Proceedings. 2005 IEEE International Joint Conference on Neural Networks, 2005..

[9]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[10]  Houshang Darabi,et al.  LSTM Fully Convolutional Networks for Time Series Classification , 2017, IEEE Access.

[11]  Ashley Prater,et al.  Spatiotemporal signal classification via principal components of reservoir states , 2017, Neural Networks.

[12]  Jason Lines,et al.  Time series classification with ensembles of elastic distance measures , 2015, Data Mining and Knowledge Discovery.

[13]  Pattreeya Tanisaro,et al.  Time Series Classification Using Time Warping Invariant Echo State Networks , 2016, 2016 15th IEEE International Conference on Machine Learning and Applications (ICMLA).

[14]  Jürgen Schmidhuber,et al.  Framewise phoneme classification with bidirectional LSTM and other neural network architectures , 2005, Neural Networks.

[15]  C. F. Beckmann,et al.  Tensorial extensions of independent component analysis for multisubject FMRI analysis , 2005, NeuroImage.

[16]  Min Han,et al.  Support Vector Echo-State Machine for Chaotic Time-Series Prediction , 2007, IEEE Transactions on Neural Networks.

[17]  Peter Tiño,et al.  Minimum Complexity Echo State Network , 2011, IEEE Transactions on Neural Networks.

[18]  L. Tucker,et al.  Some mathematical notes on three-mode factor analysis , 1966, Psychometrika.

[19]  Joachim M. Buhmann,et al.  Generative Embedding for Model-Based Classification of fMRI Data , 2011, PLoS Comput. Biol..

[20]  Nitish Srivastava,et al.  Dropout: a simple way to prevent neural networks from overfitting , 2014, J. Mach. Learn. Res..

[21]  George C. Runger,et al.  Time series representation and similarity based on local autopatterns , 2016, Data Mining and Knowledge Discovery.

[22]  Tim Oates,et al.  Time series classification from scratch with deep neural networks: A strong baseline , 2016, 2017 International Joint Conference on Neural Networks (IJCNN).

[23]  Quoc V. Le,et al.  Sequence to Sequence Learning with Neural Networks , 2014, NIPS.

[24]  Wojciech Zaremba,et al.  Recurrent Neural Network Regularization , 2014, ArXiv.

[25]  Oriol Vinyals,et al.  Representation Learning with Contrastive Predictive Coding , 2018, ArXiv.

[26]  Antonello Rizzi,et al.  Short-Term Electric Load Forecasting Using Echo State Networks and PCA Decomposition , 2015, IEEE Access.

[27]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[28]  George Manis,et al.  Heartbeat Time Series Classification With Support Vector Machines , 2009, IEEE Transactions on Information Technology in Biomedicine.

[29]  Herbert Jaeger,et al.  Reservoir computing approaches to recurrent neural network training , 2009, Comput. Sci. Rev..

[30]  Claudio Gallicchio,et al.  Human activity recognition using multisensor data fusion based on Reservoir Computing , 2016, J. Ambient Intell. Smart Environ..

[31]  Tamara G. Kolda,et al.  Tensor Decompositions and Applications , 2009, SIAM Rev..

[32]  Yoshua Bengio,et al.  Maxout Networks , 2013, ICML.

[33]  Philip Chan,et al.  Toward accurate dynamic time warping in linear time and space , 2007, Intell. Data Anal..

[34]  Jun Wang,et al.  Chaotic Time Series Prediction Based on a Novel Robust Echo State Network , 2012, IEEE Transactions on Neural Networks and Learning Systems.

[35]  Huanhuan Chen,et al.  Model Metric Co-Learning for Time Series Classification , 2015, IJCAI.

[36]  John G. Harris,et al.  Minimum mean squared error time series classification using an echo state network prediction model , 2006, 2006 IEEE International Symposium on Circuits and Systems.

[37]  Herbert Jaeger,et al.  The''echo state''approach to analysing and training recurrent neural networks , 2001 .

[38]  Razvan Pascanu,et al.  On the difficulty of training recurrent neural networks , 2012, ICML.

[39]  Yuan-Fang Wang,et al.  Learning a Mahalanobis Distance-Based Dynamic Time Warping Measure for Multivariate Time Series Classification , 2016, IEEE Transactions on Cybernetics.

[40]  James M. W. Brownjohn,et al.  ARMA modelled time-series classification for structural health monitoring of civil infrastructure , 2008 .

[41]  Simone Scardapane,et al.  Prediction of telephone calls load using Echo State Network with exogenous variables , 2015, Neural Networks.

[42]  Bülent Sankur,et al.  Probabilistic sequence clustering with spectral learning , 2014, Digit. Signal Process..

[43]  Claudio Gallicchio,et al.  Echo State Property of Deep Reservoir Computing Networks , 2017, Cognitive Computation.

[44]  Lorenzo Livi,et al.  Determination of the Edge of Criticality in Echo State Networks Through Fisher Information Maximization , 2018, IEEE Transactions on Neural Networks and Learning Systems.

[45]  Yoshua Bengio,et al.  Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation , 2014, EMNLP.

[46]  Tomasz Górecki,et al.  Multivariate time series classification with parametric derivative dynamic time warping , 2015, Expert Syst. Appl..

[47]  Robert Jenssen,et al.  Recurrent Neural Networks for Short-Term Load Forecasting , 2017, SpringerBriefs in Computer Science.

[48]  Benjamin Schrauwen,et al.  Real-time detection of epileptic seizures in animal models using reservoir computing , 2013, Epilepsy Research.

[49]  Robert Jenssen,et al.  Time Series Cluster Kernel for Learning Similarities between Multivariate Time Series with Missing Data , 2017, Pattern Recognit..

[50]  George C. Runger,et al.  Learning a symbolic representation for multivariate time series classification , 2015, Data Mining and Knowledge Discovery.

[51]  Geoffrey E. Hinton,et al.  Speech recognition with deep recurrent neural networks , 2013, 2013 IEEE International Conference on Acoustics, Speech and Signal Processing.

[52]  Razvan Pascanu,et al.  How to Construct Deep Recurrent Neural Networks , 2013, ICLR.

[53]  Robert Jenssen,et al.  Learning similarities between irregularly sampled short multivariate time series from EHRs , 2016 .

[54]  Patrick Schäfer,et al.  Scalable time series classification , 2016, Data Mining and Knowledge Discovery.

[55]  Hossam Faris,et al.  Bidirectional reservoir networks trained using SVM$$+$$+ privileged information for manufacturing process modeling , 2017, Soft Comput..

[56]  Jochen J. Steil,et al.  Time Series Classification in Reservoir- and Model-Space , 2017, Neural Processing Letters.

[57]  Daoqiang Zhang,et al.  (2D)2PCA: Two-directional two-dimensional PCA for efficient face representation and recognition , 2005, Neurocomputing.

[58]  Michael Kampffmeyer,et al.  Learning representations of multivariate time series with missing data , 2019, Pattern Recognit..

[59]  Dianhui Wang,et al.  Randomness in neural networks: an overview , 2017, WIREs Data Mining Knowl. Discov..

[60]  Robert Jenssen,et al.  Bidirectional deep-readout echo state networks , 2017, ESANN.

[61]  Robert Jenssen,et al.  Multiplex visibility graphs to investigate recurrent neural network dynamics , 2016, Scientific Reports.

[62]  X. Yao,et al.  Model-based kernel for efficient time series analysis , 2013, KDD.

[63]  Lorenzo Livi,et al.  Investigating echo state networks dynamics by means of recurrence analysis , 2016, IEEE Trans. Neural Networks Learn. Syst..

[64]  Qiao Li,et al.  AF classification from a short single lead ECG recording: The PhysioNet/computing in cardiology challenge 2017 , 2017, 2017 Computing in Cardiology (CinC).

[65]  Simone Scardapane,et al.  Kafnets: kernel-based non-parametric activation functions for neural networks , 2017, Neural Networks.

[66]  Huanhuan Chen,et al.  Multiobjective Learning in the Model Space for Time Series Classification , 2019, IEEE Transactions on Cybernetics.

[67]  Henry Markram,et al.  Real-Time Computing Without Stable States: A New Framework for Neural Computation Based on Perturbations , 2002, Neural Computation.

[68]  Dave Parry,et al.  Using Echo State Networks to Classify Unscripted, Real-World Punctual Activity , 2015, EANN.

[69]  Friedhelm Schwenker,et al.  Emotion recognition from speech signals via a probabilistic echo-state network , 2015, Pattern Recognit. Lett..

[70]  Quoc V. Le,et al.  Measuring Invariances in Deep Networks , 2009, NIPS.

[71]  Jiabin Wang,et al.  Functional echo state network for time series classification , 2016, Inf. Sci..

[72]  Michael I. Jordan,et al.  On Discriminative vs. Generative Classifiers: A comparison of logistic regression and naive Bayes , 2001, NIPS.

[73]  Yoshua Bengio,et al.  Learning long-term dependencies with gradient descent is difficult , 1994, IEEE Trans. Neural Networks.

[74]  Gunnar Rätsch,et al.  Kernel PCA and De-Noising in Feature Spaces , 1998, NIPS.

[75]  Jochen J. Steil,et al.  Time Series Classification in Reservoir- and Model-Space: A Comparison , 2016, ANNPR.

[76]  Peter Tiño,et al.  Simple Deterministically Constructed Cycle Reservoirs with Regular Jumps , 2012, Neural Computation.

[77]  Germain Forestier,et al.  Deep learning for time series classification: a review , 2018, Data Mining and Knowledge Discovery.

[78]  David Haussler,et al.  Exploiting Generative Models in Discriminative Classifiers , 1998, NIPS.

[79]  Eamonn J. Keogh,et al.  The great time series classification bake off: a review and experimental evaluation of recent algorithmic advances , 2016, Data Mining and Knowledge Discovery.

[80]  Jason Lines,et al.  Time-Series Classification with COTE: The Collective of Transformation-Based Ensembles , 2015, IEEE Transactions on Knowledge and Data Engineering.

[81]  Mikhail Belkin,et al.  Laplacian Eigenmaps for Dimensionality Reduction and Data Representation , 2003, Neural Computation.