LSTM Fully Convolutional Networks for Time Series Classification

Fully convolutional neural networks (FCNs) have been shown to achieve the state-of-the-art performance on the task of classifying time series sequences. We propose the augmentation of fully convolutional networks with long short term memory recurrent neural network (LSTM RNN) sub-modules for time series classification. Our proposed models significantly enhance the performance of fully convolutional networks with a nominal increase in model size and require minimal preprocessing of the data set. The proposed long short term memory fully convolutional network (LSTM-FCN) achieves the state-of-the-art performance compared with others. We also explore the usage of attention mechanism to improve time series classification with the attention long short term memory fully convolutional network (ALSTM-FCN). The attention mechanism allows one to visualize the decision process of the LSTM cell. Furthermore, we propose refinement as a method to enhance the performance of trained models. An overall analysis of the performance of our model is provided and compared with other techniques.

[1]  Gary King,et al.  Logistic Regression in Rare Events Data , 2001, Political Analysis.

[2]  Layth C. Alwan,et al.  Time-Series Modeling for Statistical Process Control , 1988 .

[3]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[4]  Yoshua Bengio,et al.  How transferable are features in deep neural networks? , 2014, NIPS.

[5]  Mohammed Waleed Kadous,et al.  Temporal classification: extending the classification paradigm to multivariate time series , 2002 .

[6]  Li Wei,et al.  Semi-supervised time series classification , 2006, KDD '06.

[7]  Razvan Pascanu,et al.  How to Construct Deep Recurrent Neural Networks , 2013, ICLR.

[8]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[9]  Nitish Srivastava,et al.  Dropout: a simple way to prevent neural networks from overfitting , 2014, J. Mach. Learn. Res..

[10]  Jason Lines,et al.  Time-Series Classification with COTE: The Collective of Transformation-Based Ensembles , 2015, IEEE Transactions on Knowledge and Data Engineering.

[11]  Qiang Chen,et al.  Network In Network , 2013, ICLR.

[12]  Yoshua Bengio,et al.  Neural Machine Translation by Jointly Learning to Align and Translate , 2014, ICLR.

[13]  Jason Lines,et al.  Time series classification with ensembles of elastic distance measures , 2015, Data Mining and Knowledge Discovery.

[14]  Tim Oates,et al.  Time series classification from scratch with deep neural networks: A strong baseline , 2016, 2017 International Joint Conference on Neural Networks (IJCNN).

[15]  Patrick Schäfer,et al.  Scalable time series classification , 2016, Data Mining and Knowledge Discovery.

[16]  Anthony J. Bagnall,et al.  Binary Shapelet Transform for Multiclass Time Series Classification , 2015, Trans. Large Scale Data Knowl. Centered Syst..

[17]  Jian Sun,et al.  Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[18]  Gregory D. Hager,et al.  Temporal Convolutional Networks: A Unified Approach to Action Segmentation , 2016, ECCV Workshops.

[19]  Yannis Manolopoulos,et al.  Feature-based classification of time-series data , 2001 .

[20]  Kilian Q. Weinberger,et al.  Snapshot Ensembles: Train 1, get M for free , 2017, ICLR.

[21]  Wei Wang,et al.  Time series analysis for psychological research: examining and forecasting change , 2015, Front. Psychol..

[22]  George C. Runger,et al.  A Bag-of-Features Framework to Classify Time Series , 2013, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[23]  Peng Shi,et al.  Intelligent Tracking Control for a Class of Uncertain High-Order Nonlinear Systems , 2016, IEEE Transactions on Neural Networks and Learning Systems.

[24]  R. Buizza,et al.  Wind Power Density Forecasting Using Ensemble Predictions and Time Series Models , 2009, IEEE Transactions on Energy Conversion.

[25]  Patrick Schäfer.  The BOSS is concerned with time series classification in the presence of noise , 2014, Data Mining and Knowledge Discovery.

[26]  Sergey Ioffe,et al.  Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift , 2015, ICML.

[27]  B. Koley,et al.  An ensemble system for automatic sleep stage classification using single channel EEG signal , 2012, Comput. Biol. Medicine.

[28]  Brahim Chaib-draa,et al.  Parametric Exponential Linear Unit for Deep Convolutional Neural Networks , 2016, 2017 16th IEEE International Conference on Machine Learning and Applications (ICMLA).

[29]  Li Wei,et al.  Experiencing SAX: a novel symbolic representation of time series , 2007, Data Mining and Knowledge Discovery.

[30]  Peter Xiaoping Liu,et al.  Adaptive Intelligent Control of Nonaffine Nonlinear Time-Delay Systems With Dynamic Uncertainties , 2017, IEEE Transactions on Systems, Man, and Cybernetics: Systems.

[31]  Yixin Chen,et al.  Multi-Scale Convolutional Neural Networks for Time Series Classification , 2016, ArXiv.

[32]  Ulf Leser,et al.  Fast and Accurate Time Series Classification with WEASEL , 2017, CIKM.

[33]  Alex Graves,et al.  Supervised Sequence Labelling with Recurrent Neural Networks , 2012, Studies in Computational Intelligence.