Integration of residual network and convolutional neural network along with various activation functions and global pooling for time series classification

Abstract In this paper, we devise a hybrid scheme, which integrates residual network with convolutional neural network, for time series classification. In the devised method, the architecture of network is constructed by facilitating a residual learning block at the first three convolutional layers to combine the strength of both methods. Further, different activation functions are used in different layers to achieve a decent abstraction. Additionally, to alleviate overfitting, the pooling operation is removed and the features are fed into a global average pooling instead of a fully connected layer. The resulting scheme requires no heavy preprocessing of raw data or feature crafting, thus could be easily deployed. To evaluate our method, we test it on 44 benchmark datasets and compare its performance with related methods. The results show that our method can deliver competitive performance among state-of-the-art methods.

[1]  Jian Pei,et al.  A brief survey on sequence classification , 2010, SKDD.

[2]  Jian Sun,et al.  Identity Mappings in Deep Residual Networks , 2016, ECCV.

[3]  George C. Runger,et al.  A Bag-of-Features Framework to Classify Time Series , 2013, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[4]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[5]  Patrick Schäfer,et al.  Towards Time Series Classification without Human Preprocessing , 2014, MLDM.

[6]  Yi Zheng,et al.  Time Series Classification Using Multi-Channels Deep Convolutional Neural Networks , 2014, WAIM.

[7]  Hui Ding,et al.  Querying and mining of time series data: experimental comparison of representations and distance measures , 2008, Proc. VLDB Endow..

[8]  MengChu Zhou,et al.  An Inherently Nonnegative Latent Factor Model for High-Dimensional and Sparse Matrices from Industrial Applications , 2018, IEEE Transactions on Industrial Informatics.

[9]  Eamonn J. Keogh,et al.  Fast Shapelets: A Scalable Algorithm for Discovering Time Series Shapelets , 2013, SDM.

[10]  Douglas D. Moesel,et al.  CONSTRUCT VALIDITY OF AN OBJECTIVE (ENTROPY) CATEGORICAL MEASURE OF DIVERSIFICATION STRATEGY , 1993 .

[11]  Jason Lines,et al.  Time-Series Classification with COTE: The Collective of Transformation-Based Ensembles , 2015, IEEE Trans. Knowl. Data Eng..

[12]  Sergey Ioffe,et al.  Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning , 2016, AAAI.

[13]  Jürgen Schmidhuber,et al.  Training Very Deep Networks , 2015, NIPS.

[14]  Patrick Schäfer.  The BOSS is concerned with time series classification in the presence of noise , 2014, Data Mining and Knowledge Discovery.

[15]  Geoffrey E. Hinton,et al.  On the importance of initialization and momentum in deep learning , 2013, ICML.

[16]  Tim Oates,et al.  Time series classification from scratch with deep neural networks: A strong baseline , 2016, 2017 International Joint Conference on Neural Networks (IJCNN).

[17]  Li Wei,et al.  Experiencing SAX: a novel symbolic representation of time series , 2007, Data Mining and Knowledge Discovery.

[18]  MengChu Zhou,et al.  Temporal Pattern-Aware QoS Prediction via Biased Non-Negative Latent Factorization of Tensors , 2020, IEEE Transactions on Cybernetics.

[19]  Jason Lines,et al.  Transformation Based Ensembles for Time Series Classification , 2012, SDM.

[20]  Lars Schmidt-Thieme,et al.  Learning time-series shapelets , 2014, KDD.

[21]  Eamonn J. Keogh,et al.  Data Mining a Trillion Time Series Subsequences Under Dynamic Time Warping , 2013, IJCAI.

[22]  Eamonn J. Keogh,et al.  The great time series classification bake off: a review and experimental evaluation of recent algorithmic advances , 2016, Data Mining and Knowledge Discovery.

[23]  MengChu Zhou,et al.  A Nonnegative Latent Factor Model for Large-Scale Sparse Matrices in Recommender Systems via Alternating Direction Method , 2016, IEEE Transactions on Neural Networks and Learning Systems.

[24]  Geoffrey E. Hinton,et al.  Rectified Linear Units Improve Restricted Boltzmann Machines , 2010, ICML.

[25]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[26]  Patrick Schäfer,et al.  Scalable time series classification , 2016, Data Mining and Knowledge Discovery.

[27]  Eamonn J. Keogh,et al.  A Complexity-Invariant Distance Measure for Time Series , 2011, SDM.

[28]  Gang Zeng,et al.  Weighted residuals for very deep networks , 2016, 2016 3rd International Conference on Systems and Informatics (ICSAI).

[29]  George C. Runger,et al.  A time series forest for classification and feature extraction , 2013, Inf. Sci..

[30]  MengChu Zhou,et al.  Incorporation of Efficient Second-Order Solvers Into Latent Factor Models for Accurate Prediction of Missing QoS Data , 2018, IEEE Transactions on Cybernetics.

[31]  Shuai Li,et al.  Symmetric and Nonnegative Latent Factor Models for Undirected, High-Dimensional, and Sparse Networks in Industrial Applications , 2017, IEEE Transactions on Industrial Informatics.

[32]  Luis Gravano,et al.  k-Shape: Efficient and Accurate Clustering of Time Series , 2015, SIGMOD Conference.

[33]  Sergey Malinchik,et al.  SAX-VSM: Interpretable Time Series Classification Using SAX and Vector Space Model , 2013, 2013 IEEE 13th International Conference on Data Mining.

[34]  Jian Sun,et al.  Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[35]  Yoshua Bengio,et al.  Deep Sparse Rectifier Neural Networks , 2011, AISTATS.

[36]  Yannis Manolopoulos,et al.  Feature-based classification of time-series data , 2001 .

[37]  Jason Lines,et al.  Time series classification with ensembles of elastic distance measures , 2015, Data Mining and Knowledge Discovery.

[38]  Eamonn J. Keogh,et al.  Experimental comparison of representation methods and distance measures for time series data , 2010, Data Mining and Knowledge Discovery.