Time-frequency deep metric learning for multivariate time series classification

Abstract Multivariate time series (MTS) data exist in various fields of studies and MTS classification is an important research topic in the machine learning community. Researchers have proposed many MTS classification models over the years and the distance-based methods along with nearest neighbor classifier achieve good performance. However, the current methods mainly focus on defining distance metric on time-domain of MTS and ignore frequency information. Besides, these methods usually define the same linear distance metric for different datasets, which is not suitable for capturing the nonlinear relationship of MTS and degrades the discriminative power of the distance metric. In this paper, we propose a time–frequency deep metric learning (TFDM) approach for MTS classification. The multilevel discrete wavelet decomposition is first adopted to decompose an MTS into a group of sub-MTS so as to extract multilevel time–frequency representations. Then, a deep convolutional neural network is developed for each level to learn level-specific nonlinear features and a metric learning layer is added on the top of the network to learn the semantic similarity of MTS. Moreover, a cross-level consistency regularization term is designed to encourage the distance metrics of different levels to be consistent for capturing the correlations among different levels. Finally, we use 1-nearest neighbor to classify MTS according to the learned distance metrics. Extensive experiments on 18 benchmark datasets show the effectiveness of our approach.