TimeAutoML: Autonomous Representation Learning for Multivariate Irregularly Sampled Time Series

Multivariate time series (MTS) data are becoming increasingly ubiquitous in diverse domains, e.g., IoT systems, health informatics, and 5G networks. To obtain an effective representation of MTS data, it is not only essential to consider unpredictable dynamics and highly variable lengths of these data but also important to address the irregularities in the sampling rates of MTS. Existing parametric approaches rely on manual hyperparameter tuning and may cost a huge amount of labor effort. Therefore, it is desirable to learn the representation automatically and efficiently. To this end, we propose an autonomous representation learning approach for multivariate time series (TimeAutoML) with irregular sampling rates and variable lengths. As opposed to previous works, we first present a representation learning pipeline in which the configuration and hyperparameter optimization are fully automatic and can be tailored for various tasks, e.g., anomaly detection, clustering, etc. Next, a negative sample generation approach and an auxiliary classification task are developed and integrated within TimeAutoML to enhance its representation capability. Extensive empirical studies on real-world datasets demonstrate that the proposed TimeAutoML outperforms competing approaches on various tasks by a large margin. In fact, it achieves the best anomaly detection performance among all comparison algorithms on 78 out of all 85 UCR datasets, acquiring up to 20% performance improvement in terms of AUC score.

[1]  Christian S. Jensen,et al.  Outlier Detection for Time Series with Recurrent Autoencoder Ensembles , 2019, IJCAI.

[2]  Yan Liu,et al.  Recurrent Neural Networks for Multivariate Time Series with Missing Values , 2016, Scientific Reports.

[3]  Ali Farhadi,et al.  Unsupervised Deep Embedding for Clustering Analysis , 2015, ICML.

[4]  Michael Flynn,et al.  The UEA multivariate time series classification archive, 2018 , 2018, ArXiv.

[5]  Bo Zong,et al.  Deep Autoencoding Gaussian Mixture Model for Unsupervised Anomaly Detection , 2018, ICLR.

[6]  Ada Wai-Chee Fu,et al.  Efficient time series matching by wavelets , 1999, Proceedings 15th International Conference on Data Engineering (Cat. No.99CB36337).

[7]  Yujie Li,et al.  NAS-Unet: Neural Architecture Search for Medical Image Segmentation , 2019, IEEE Access.

[8]  Hang Xu,et al.  Auto-FPN: Automatic Network Architecture Adaptation for Object Detection Beyond Classification , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[9]  Jianping Yin,et al.  Improved Deep Embedded Clustering with Local Structure Preservation , 2017, IJCAI.

[10]  Vipin Kumar,et al.  Comparative Evaluation of Anomaly Detection Techniques for Sequence Data , 2008, 2008 Eighth IEEE International Conference on Data Mining.

[11]  Wei Cao,et al.  BRITS: Bidirectional Recurrent Imputation for Time Series , 2018, NeurIPS.

[12]  Bin Zhou,et al.  BeatGAN: Anomalous Rhythm Detection using Adversarially Generated Time Series , 2019, IJCAI.

[13]  Yoshua Bengio,et al.  Random Search for Hyper-Parameter Optimization , 2012, J. Mach. Learn. Res..

[14]  Gunnar Rätsch,et al.  SOM-VAE: Interpretable Discrete Representation Learning on Time Series , 2018, ICLR 2018.

[15]  Satya Narayan Shukla,et al.  Interpolation-Prediction Networks for Irregularly Sampled Time Series , 2019, ICLR.

[16]  David Duvenaud,et al.  Neural Ordinary Differential Equations , 2018, NeurIPS.

[17]  Quoc V. Le,et al.  NAS-FPN: Learning Scalable Feature Pyramid Architecture for Object Detection , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[18]  Xuanjing Huang,et al.  Exploring Shared Structures and Hierarchies for Multiple NLP Tasks , 2018, ArXiv.

[19]  Soma Bandyopadhyay,et al.  AutoSensing : Automated Feature Engineering and Learning for Classification Task of Time-Series Sensor Signals , .

[20]  Anton van den Hengel,et al.  Deep Anomaly Detection with Deviation Networks , 2019, KDD.

[21]  Alexander Binder,et al.  Deep Semi-Supervised Anomaly Detection , 2019, ICLR.

[22]  Sahely Bhadra,et al.  Warping Resilient Time Series Embeddings , 2019, ArXiv.

[23]  Zhi-Hua Zhou,et al.  Isolation Forest , 2008, 2008 Eighth IEEE International Conference on Data Mining.

[24]  Jinfeng Yi,et al.  Similarity Preserving Representation Learning for Time Series Clustering , 2019, IJCAI.

[25]  Hans-Peter Kriegel,et al.  LOF: identifying density-based local outliers , 2000, SIGMOD 2000.

[26]  Pierre Gançarski,et al.  A global averaging method for dynamic time warping, with applications to clustering , 2011, Pattern Recognit..

[27]  Min Chi,et al.  Temporal Belief Memory: Imputing Missing Data during RNN Training , 2018, IJCAI.

[28]  David Duvenaud,et al.  Latent Ordinary Differential Equations for Irregularly-Sampled Time Series , 2019, NeurIPS.

[29]  Hao Chen,et al.  Fast Neural Architecture Search of Compact Semantic Segmentation Models via Auxiliary Cells , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[30]  Nando de Freitas,et al.  Taking the Human Out of the Loop: A Review of Bayesian Optimization , 2016, Proceedings of the IEEE.

[31]  Lovekesh Vig,et al.  LSTM-based Encoder-Decoder for Multi-sensor Anomaly Detection , 2016, ArXiv.

[32]  Yeha Lee,et al.  Resource Optimized Neural Architecture Search for 3D Medical Image Segmentation , 2019, MICCAI.

[33]  Adam Prügel-Bennett,et al.  SVM Parameter Optimization using Grid Search and Genetic Algorithm to Improve Classification Performance , 2016 .

[34]  Abubakar Abid,et al.  Autowarp: Learning a Warping Distance from Unlabeled Time Series Using Sequence Autoencoders , 2018, NeurIPS.

[35]  Martin Jaggi,et al.  Unsupervised Scalable Representation Learning for Multivariate Time Series , 2019, NeurIPS.

[36]  Xiaojie Yuan,et al.  E²GAN: End-to-End Generative Adversarial Network for Multivariate Time Series Imputation , 2019, IJCAI.

[37]  Tie-Yan Liu,et al.  Searching Better Architectures for Neural Machine Translation , 2020, IEEE/ACM Transactions on Audio, Speech, and Language Processing.

[38]  Qianli Ma,et al.  Learning Representations for Time Series Clustering , 2019, NeurIPS.

[39]  Luis Gravano,et al.  k-Shape: Efficient and Accurate Clustering of Time Series , 2016, SGMD.

[40]  Ying Zhang,et al.  Multivariate Time Series Imputation with Generative Adversarial Networks , 2018, NeurIPS.

[41]  Mihaela van der Schaar,et al.  Time-series Generative Adversarial Networks , 2019, NeurIPS.

[42]  Homa Karimabadi,et al.  Deep Temporal Clustering : Fully Unsupervised Learning of Time-Domain Features , 2018, ArXiv.

[43]  Bernhard Schölkopf,et al.  Estimating the Support of a High-Dimensional Distribution , 2001, Neural Computation.

[44]  Clu-istos Foutsos,et al.  Fast subsequence matching in time-series databases , 1994, SIGMOD '94.

[45]  Victor Olsavszky,et al.  Time Series Analysis and Forecasting with Automated Machine Learning on a National ICD-10 Database , 2020, International journal of environmental research and public health.

[46]  Lei Chen,et al.  Robust and fast similarity search for moving object trajectories , 2005, SIGMOD '05.