Networked Time Series Imputation via Position-aware Graph Enhanced Variational Autoencoders

Multivariate time series (MTS) imputation is a widely studied problem in recent years. Existing methods can be divided into two main groups, including (1) deep recurrent or generative models that primarily focus on time series features, and (2) graph neural networks (GNNs) based models that utilize the topological information from the inherent graph structure of MTS as relational inductive bias for imputation. Nevertheless, these methods either neglect topological information or assume the graph structure is fixed and accurately known. Thus, they fail to fully utilize the graph dynamics for precise imputation in more challenging MTS data such as networked time series (NTS), where the underlying graph is constantly changing and might have missing edges. In this paper, we propose a novel approach to overcome these limitations. First, we define the problem of imputation over NTS which contains missing values in both node time series features and graph structures. Then, we design a new model named PoGeVon which leverages variational autoencoder (VAE) to predict missing values over both node time series features and graph structures. In particular, we propose a new node position embedding based on random walk with restart (RWR) in the encoder with provable higher expressive power compared with message-passing based graph neural networks (GNNs). We further design a decoder with 3-stage predictions from the perspective of multi-task learning to impute missing values in both time series and graph structures reciprocally. Experiment results demonstrate the effectiveness of our model over baselines.

[1]  T. Abdelzaher,et al.  Dissecting Cross-Layer Dependency Inference on Multi-Layered Inter-Dependent Networks , 2022, CIKM.

[2]  Jianmin Wang,et al.  TimesNet: Temporal 2D-Variation Modeling for General Time Series Analysis , 2022, ICLR.

[3]  Hanghang Tong,et al.  Retrieval Based Time Series Forecasting , 2022, ArXiv.

[4]  Vetle I. Torvik,et al.  Meta-Learned Metrics over Multi-Evolution Temporal Graphs , 2022, KDD.

[5]  Ravi Teja Bhupatiraju,et al.  Examining the COVID-19 case growth rate due to visitor vs. local mobility in the United States using machine learning , 2022, Scientific Reports.

[6]  C. Alippi,et al.  Learning to Reconstruct Missing Data from Spatiotemporal Graphs with Sparse Observations , 2022, Neural Information Processing Systems.

[7]  Haoteng Yin,et al.  Equivariant and Stable Positional Encoding for More Powerful Graph Neural Networks , 2022, ICLR.

[8]  Y. Liu,et al.  SAITS: Self-Attention-based Imputation for Time Series , 2022, Expert Syst. Appl..

[9]  Philip S. Yu,et al.  Graph Neural Networks for Graphs with Heterophily: A Survey , 2022, ArXiv.

[10]  Omkar Bhalerao,et al.  Large Scale Learning on Non-Homophilous Graphs: New Benchmarks and Strong Simple Methods , 2021, NeurIPS.

[11]  Jiang Bian,et al.  HIST: A Graph-based Framework for Stock Trend Forecasting via Mining Concept-Oriented Shared Information , 2021, ArXiv.

[12]  Vijay Prakash Dwivedi,et al.  Graph Neural Networks with Learnable Structural and Positional Representations , 2021, ICLR.

[13]  C. Alippi,et al.  Filling the G_ap_s: Multivariate Time Series Imputation by Graph Neural Networks , 2021, ICLR.

[14]  Dongqi Fu,et al.  SDG: A Simplified and Dynamic Graph Neural Network , 2021, SIGIR.

[15]  Jianwei Yin,et al.  Generative Semi-supervised Learning for Multivariate Time Series Imputation , 2021, AAAI.

[16]  Hanghang Tong,et al.  Dynamic Knowledge Graph Alignment , 2021, AAAI Conference on Artificial Intelligence.

[17]  Yulia R. Gel,et al.  Z-GCNETs: Time Zigzags at Graph Convolutional Networks for Time Series Forecasting , 2021, ICML.

[18]  Hanghang Tong,et al.  BRIGHT: A Bridging Algorithm for Network Alignment , 2021, WWW.

[19]  Yada Zhu,et al.  Network of Tensor Time Series , 2021, WWW.

[20]  Hanghang Tong,et al.  HDMI: High-order Deep Multiplex Infomax , 2021, WWW.

[21]  Chen Wang,et al.  Time Series Data Imputation: A Survey on Deep Learning Approaches , 2020, ArXiv.

[22]  Philip S. Yu,et al.  Dynamic Graph Collaborative Filtering , 2020, 2020 IEEE International Conference on Data Mining (ICDM).

[23]  Ryan A. Rossi,et al.  Graph Neural Networks with Heterophily , 2020, AAAI.

[24]  Yi Tay,et al.  Efficient Transformers: A Survey , 2020, ACM Comput. Surv..

[25]  Michalis Vazirgiannis,et al.  Transfer Graph Neural Networks for Pandemic Forecasting , 2020, AAAI.

[26]  Jure Leskovec,et al.  Distance Encoding: Design Provably More Powerful Neural Networks for Graph Representation Learning , 2020, NeurIPS.

[27]  Luyang Liu,et al.  Examining COVID-19 Forecasting using Spatio-Temporal Graph Neural Networks , 2020, ArXiv.

[28]  Jingrui He,et al.  Local Motif Clustering on Time-Evolving Graphs , 2020, KDD.

[29]  Sibo Wang,et al.  Personalized PageRank to a Target Node, Revisited , 2020, KDD.

[30]  Davide Eynard,et al.  Temporal Graph Networks for Deep Learning on Dynamic Graphs , 2020, ArXiv.

[31]  Olgica Milenkovic,et al.  Adaptive Universal Generalized PageRank Graph Neural Network , 2020, ICLR.

[32]  Christopher K. I. Williams,et al.  VAEs in the Presence of Missing Data , 2020, ArXiv.

[33]  Han Fang,et al.  Linformer: Self-Attention with Linear Complexity , 2020, ArXiv.

[34]  Xiaojun Chang,et al.  Connecting the Dots: Multivariate Time Series Forecasting with Graph Neural Networks , 2020, KDD.

[35]  Gaofeng Meng,et al.  Spatio-Temporal Graph Structure Learning for Traffic Forecasting , 2020, AAAI.

[36]  Da Xu,et al.  Inductive Representation Learning on Temporal Graphs , 2020, ICLR.

[37]  Minnan Luo,et al.  Graph Representation Learning via Graphical Mutual Information Maximization , 2020, WWW.

[38]  Lukasz Kaiser,et al.  Reformer: The Efficient Transformer , 2020, ICLR.

[39]  Natalia Gimelshein,et al.  PyTorch: An Imperative Style, High-Performance Deep Learning Library , 2019, NeurIPS.

[40]  Da Xu,et al.  Self-attention with Functional Time Representation Learning , 2019, NeurIPS.

[41]  Mingyuan Zhou,et al.  Variational Graph Recurrent Neural Networks , 2019, NeurIPS.

[42]  Xiaojie Yuan,et al.  E²GAN: End-to-End Generative Adversarial Network for Multivariate Time Series Imputation , 2019, IJCAI.

[43]  Sanjay Thakur,et al.  Time2Vec: Learning a Vector Representation of Time , 2019, ArXiv.

[44]  Gunnar Rätsch,et al.  GP-VAE: Deep Probabilistic Time Series Imputation , 2019, AISTATS.

[45]  Jure Leskovec,et al.  Position-aware Graph Neural Networks , 2019, ICML.

[46]  Xiang Ren,et al.  Recurrent Event Network: Autoregressive Structure Inference over Temporal Knowledge Graphs , 2019, EMNLP.

[47]  Jure Leskovec,et al.  How Powerful are Graph Neural Networks? , 2018, ICLR.

[48]  Pablo M. Olmos,et al.  Handling Incomplete Heterogeneous Data using VAEs , 2018, Pattern Recognit..

[49]  Mihaela van der Schaar,et al.  GAIN: Missing Data Imputation using Generative Adversarial Nets , 2018, ICML.

[50]  Dmitry Vetrov,et al.  Variational Autoencoder with Arbitrary Conditioning , 2018, ICLR.

[51]  Wei Cao,et al.  BRITS: Bidirectional Recurrent Imputation for Time Series , 2018, NeurIPS.

[52]  Guillaume Desjardins,et al.  Understanding disentangling in β-VAE , 2018, ArXiv.

[53]  Xiao-Ming Wu,et al.  Deeper Insights into Graph Convolutional Networks for Semi-Supervised Learning , 2018, AAAI.

[54]  Zhanxing Zhu,et al.  Spatio-temporal Graph Convolutional Neural Network: A Deep Learning Framework for Traffic Forecasting , 2017, IJCAI.

[55]  Cyrus Shahabi,et al.  Diffusion Convolutional Recurrent Neural Network: Data-Driven Traffic Forecasting , 2017, ICLR.

[56]  Lukasz Kaiser,et al.  Attention is All you Need , 2017, NIPS.

[57]  Samuel S. Schoenholz,et al.  Neural Message Passing for Quantum Chemistry , 2017, ICML.

[58]  Alexander A. Alemi,et al.  Deep Variational Information Bottleneck , 2017, ICLR.

[59]  Max Welling,et al.  Variational Graph Auto-Encoders , 2016, ArXiv.

[60]  Christopher Burgess,et al.  beta-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework , 2016, ICLR 2016.

[61]  Max Welling,et al.  Semi-Supervised Classification with Graph Convolutional Networks , 2016, ICLR.

[62]  Frank Hutter,et al.  SGDR: Stochastic Gradient Descent with Warm Restarts , 2016, ICLR.

[63]  Tianrui Li,et al.  ST-MVL: Filling Missing Values in Geo-Sensory Time Series Data , 2016, IJCAI.

[64]  Carl Doersch,et al.  Tutorial on Variational Autoencoders , 2016, ArXiv.

[65]  G. P. Nason,et al.  Modelling, Detrending and Decorrelation of Network Time Series , 2016, 1603.03221.

[66]  Ming Li,et al.  Forecasting Fine-Grained Air Quality Based on Big Data , 2015, KDD.

[67]  Yue Zhang,et al.  Deep Learning for Event-Driven Stock Prediction , 2015, IJCAI.

[68]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[69]  Yoshua Bengio,et al.  Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling , 2014, ArXiv.

[70]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[71]  Max Welling,et al.  Auto-Encoding Variational Bayes , 2013, ICLR.

[72]  Pascal Frossard,et al.  The emerging field of signal processing on graphs: Extending high-dimensional data analysis to networks and other irregular domains , 2012, IEEE Signal Processing Magazine.

[73]  Patrick Royston,et al.  Multiple imputation using chained equations: Issues and guidance for practice , 2011, Statistics in medicine.

[74]  Christos Faloutsos,et al.  Fast Random Walk with Restart and Its Applications , 2006, Sixth International Conference on Data Mining (ICDM'06).

[75]  Naftali Tishby,et al.  The information bottleneck method , 2000, ArXiv.

[76]  Raymond W. Yeung,et al.  A new outlook of Shannon's information measures , 1991, IEEE Trans. Inf. Theory.

[77]  P. Young,et al.  Time series analysis, forecasting and control , 1972, IEEE Transactions on Automatic Control.

[78]  W. J. McGill Multivariate information transmission , 1954, Trans. IRE Prof. Group Inf. Theory.

[79]  Ying Zhang,et al.  Multivariate Time Series Imputation with Generative Adversarial Networks , 2018, NeurIPS.

[80]  Hanghang Tong,et al.  Fast Mining of a Network of Coevolving Time Series , 2015, SDM.

[81]  Pravin Varaiya,et al.  Freeway Performance Measurement System: Mining Loop Detector Data , 2001 .

[82]  J. Shao,et al.  Nearest Neighbor Imputation for Survey Data , 2000 .