Spectral Invariant Learning for Dynamic Graphs under Distribution Shifts

Dynamic graph neural networks (DyGNNs) currently struggle with handling distribution shifts that are inherent in dynamic graphs. Existing work on DyGNNs with out-of-distribution settings only focuses on the time domain, failing to handle cases involving distribution shifts in the spectral domain. In this paper, we discover that there exist cases with distribution shifts unobservable in the time domain while observable in the spectral domain, and propose to study distribution shifts on dynamic graphs in the spectral domain for the first time. However, this investigation poses two key challenges: i) it is non-trivial to capture different graph patterns that are driven by various frequency components entangled in the spectral domain; and ii) it remains unclear how to handle distribution shifts with the discovered spectral patterns. To address these challenges, we propose Spectral Invariant Learning for Dynamic Graphs under Distribution Shifts (SILD), which can handle distribution shifts on dynamic graphs by capturing and utilizing invariant and variant spectral patterns. Specifically, we first design a DyGNN with Fourier transform to obtain the ego-graph trajectory spectrums, allowing the mixed dynamic graph patterns to be transformed into separate frequency components. We then develop a disentangled spectrum mask to filter graph dynamics from various frequency components and discover the invariant and variant spectral patterns. Finally, we propose invariant spectral filtering, which encourages the model to rely on invariant patterns for generalization under distribution shifts. Experimental results on synthetic and real-world dynamic graph datasets demonstrate the superiority of our method for both node classification and link prediction tasks under distribution shifts.

[1]  Xin Wang,et al.  Unsupervised Graph Neural Architecture Search with Disentangled Self-Supervision , 2024, NeurIPS.

[2]  Xin Wang,et al.  Out-of-Distribution Generalized Dynamic Graph Neural Network with Disentangled Intervention and Invariance Promotion , 2023, ArXiv.

[3]  Xin Wang,et al.  Out-of-Distribution Generalized Dynamic Graph Neural Network for Human Albumin Prediction , 2023, 2023 IEEE International Conference on Medical Artificial Intelligence (MedAI).

[4]  Yi Qin,et al.  Dynamic Heterogeneous Graph Attention Neural Architecture Search , 2023, AAAI.

[5]  Junchi Yan,et al.  EasyDGL: Encode, Train and Interpret for Continuous-time Dynamic Graph Learning , 2023, ArXiv.

[6]  Renjie Liao,et al.  Specformer: Spectral Graph Neural Networks Meet Transformers , 2023, ICLR.

[7]  Xiao Wang,et al.  A Survey on Spectral Graph Neural Networks , 2023, ArXiv.

[8]  Pang Wei Koh,et al.  Wild-Time: A Benchmark of in-the-Wild Distribution Shift over Time , 2022, NeurIPS.

[9]  T. Suzumura,et al.  Learnable Spectral Wavelets on Dynamic Graphs to Capture Global Interactions , 2022, AAAI.

[10]  J. Leskovec,et al.  ROLAND: Graph Learning Framework for Dynamic Graphs , 2022, KDD.

[11]  Yi Qin,et al.  NAS-Bench-Graph: Benchmarking Graph Neural Architecture Search , 2022, NeurIPS.

[12]  M. Zitnik,et al.  Self-Supervised Contrastive Pre-Training For Time Series via Time-Frequency Consistency , 2022, NeurIPS.

[13]  Muhan Zhang,et al.  How Powerful are Spectral Graph Neural Networks , 2022, International Conference on Machine Learning.

[14]  Min Lin,et al.  Causal Representation Learning for Out-of-Distribution Recommendation , 2022, WWW.

[15]  Xin Wang,et al.  Learning to Solve Travelling Salesman Problem with Hardness-adaptive Curriculum , 2022, AAAI.

[16]  I. Rish,et al.  WOODS: Benchmarks for Out-of-Distribution Generalization in Time Series Tasks , 2022, Trans. Mach. Learn. Res..

[17]  Wenwu Zhu,et al.  Out-Of-Distribution Generalization on Graphs: A Survey , 2022, ArXiv.

[18]  Junchi Yan,et al.  Handling Distribution Shifts on Graphs: An Invariance Perspective , 2022, ICLR.

[19]  Xiangnan He,et al.  Discovering Invariant Rationales for Graph Neural Networks , 2022, ICLR.

[20]  Tian Zhou,et al.  FEDformer: Frequency Enhanced Decomposed Transformer for Long-term Series Forecasting , 2022, ICML.

[21]  James Y. Zou,et al.  Improving Out-of-Distribution Robustness via Selective Augmentation , 2022, ICML.

[22]  Peng Cui,et al.  Revisiting Transformation Invariant Geometric Deep Learning: Are Initial Representations All You Need? , 2021, arXiv.org.

[23]  Xin Wang,et al.  OOD-GNN: Out-of-Distribution Generalized Graph Neural Network , 2021, IEEE Transactions on Knowledge and Data Engineering.

[24]  Peng Cui,et al.  Generalizing Graph Neural Networks on Out-of-Distribution Graphs , 2021, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[25]  Paul Bogdan,et al.  Multiwavelet-based Operator Learning for Differential Equations , 2021, NeurIPS.

[26]  Peng Cui,et al.  Towards Out-Of-Distribution Generalization: A Survey , 2021, ArXiv.

[27]  Nalini Venkatasubramanian,et al.  Environment Agnostic Invariant Risk Minimization for Classification of Sequential Datasets , 2021, KDD.

[28]  Sinno Jialin Pan,et al.  AdaRNN: Adaptive Learning and Forecasting of Time Series , 2021, CIKM.

[29]  Bryan Perozzi,et al.  Shift-Robust GNNs: Overcoming the Limitations of Localized Graph Training data , 2021, NeurIPS.

[30]  Irwin King,et al.  Discrete-time Temporal Network Embedding via Implicit Hierarchical Learning in Hyperbolic Space , 2021, KDD.

[31]  Qi Tian,et al.  A Fourier-based Framework for Domain Generalization , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[32]  Jure Leskovec,et al.  TEDIC: Neural Modeling of Behavioral Patterns in Dynamic Social Interaction Networks , 2021, WWW.

[33]  Wenwu Zhu,et al.  AutoGL: A Library for Automated Graph Learning , 2021, ArXiv.

[34]  Philip S. Yu,et al.  Hyperbolic Variational Graph Neural Network for Modeling Dynamic Graphs , 2021, AAAI.

[35]  Chen Change Loy,et al.  Domain Generalization: A Survey , 2021, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[36]  Xiaowen Dong,et al.  Interpretable Stability Bounds for Spectral Graph Filters , 2021, ICML.

[37]  J. Leskovec,et al.  Inductive Representation Learning in Temporal Networks via Causal Anonymous Walks , 2021, ICLR.

[38]  Chen Change Loy,et al.  Focal Frequency Loss for Image Reconstruction and Synthesis , 2020, 2021 IEEE/CVF International Conference on Computer Vision (ICCV).

[39]  Qi Zhang,et al.  Spectral Temporal Graph Neural Network for Multivariate Time-series Forecasting , 2020, NeurIPS.

[40]  Jackie Chi Kit Cheung,et al.  TeMP: Temporal Message Passing for Temporal Knowledge Graph Completion , 2020, EMNLP.

[41]  Tian Jin,et al.  Community detection and co-author recommendation in co-author networks , 2020, International Journal of Machine Learning and Cybernetics.

[42]  Jozef Barunik,et al.  Dynamic Networks in Large Financial and Economic Systems , 2020, SSRN Electronic Journal.

[43]  Huzefa Rangwala,et al.  Dynamic Knowledge Graph based Multi-Event Forecasting , 2020, KDD.

[44]  Davide Eynard,et al.  Temporal Graph Networks for Deep Learning on Dynamic Graphs , 2020, ArXiv.

[45]  Haifeng Chen,et al.  Structural Temporal Graph Neural Networks for Anomaly Detection in Dynamic Graphs , 2020, CIKM.

[46]  Katarzyna Musial,et al.  Foundations and Modeling of Dynamic Networks Using Dynamic Graph Neural Networks: A Survey , 2020, IEEE Access.

[47]  Steven L. Brunton,et al.  From Fourier to Koopman: Spectral Methods for Long-term Time Series Prediction , 2020, J. Mach. Learn. Res..

[48]  Tommi S. Jaakkola,et al.  Invariant Rationalization , 2020, ICML.

[49]  Aaron C. Courville,et al.  Out-of-Distribution Generalization via Risk Extrapolation (REx) , 2020, ICML.

[50]  Da Xu,et al.  Inductive Representation Learning on Temporal Graphs , 2020, ICLR.

[51]  Kush R. Varshney,et al.  Invariant Risk Minimization Games , 2020, ICML.

[52]  Liang Gou,et al.  DySAT: Deep Neural Representation Learning on Dynamic Graphs via Self-Attention Networks , 2020, WSDM.

[53]  Natalia Gimelshein,et al.  PyTorch: An Imperative Style, High-Performance Deep Learning Library , 2019, NeurIPS.

[54]  Tatsunori B. Hashimoto,et al.  Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization , 2019, ArXiv.

[55]  R. Kohn,et al.  Spectral Subsampling MCMC for Stationary Time Series , 2019, ICML.

[56]  Mingyuan Zhou,et al.  Variational Graph Recurrent Neural Networks , 2019, NeurIPS.

[57]  Wenwu Zhu,et al.  Fates of Microscopic Social Ecosystems: Keep Alive or Dead? , 2019, KDD.

[58]  David Lopez-Paz,et al.  Invariant Risk Minimization , 2019, ArXiv.

[59]  Xueqi Cheng,et al.  Graph Wavelet Neural Network , 2019, ICLR.

[60]  Jure Leskovec,et al.  Hierarchical Temporal Convolutional Networks for Dynamic Recommender Systems , 2019, WWW.

[61]  Yanfang Ye,et al.  Heterogeneous Graph Attention Network , 2019, WWW.

[62]  Jan Eric Lenssen,et al.  Fast Graph Representation Learning with PyTorch Geometric , 2019, ArXiv.

[63]  Charles E. Leisersen,et al.  EvolveGCN: Evolving Graph Convolutional Networks for Dynamic Graphs , 2019, AAAI.

[64]  Wenwu Zhu,et al.  Deep Learning on Graphs: A Survey , 2018, IEEE Transactions on Knowledge and Data Engineering.

[65]  Pietro Liò,et al.  Graph Attention Networks , 2017, ICLR.

[66]  Jian Pei,et al.  Community Preserving Network Embedding , 2017, AAAI.

[67]  Xavier Bresson,et al.  Structured Sequence Modeling with Graph Convolutional Recurrent Networks , 2016, ICONIP.

[68]  Max Welling,et al.  Variational Graph Auto-Encoders , 2016, ArXiv.

[69]  Xavier Bresson,et al.  Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering , 2016, NIPS.

[70]  Angus Graeme Forbes,et al.  TimeArcs: Visualizing Fluctuations in Dynamic Networks , 2016, Comput. Graph. Forum.

[71]  Yang Song,et al.  An Overview of Microsoft Academic Service (MAS) and Applications , 2015, WWW.

[72]  Yoshua Bengio,et al.  Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation , 2014, EMNLP.

[73]  J. Dean,et al.  Efficient Estimation of Word Representations in Vector Space , 2013, ICLR.

[74]  Jimeng Sun,et al.  Cross-domain collaboration recommendation , 2012, KDD.

[75]  Amol Deshpande,et al.  Managing large dynamic graphs efficiently , 2012, SIGMOD Conference.

[76]  Jie Tang,et al.  ArnetMiner: extraction and mining of academic social networks , 2008, KDD.

[77]  Bruce C Hansen,et al.  Structural sparseness and spatial phase alignment in natural scenes. , 2007, Journal of the Optical Society of America. A, Optics, image science, and vision.

[78]  S. Hochreiter,et al.  Long Short-Term Memory , 1997, Neural Computation.

[79]  William N. Goetzmann,et al.  Survivorship Bias in Performance Studies , 1992 .

[80]  Kathryn B. Laskey,et al.  Stochastic blockmodels: First steps , 1983 .

[81]  R. Berk.  An introduction to sample selection bias in sociological data. , 1983 .

[82]  L N Piotrowski,et al.  A Demonstration of the Visual Importance and Flexibility of Spatial-Frequency Amplitude and Phase , 1982, Perception.

[83]  A.V. Oppenheim,et al.  The importance of phase in signals , 1980, Proceedings of the IEEE.

[84]  Jae S. Lim,et al.  Phase in speech and pictures , 1979, ICASSP.

[85]  J. Tukey,et al.  An algorithm for the machine calculation of complex Fourier series , 1965 .

[86]  Zeyang Zhang,et al.  Dynamic Graph Neural Networks Under Spatio-Temporal Distribution Shift , 2022, NeurIPS.

[87]  Xin Wang,et al.  Learning Invariant Graph Representations for Out-of-Distribution Generalization , 2022, NeurIPS.

[88]  Tongliang Liu,et al.  Invariance Principle Meets Out-of-Distribution Generalization on Graphs , 2022, ArXiv.

[89]  P. Xie,et al.  Graph Neural Architecture Search Under Distribution Shifts , 2022, ICML.

[90]  Yu Guang Wang,et al.  Well-Conditioned Spectral Transforms for Dynamic Graph Representation , 2022, LoG.

[91]  X. Chen,et al.  Learnable Encoder-Decoder Architecture for Dynamic Graph: A Survey , 2022 .

[92]  J. Choo,et al.  Reversible Instance Normalization for Accurate Time-Series Forecasting against Distribution Shift , 2022, ICLR.

[93]  Wenwu Zhu,et al.  Graph Differentiable Architecture Search with Structure Learning , 2021, NeurIPS.

[94]  Yinglong Xia,et al.  Dynamic Graph Representation Learning via Graph Transformer Networks , 2021, ArXiv.

[95]  Furong Huang,et al.  A Closer Look at Distribution Shifts and Out-of-Distribution Generalization on Graphs , 2021 .