A Survey on Embedding Dynamic Graphs

Embedding static graphs in low-dimensional vector spaces plays a key role in network analytics and inference, supporting applications like node classification, link prediction, and graph visualization. However, many real-world networks present dynamic behavior, including topological evolution, feature evolution, and diffusion. Therefore, several methods for embedding dynamic graphs have been proposed to learn network representations over time, facing novel challenges, such as time-domain modeling, temporal features to be captured, and the temporal granularity to be embedded. In this survey, we overview dynamic graph embedding, discussing its fundamentals and the recent advances developed so far. We introduce the formal definition of dynamic graph embedding, focusing on the problem setting and introducing a novel taxonomy for dynamic graph embedding input and output. We further explore different dynamic behaviors that may be encompassed by embeddings, classifying by topological evolution, feature evolution, and processes on networks. Afterward, we describe existing techniques and propose a taxonomy for dynamic graph embedding techniques based on algorithmic approaches, from matrix and tensor factorization to deep learning, random walks, and temporal point processes. We also elucidate main applications, including dynamic link prediction, anomaly detection, and diffusion prediction, and we further state some promising research directions in the area.

[1]  Cesare Alippi,et al.  Graph Neural Networks in TensorFlow and Keras with Spektral , 2020, IEEE Comput. Intell. Mag..

[2]  Ciro Cattuto,et al.  DyANE: Dynamics-aware node embedding for temporal networks , 2019, ArXiv.

[3]  Aynaz Taheri,et al.  Predictive Temporal Embedding of Dynamic Graphs , 2019, 2019 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM).

[4]  Yue Gao,et al.  Hypergraph Neural Networks , 2018, AAAI.

[5]  Luca Cardelli,et al.  The World Wide Web Conference , 2019, WWW.

[6]  Philip S. Yu,et al.  Temporal Network Embedding with Micro- and Macro-dynamics , 2019, CIKM.

[7]  Zhisong Pan,et al.  Lifelong representation learning in dynamic attributed networks , 2019, Neurocomputing.

[8]  Minyi Guo,et al.  GraphGAN: Graph Representation Learning with Generative Adversarial Nets , 2017, AAAI.

[9]  Omer Levy,et al.  Neural Word Embedding as Implicit Matrix Factorization , 2014, NIPS.

[10]  Zhifang Sui,et al.  Encoding Temporal Information for Time-Aware Link Prediction , 2016, EMNLP.

[11]  Philip S. Yu,et al.  DynGraphGAN: Dynamic Graph Embedding via Generative Adversarial Networks , 2019, DASFAA.

[12]  Qi Xuan,et al.  E-LSTM-D: A Deep Learning Framework for Dynamic Network Link Prediction , 2019, IEEE Transactions on Systems, Man, and Cybernetics: Systems.

[13]  Jian Pei,et al.  A Survey on Network Embedding , 2017, IEEE Transactions on Knowledge and Data Engineering.

[14]  Tsuyoshi Murata,et al.  A General View for Network Embedding as Matrix Factorization , 2019, WSDM.

[15]  Aynaz Taheri,et al.  Learning to Represent the Evolution of Dynamic Graphs with Recurrent Models , 2019, WWW.

[16]  Mohammad Al Hasan,et al.  DyLink2Vec: Effective Feature Representation for Link Prediction in Dynamic Networks , 2018, ArXiv.

[17]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[18]  Wenjie Li,et al.  Attention Network for Information Diffusion Prediction , 2018, WWW.

[19]  Shafiq R. Joty,et al.  Models for Capturing Temporal Smoothness in Evolving Networks for Learning Latent Representation of Nodes , 2018, ArXiv.

[20]  Mathias Niepert,et al.  Learning Convolutional Neural Networks for Graphs , 2016, ICML.

[21]  Boris Knyazev,et al.  Learning temporal attention in dynamic graphs with bilinear interactions , 2019, PloS one.

[22]  Bülent Yener,et al.  Unsupervised Multiway Data Analysis: A Literature Survey , 2009, IEEE Transactions on Knowledge and Data Engineering.

[23]  Yuan Zhang,et al.  COSINE: Community-Preserving Social Network Embedding From Information Diffusion Cascades , 2018, AAAI.

[24]  Le Song,et al.  Know-Evolve: Deep Temporal Reasoning for Dynamic Knowledge Graphs , 2017, ICML.

[25]  Rong Pan,et al.  Personalized Tag Recommendation through Nonlinear Tensor Factorization Using Gaussian Kernel , 2015, AAAI.

[26]  Bo Zong,et al.  Adaptive Neural Network for Node Classification in Dynamic Networks , 2019, 2019 IEEE International Conference on Data Mining (ICDM).

[27]  Le Song,et al.  Recurrent Coevolutionary Latent Feature Processes for Continuous-Time Recommendation , 2016, DLRS@RecSys.

[28]  Jia Wang,et al.  Topological Recurrent Neural Network for Diffusion Prediction , 2017, 2017 IEEE International Conference on Data Mining (ICDM).

[29]  Lada A. Adamic,et al.  Friends and neighbors on the Web , 2003, Soc. Networks.

[30]  Jacquelien M. A. Scherpen,et al.  Graph structure-preserving model reduction of linear network systems , 2016, 2016 European Control Conference (ECC).

[31]  Kevin Chen-Chuan Chang,et al.  A Comprehensive Survey of Graph Embedding: Problems, Techniques, and Applications , 2017, IEEE Transactions on Knowledge and Data Engineering.

[32]  Yue Gao,et al.  Dynamic Hypergraph Structure Learning , 2018, IJCAI.

[33]  Xiang Zhang,et al.  Spatio-Temporal Attentive RNN for Node Classification in Temporal Attributed Graphs , 2019, IJCAI.

[34]  Ido Guy,et al.  Node Embedding over Temporal Graphs , 2019, IJCAI.

[35]  Zhendong Mao,et al.  Knowledge Graph Embedding: A Survey of Approaches and Applications , 2017, IEEE Transactions on Knowledge and Data Engineering.

[36]  Sylvain Lamprier A Variational Topological Neural Model for Cascade-based Diffusion in Networks , 2018, ArXiv.

[37]  Huzefa Rangwala,et al.  Learning Dynamic Context Graphs for Predicting Social Events , 2019, KDD.

[38]  Katarzyna Musial,et al.  Foundations and Modeling of Dynamic Networks Using Dynamic Graph Neural Networks: A Survey , 2020, IEEE Access.

[39]  Vijay K. Devabhaktuni,et al.  Evaluating Link Prediction Accuracy in Dynamic Networks with Added and Removed Edges , 2016, 2016 IEEE International Conferences on Big Data and Cloud Computing (BDCloud), Social Computing and Networking (SocialCom), Sustainable Computing and Communications (SustainCom) (BDCloud-SocialCom-SustainCom).

[40]  Ryan A. Rossi,et al.  Continuous-Time Dynamic Network Embeddings , 2018, WWW.

[41]  W. Wang,et al.  NetWalk , 2018, Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining.

[42]  Maosong Sun,et al.  Multi-scale Information Diffusion Prediction with Reinforced Recurrent Networks , 2019, IJCAI.

[43]  Ah Chung Tsoi,et al.  The Graph Neural Network Model , 2009, IEEE Transactions on Neural Networks.

[44]  A. Stephen McGough,et al.  Temporal Graph Offset Reconstruction: Towards Temporally Robust Graph Representation Learning , 2018, 2018 IEEE International Conference on Big Data (Big Data).

[45]  Gong Zhang,et al.  GCN-GAN: A Non-linear Temporal Link Prediction Model for Weighted Dynamic Networks , 2019, IEEE INFOCOM 2019 - IEEE Conference on Computer Communications.

[46]  Richard S. Zemel,et al.  Gated Graph Sequence Neural Networks , 2015, ICLR.

[47]  Ryan A. Rossi Relational time series forecasting , 2018, The Knowledge Engineering Review.

[48]  Hai Lin,et al.  Dynamic Network Embeddings for Network Evolution Analysis , 2019, ArXiv.

[49]  Wenwu Zhu,et al.  Structural Deep Network Embedding , 2016, KDD.

[50]  Vineeth N. Balasubramanian,et al.  STwalk: learning trajectory representations in temporal graphs , 2017, COMAD/CODS.

[51]  A. Stephen McGough,et al.  Temporal Neighbourhood Aggregation: Predicting Future Links in Temporal Graphs via Recurrent Variational Graph Convolutions , 2019, 2019 IEEE International Conference on Big Data (Big Data).

[52]  Aijun An,et al.  dynnode2vec: Scalable Dynamic Network Embedding , 2018, 2018 IEEE International Conference on Big Data (Big Data).

[53]  Pascal Poupart,et al.  Representation Learning for Dynamic Graphs: A Survey , 2020, J. Mach. Learn. Res..

[54]  Yu Liu,et al.  T-GCN: A Temporal Graph Convolutional Network for Traffic Prediction , 2018, IEEE Transactions on Intelligent Transportation Systems.

[55]  Charu C. Aggarwal,et al.  Temporally Factorized Network Modeling for Evolutionary Network Analysis , 2017, WSDM.

[56]  Le Song,et al.  Large Scale Evolving Graphs with Burst Detection , 2019, IJCAI.

[57]  Laetitia Gauvin,et al.  weg2vec: Event embedding for temporal networks , 2019, Scientific Reports.

[58]  Tamara G. Kolda,et al.  Temporal Link Prediction Using Matrix and Tensor Factorizations , 2010, TKDD.

[59]  V. N. Bogaevski,et al.  Matrix Perturbation Theory , 1991 .

[60]  Éva Tardos,et al.  Maximizing the Spread of Influence through a Social Network , 2015, Theory Comput..

[61]  Jingrui He,et al.  Towards Explainable Representation of Time-Evolving Graphs via Spatial-Temporal Graph Attention Networks , 2019, CIKM.

[62]  Charu C. Aggarwal,et al.  NetWalk: A Flexible Deep Embedding Approach for Anomaly Detection in Dynamic Networks , 2018, KDD.

[63]  Sedigheh Mahdavi,et al.  Dynamic Joint Variational Graph Autoencoders , 2019, PKDD/ECML Workshops.

[64]  Charu C. Aggarwal,et al.  Link Prediction with Spatial and Temporal Consistency in Dynamic Networks , 2017, IJCAI.

[65]  Wei Cheng,et al.  Transformer-Style Relational Reasoning with Dynamic Memory Updating for Temporal Network Modeling , 2021, AAAI.

[66]  Ulrik Brandes,et al.  What is network science? , 2013, Network Science.

[67]  Jinyin Chen,et al.  GC-LSTM: graph convolution embedded LSTM for dynamic network link prediction , 2018, Applied Intelligence.

[68]  Le Song,et al.  Deep Coevolutionary Network: Embedding User and Item Features for Recommendation , 2016, 1609.03675.

[69]  Palash Goyal,et al.  DynamicGEM: A Library for Dynamic Graph Embedding Methods , 2018, ArXiv.

[70]  Geoffrey E. Hinton,et al.  Visualizing Data using t-SNE , 2008 .

[71]  Chengqi Zhang,et al.  Network Representation Learning: A Survey , 2017, IEEE Transactions on Big Data.

[72]  Jie Chen,et al.  EvolveGCN: Evolving Graph Convolutional Networks for Dynamic Graphs , 2020, AAAI.

[73]  Yoshua Bengio,et al.  Neural Machine Translation by Jointly Learning to Align and Translate , 2014, ICLR.

[74]  Jennifer Neville,et al.  Exploiting Interaction Links for Node Classification with Deep Graph Neural Networks , 2019, IJCAI.

[75]  Yuriy Tyshetskiy,et al.  Efficient Representation Learning Using Random Walks for Dynamic Graphs , 2019, ArXiv.

[76]  Rui Chen,et al.  Real-Time Streaming Graph Embedding Through Local Actions , 2019, WWW.

[77]  Guojie Song,et al.  Dynamic Network Embedding : An Extended Approach for Skip-gram based Network Embedding , 2018, IJCAI.

[78]  Yueting Zhuang,et al.  Dynamic Network Embedding by Modeling Triadic Closure Process , 2018, AAAI.

[79]  Mingzhe Wang,et al.  LINE: Large-scale Information Network Embedding , 2015, WWW.

[80]  Matthieu Latapy,et al.  Stream graphs and link streams for the modeling of interactions over time , 2017, Social Network Analysis and Mining.

[81]  Aram Galstyan,et al.  Scalable Link Prediction in Dynamic Networks via Non-Negative Matrix Factorization , 2014, ArXiv.

[82]  Arijit Khan,et al.  Efficiently Embedding Dynamic Knowledge Graphs , 2019, ArXiv.

[83]  Ying Yin,et al.  DHNE: Network Representation Learning Method for Dynamic Heterogeneous Networks , 2019, IEEE Access.

[84]  Hongyuan Zha,et al.  DyRep: Learning Representations over Dynamic Graphs , 2019, ICLR.

[85]  Xiaoning Qian,et al.  Variational Graph Recurrent Neural Networks , 2019, NeurIPS.

[86]  Jure Leskovec,et al.  Inductive Representation Learning on Large Graphs , 2017, NIPS.

[87]  Palash Goyal,et al.  dyngraph2vec: Capturing Network Dynamics using Dynamic Graph Representation Learning , 2018, Knowl. Based Syst..

[88]  Pascal Poupart,et al.  Diachronic Embedding for Temporal Knowledge Graph Completion , 2019, AAAI.

[89]  Yonggang Hu,et al.  Dynamic Graph Embedding via LSTM History Tracking , 2019, 2019 IEEE International Conference on Data Science and Advanced Analytics (DSAA).

[90]  Yao Zhao,et al.  Learning Heterogeneous Spatial-Temporal Representation for Bike-Sharing Demand Prediction , 2019, AAAI.

[91]  Manos Papagelis,et al.  Evolving network representation learning based on random walks , 2018, Applied Network Science.

[92]  Bart Baesens,et al.  Combining Temporal Aspects of Dynamic Networks with Node2Vec for a more Efficient Dynamic Link Prediction , 2018, 2018 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM).

[93]  Evgeniy Gabrilovich,et al.  A Review of Relational Machine Learning for Knowledge Graphs , 2015, Proceedings of the IEEE.

[94]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[95]  Jeffrey Dean,et al.  Distributed Representations of Words and Phrases and their Compositionality , 2013, NIPS.

[96]  Jason Weston,et al.  Translating Embeddings for Modeling Multi-relational Data , 2013, NIPS.

[97]  Hongyuan Zha,et al.  Modeling Event Propagation via Graph Biased Temporal Point Process , 2019, IEEE Transactions on Neural Networks and Learning Systems.

[98]  Junchi Yan,et al.  Modeling and Applications for Temporal Point Processes , 2019, KDD.

[99]  Yoshua Bengio,et al.  Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation , 2014, EMNLP.

[100]  Lixin Ji,et al.  MHDNE: Network Embedding Based on Multivariate Hawkes Process , 2019, PKDD/ECML Workshops.

[101]  Jure Leskovec,et al.  Representation Learning on Graphs: Methods and Applications , 2017, IEEE Data Eng. Bull..

[102]  Alexandros Nanopoulos,et al.  Modeling the dynamics of user preferences in coupled tensor factorization , 2014, RecSys '14.

[103]  Ryan A. Rossi,et al.  Modeling dynamic behavior in large evolving graphs , 2013, WSDM.

[104]  Jian Pei,et al.  TIMERS: Error-Bounded SVD Restart on Dynamic Networks , 2017, AAAI.

[105]  Jeffrey Dean,et al.  Efficient Estimation of Word Representations in Vector Space , 2013, ICLR.

[106]  Yulong Pei,et al.  Node classification in dynamic social networks , 2016 .

[107]  Stephan Mandt,et al.  Dynamic Word Embeddings , 2017, ICML.

[108]  Vachik S. Dave,et al.  Triangle Completion Time Prediction Using Time-Conserving Embedding , 2019, ECML/PKDD.

[109]  Max Welling,et al.  Auto-Encoding Variational Bayes , 2013, ICLR.

[110]  Junjie Wu,et al.  Embedding Temporal Network via Neighborhood Formation , 2018, KDD.

[111]  Jussara M. Almeida,et al.  Modeling Dynamic Ideological Behavior in Political Networks , 2019 .

[112]  Palash Goyal,et al.  Graph Embedding Techniques, Applications, and Performance: A Survey , 2017, Knowl. Based Syst..

[113]  Jochen De Weerdt,et al.  Dyn2Vec: Exploiting dynamic behaviour using difference networks-based node embeddings for classification , 2018 .

[114]  Zibin Zheng,et al.  T-EDGE: Temporal WEighted MultiDiGraph Embedding for Ethereum Transaction Network Analysis , 2019, Frontiers in Physics.

[115]  Lei Wang,et al.  Dynamic Network Embedding by Semantic Evolution , 2019, 2019 International Joint Conference on Neural Networks (IJCNN).

[116]  Cheng Li,et al.  DeepCas: An End-to-end Predictor of Information Cascades , 2016, WWW.

[117]  Max Welling,et al.  Semi-Supervised Classification with Graph Convolutional Networks , 2016, ICLR.

[118]  Liang Zhao,et al.  DynGraph2Seq: Dynamic-Graph-to-Sequence Interpretable Learning for Health Stage Prediction in Online Health Forums , 2019, 2019 IEEE International Conference on Data Mining (ICDM).

[119]  Eric Fleury,et al.  A unifying model for representing time-varying graphs , 2014, 2015 IEEE International Conference on Data Science and Advanced Analytics (DSAA).

[120]  Bruno Ribeiro,et al.  Subgraph Pattern Neural Networks for High-Order Graph Evolution Prediction , 2018, AAAI.

[121]  Dustin Arendt,et al.  Learning from Dynamic User Interaction Graphs to Forecast Diverse Social Behavior , 2019, CIKM.

[122]  Ryohei Hisano,et al.  Semi-supervised Graph Embedding Approach to Dynamic Link Prediction , 2016, ArXiv.

[123]  Wei Zhang,et al.  Dynamic Graph Representation Learning via Self-Attention Networks , 2018, ArXiv.

[124]  Mohammad Al Hasan,et al.  Link Prediction in Dynamic Networks Using Graphlet , 2016, ECML/PKDD.

[125]  Abhinav Gupta,et al.  Temporal Dynamic Graph LSTM for Action-Driven Video Object Detection , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[126]  Leo Katz,et al.  A new status index derived from sociometric analysis , 1953 .

[127]  Yan Liu,et al.  DynGEM: Deep Embedding Method for Dynamic Graphs , 2018, ArXiv.

[128]  Huan Liu,et al.  Attributed Network Embedding for Learning in a Dynamic Environment , 2017, CIKM.

[129]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[130]  Bin Yu,et al.  A Survey on Dynamic Network Embedding , 2020, ArXiv.

[131]  Jiliang Tang,et al.  Streaming Graph Neural Networks , 2018, SIGIR.

[132]  Zhanxing Zhu,et al.  Spatio-temporal Graph Convolutional Neural Network: A Deep Learning Framework for Traffic Forecasting , 2017, IJCAI.

[133]  András A. Benczúr,et al.  Node embeddings in dynamic graphs , 2019, Appl. Netw. Sci..

[134]  Max Welling,et al.  Variational Graph Auto-Encoders , 2016, ArXiv.