Foundations and Modeling of Dynamic Networks Using Dynamic Graph Neural Networks: A Survey

Dynamic networks are used in a wide range of fields, including social network analysis, recommender systems and epidemiology. Representing complex networks as structures changing over time allow network models to leverage not only structural but also temporal patterns. However, as dynamic network literature stems from diverse fields and makes use of inconsistent terminology, it is challenging to navigate. Meanwhile, graph neural networks (GNNs) have gained a lot of attention in recent years for their ability to perform well on a range of network science tasks, such as link prediction and node classification. Despite the popularity of graph neural networks and the proven benefits of dynamic network models, there has been little focus on graph neural networks for dynamic networks. To address the challenges resulting from the fact that this research crosses diverse fields as well as to survey dynamic graph neural networks, this work is split into two main parts. First, to address the ambiguity of the dynamic network terminology we establish a foundation of dynamic networks with consistent, detailed terminology and notation. Second, we present a comprehensive survey of dynamic graph neural network models using the proposed terminology.

[1]  Tomas E. Ward,et al.  Generative Adversarial Networks in Computer Vision , 2019, ACM Comput. Surv..

[2]  Da Xu,et al.  Generative Graph Convolutional Network for Growing Graphs , 2019, ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[3]  Ah Chung Tsoi,et al.  The Graph Neural Network Model , 2009, IEEE Transactions on Neural Networks.

[4]  Le Song,et al.  Know-Evolve: Deep Temporal Reasoning for Dynamic Knowledge Graphs , 2017, ICML.

[5]  S. Strogatz Exploring complex networks , 2001, Nature.

[6]  Da Xu,et al.  Self-attention with Functional Time Representation Learning , 2019, NeurIPS.

[7]  Tianqi Chen,et al.  Net2Net: Accelerating Learning via Knowledge Transfer , 2015, ICLR.

[8]  Kathleen M. Carley,et al.  Toward an interoperable dynamic network analysis toolkit , 2007, Decis. Support Syst..

[9]  Kevin Lee,et al.  A review of dynamic network models with latent variables. , 2017, Statistics surveys.

[10]  Mathias Niepert,et al.  Learning Convolutional Neural Networks for Graphs , 2016, ICML.

[11]  Gong Zhang,et al.  GCN-GAN: A Non-linear Temporal Link Prediction Model for Weighted Dynamic Networks , 2019, IEEE INFOCOM 2019 - IEEE Conference on Computer Communications.

[12]  Katarzyna Musial,et al.  Simulation and Augmentation of Social Networks for Building Deep Learning Models , 2019, ArXiv.

[13]  Yuhui Shi,et al.  Continuous-Time Link Prediction via Temporal Dependent Graph Neural Network , 2020, WWW.

[14]  Boris Knyazev,et al.  Learning temporal attention in dynamic graphs with bilinear interactions , 2019, PloS one.

[15]  Hui Li,et al.  A Deep Learning Approach to Link Prediction in Dynamic Networks , 2014, SDM.

[16]  Yixin Chen,et al.  Link Prediction Based on Graph Neural Networks , 2018, NeurIPS.

[17]  A. Stephen McGough,et al.  Temporal Neighbourhood Aggregation: Predicting Future Links in Temporal Graphs via Recurrent Variational Graph Convolutions , 2019, 2019 IEEE International Conference on Big Data (Big Data).

[18]  Petter Holme,et al.  Modern temporal network theory: a colloquium , 2015, The European Physical Journal B.

[19]  Jure Leskovec,et al.  TEDIC: Neural Modeling of Behavioral Patterns in Dynamic Social Interaction Networks , 2021, WWW.

[20]  Jari Saramäki,et al.  Temporal Networks , 2011, Encyclopedia of Social Network Analysis and Mining.

[21]  R. Pastor-Satorras,et al.  Activity driven modeling of time varying networks , 2012, Scientific Reports.

[22]  Qi Xuan,et al.  E-LSTM-D: A Deep Learning Framework for Dynamic Network Link Prediction , 2019, IEEE Transactions on Systems, Man, and Cybernetics: Systems.

[23]  Yonghong Yan,et al.  Restricted Boltzmann Machine-Based Approaches for Link Prediction in Dynamic Networks , 2018, IEEE Access.

[24]  Xiang Ren,et al.  Recurrent Event Network: Autoregressive Structure Inference over Temporal Knowledge Graphs , 2019, EMNLP.

[25]  Albert,et al.  Emergence of scaling in random networks , 1999, Science.

[26]  Xavier Bresson,et al.  Structured Sequence Modeling with Graph Convolutional Recurrent Networks , 2016, ICONIP.

[27]  Jürgen Schmidhuber,et al.  Learning Precise Timing with LSTM Recurrent Networks , 2003, J. Mach. Learn. Res..

[28]  Cornelius Fritz,et al.  Tempus volat, hora fugit: A survey of tie‐oriented dynamic network models in discrete and continuous time , 2019, Statistica Neerlandica.

[29]  Naoki Masuda,et al.  A Guide to Temporal Networks , 2016, Series on Complexity Science.

[30]  Pascal Poupart,et al.  Relational Representation Learning for Dynamic (Knowledge) Graphs: A Survey , 2019, ArXiv.

[31]  Dit-Yan Yeung,et al.  Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting , 2015, NIPS.

[32]  Eric P. Xing,et al.  Discrete Temporal Models of Social Networks , 2006, SNA@ICML.

[33]  Steven Skiena,et al.  DeepWalk: online learning of social representations , 2014, KDD.

[34]  Zhiyuan Liu,et al.  Graph Neural Networks: A Review of Methods and Applications , 2018, AI Open.

[35]  Yuxiao Dong,et al.  DeepInf: Social Influence Prediction with Deep Learning , 2018, KDD.

[36]  Heiga Zen,et al.  WaveNet: A Generative Model for Raw Audio , 2016, SSW.

[37]  Edoardo M. Airoldi,et al.  A Survey of Statistical Network Models , 2009, Found. Trends Mach. Learn..

[38]  Piotr Indyk,et al.  Maintaining Stream Statistics over Sliding Windows , 2002, SIAM J. Comput..

[39]  Alex Borges Vieira,et al.  A Survey on Embedding Dynamic Graphs , 2021, ACM Comput. Surv..

[40]  Jinyin Chen,et al.  GC-LSTM: graph convolution embedded LSTM for dynamic network link prediction , 2018, Applied Intelligence.

[41]  A. Moore,et al.  Dynamic social network analysis using latent space models , 2005, SKDD.

[42]  Jure Leskovec,et al.  Graph Convolutional Neural Networks for Web-Scale Recommender Systems , 2018, KDD.

[43]  Kevin S. Xu Stochastic Block Transition Models for Dynamic Networks , 2014, AISTATS.

[44]  O. Aalen,et al.  Survival and Event History Analysis: A Process Point of View , 2008 .

[45]  Liang Gou,et al.  DySAT: Deep Neural Representation Learning on Dynamic Graphs via Self-Attention Networks , 2020, WSDM.

[46]  Philip S. Yu,et al.  DynGraphGAN: Dynamic Graph Embedding via Generative Adversarial Networks , 2019, DASFAA.

[47]  Pablo Jensen,et al.  Revealing evolutions in dynamical networks , 2017, ArXiv.

[48]  Germain Forestier,et al.  Deep learning for time series classification: a review , 2018, Data Mining and Knowledge Discovery.

[49]  Per Block,et al.  Change we can believe in: Comparing longitudinal network models on consistency, interpretability and predictive power , 2018, Soc. Networks.

[50]  Haifeng Chen,et al.  Structural Temporal Graph Neural Networks for Anomaly Detection in Dynamic Graphs , 2020, CIKM.

[51]  Michalis Vazirgiannis,et al.  EvoNet: A Neural Network for Predicting the Evolution of Dynamic Graphs , 2020, ICANN.

[52]  Andrew McGregor,et al.  Graph stream algorithms: a survey , 2014, SGMD.

[53]  Stephan Günnemann,et al.  Graph Hawkes Neural Network for Forecasting on Temporal Knowledge Graphs , 2020, AKBC.

[54]  Jure Leskovec,et al.  Representation Learning on Graphs: Methods and Applications , 2017, IEEE Data Eng. Bull..

[55]  Da Xu,et al.  Inductive Representation Learning on Temporal Graphs , 2020, ICLR.

[56]  Palash Goyal,et al.  DynamicGEM: A Library for Dynamic Graph Embedding Methods , 2018, ArXiv.

[57]  Yihong Gong,et al.  Detecting communities and their evolutions in dynamic social networks—a Bayesian approach , 2011, Machine Learning.

[58]  Stephan Günnemann,et al.  Graph Hawkes Network for Reasoning on Temporal Knowledge Graphs , 2020, ArXiv.

[59]  Lukasz Kaiser,et al.  Attention is All you Need , 2017, NIPS.

[60]  Jure Leskovec,et al.  GraphRNN: A Deep Generative Model for Graphs , 2018, ICML 2018.

[61]  Tamara G. Kolda,et al.  Temporal Link Prediction Using Matrix and Tensor Factorizations , 2010, TKDD.

[62]  P. Grambsch Survival and Event History Analysis: A Process Point of View by AALEN, O. O., BORGAN, O., and GJESSING, H. K. , 2009 .

[63]  Palash Goyal,et al.  Graph Embedding Techniques, Applications, and Performance: A Survey , 2017, Knowl. Based Syst..

[64]  Jian Zhang,et al.  A Survey on Streaming Algorithms for Massive Graphs , 2010, Managing and Mining Graph Data.

[65]  Aynaz Taheri,et al.  Learning to Represent the Evolution of Dynamic Graphs with Recurrent Models , 2019, WWW.

[66]  Sanjay Thakur,et al.  Time2Vec: Learning a Vector Representation of Time , 2019, ArXiv.

[67]  Carter T. Butts,et al.  A Relational Event Approach to Modeling Behavioral Dynamics , 2017, 1707.09902.

[68]  Jackie Chi Kit Cheung,et al.  TeMP: Temporal Message Passing for Temporal Knowledge Graph Completion , 2020, EMNLP.

[69]  Fernando Berzal Galiano,et al.  A Survey of Link Prediction in Complex Networks , 2016, ACM Comput. Surv..

[70]  Wenwu Zhu,et al.  Structural Deep Network Embedding , 2016, KDD.

[71]  Linyuan Lu,et al.  Link Prediction in Complex Networks: A Survey , 2010, ArXiv.

[72]  Fei Wang,et al.  Patient Subtyping via Time-Aware LSTM Networks , 2017, KDD.

[73]  Mingyuan Zhou,et al.  Semi-Implicit Variational Inference , 2018, ICML.

[74]  Philip S. Yu,et al.  Deep Dynamic Network Embedding for Link Prediction , 2018, IEEE Access.

[75]  Feng Liu,et al.  Deep Learning Approaches for Link Prediction in Social Network Services , 2013, ICONIP.

[76]  Max Welling,et al.  Semi-Supervised Classification with Graph Convolutional Networks , 2016, ICLR.

[77]  Davide Eynard,et al.  Temporal Graph Networks for Deep Learning on Dynamic Graphs , 2020, ArXiv.

[78]  Hong Cheng,et al.  Predicting Path Failure In Time-Evolving Graphs , 2019, KDD.

[79]  Apurva Narayan,et al.  Learning Graph Dynamics using Deep Neural Networks , 2018 .

[80]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[81]  Nicola Santoro,et al.  Time-varying graphs and dynamic networks , 2010, Int. J. Parallel Emergent Distributed Syst..

[82]  Xiang Ren,et al.  Characterizing and Forecasting User Engagement with In-App Action Graph: A Case Study of Snapchat , 2019, KDD.

[83]  AcarEvrim,et al.  Temporal Link Prediction Using Matrix and Tensor Factorizations , 2011 .

[84]  Yanbang Wang,et al.  Generic Representation Learning for Dynamic Social Interaction , 2020 .

[85]  Vijay S. Pande,et al.  Molecular graph convolutions: moving beyond fingerprints , 2016, Journal of Computer-Aided Molecular Design.

[86]  Wenwu Zhu,et al.  Deep Learning on Graphs: A Survey , 2018, IEEE Transactions on Knowledge and Data Engineering.

[87]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[88]  Faïez Gargouri,et al.  Modeling dynamics of social networks: A survey , 2014, 2014 6th International Conference on Computational Aspects of Social Networks.

[89]  Bin Yu,et al.  A Survey on Dynamic Network Embedding , 2020, ArXiv.

[90]  Jiliang Tang,et al.  Streaming Graph Neural Networks , 2018, SIGIR.

[91]  Zhanxing Zhu,et al.  Spatio-temporal Graph Convolutional Neural Network: A Deep Learning Framework for Traffic Forecasting , 2017, IJCAI.

[92]  Max Welling,et al.  Variational Graph Auto-Encoders , 2016, ArXiv.

[93]  Kunpeng Zhang,et al.  A Heterogeneous Dynamical Graph Neural Networks Approach to Quantify Scientific Impact , 2020, ArXiv.

[94]  Silvio Savarese,et al.  Structural-RNN: Deep Learning on Spatio-Temporal Graphs , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[95]  Alfred O. Hero,et al.  Dynamic Stochastic Blockmodels for Time-Evolving Social Networks , 2014, IEEE Journal of Selected Topics in Signal Processing.

[96]  Jie Chen,et al.  EvolveGCN: Evolving Graph Convolutional Networks for Dynamic Graphs , 2020, AAAI.

[97]  Bernard Ghanem,et al.  DeepGCNs: Can GCNs Go As Deep As CNNs? , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[98]  Christos Faloutsos,et al.  Graphs over time: densification laws, shrinking diameters and possible explanations , 2005, KDD '05.

[99]  Pascal Poupart,et al.  Representation Learning for Dynamic Graphs: A Survey , 2020, J. Mach. Learn. Res..

[100]  Xavier Bresson,et al.  Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering , 2016, NIPS.

[101]  Tomas E. Ward,et al.  Generative Adversarial Networks: A Survey and Taxonomy , 2019, ArXiv.

[102]  Nitesh V. Chawla,et al.  Evaluating link prediction methods , 2014, Knowledge and Information Systems.

[103]  X ZhengAlice,et al.  A Survey of Statistical Network Models , 2010 .

[104]  Hongyuan Zha,et al.  DyRep: Learning Representations over Dynamic Graphs , 2019, ICLR.

[105]  Xiaoning Qian,et al.  Variational Graph Recurrent Neural Networks , 2019, NeurIPS.

[106]  Charu C. Aggarwal,et al.  Evolutionary Network Analysis , 2014, ACM Comput. Surv..

[107]  Jure Leskovec,et al.  Inductive Representation Learning on Large Graphs , 2017, NIPS.

[108]  Pietro Liò,et al.  Graph Attention Networks , 2017, ICLR.

[109]  Palash Goyal,et al.  dyngraph2vec: Capturing Network Dynamics using Dynamic Graph Representation Learning , 2018, Knowl. Based Syst..

[110]  Alessandro Rozza,et al.  Dynamic Graph Convolutional Networks , 2017, Pattern Recognit..

[111]  A. Barabasi,et al.  Evolution of the social network of scientific collaborations , 2001, cond-mat/0104162.

[112]  Paul G. Spirakis,et al.  Elements of the theory of dynamic networks , 2018, Commun. ACM.

[113]  Anuraj Mohan,et al.  Temporal Link Prediction: A Survey , 2019, New Generation Computing.

[114]  A. Tordai,et al.  Modeling Relational Data with Graph Convolutional Networks , 2017 .

[115]  Tom A. B. Snijders,et al.  Introduction to stochastic actor-based models for network dynamics , 2010, Soc. Networks.

[116]  Philip S. Yu,et al.  A Comprehensive Survey on Graph Neural Networks , 2019, IEEE Transactions on Neural Networks and Learning Systems.

[117]  Alán Aspuru-Guzik,et al.  Convolutional Networks on Graphs for Learning Molecular Fingerprints , 2015, NIPS.

[118]  Xavier Bresson,et al.  Geometric Matrix Completion with Recurrent Multi-Graph Neural Networks , 2017, NIPS.

[119]  Max Welling,et al.  Modeling Relational Data with Graph Convolutional Networks , 2017, ESWC.

[120]  R. Zemel,et al.  Neural Relational Inference for Interacting Systems , 2018, ICML.

[121]  Z. Wang,et al.  The structure and dynamics of multilayer networks , 2014, Physics Reports.

[122]  Jure Leskovec,et al.  Predicting Dynamic Embedding Trajectory in Temporal Interaction Networks , 2019, KDD.

[123]  Pedro A. Szekely,et al.  Recurrent Event Network : Global Structure Inference Over Temporal Knowledge Graph , 2019 .

[124]  Jason Eisner,et al.  The Neural Hawkes Process: A Neurally Self-Modulating Multivariate Point Process , 2016, NIPS.

[125]  Vijay K. Devabhaktuni,et al.  Evaluating Link Prediction Accuracy in Dynamic Networks with Added and Removed Edges , 2016, 2016 IEEE International Conferences on Big Data and Cloud Computing (BDCloud), Social Computing and Networking (SocialCom), Sustainable Computing and Communications (SustainCom) (BDCloud-SocialCom-SustainCom).

[126]  Richard S. Zemel,et al.  Gated Graph Sequence Neural Networks , 2015, ICLR.