Foundations and modelling of dynamic networks using Dynamic Graph Neural Networks: A survey

Dynamic networks are used in a wide range of fields, including social network analysis, recommender systems and epidemiology. Representing complex networks as structures changing over time allow network models to leverage not only structural but also temporal patterns. However, as dynamic network literature stems from diverse fields and makes use of inconsistent terminology, it is challenging to navigate. Meanwhile, graph neural networks (GNNs) have gained a lot of attention in recent years for their ability to perform well on a range of network science tasks, such as link prediction and node classification. Despite the popularity of graph neural networks and the proven benefits of dynamic network models, there has been little focus on graph neural networks for dynamic networks. We aim to provide a review that demystifies dynamic networks, introduces dynamic graph neural networks (DGNNs) and appeals to researchers with a background in either network science or data science. We contribute: (i) a comprehensive dynamic network taxonomy, (ii) a survey of dynamic graph neural networks and (iii) an overview of how dynamic graph neural networks can be used for dynamic link prediction.

[1]  Hongyuan Zha,et al.  DyRep: Learning Representations over Dynamic Graphs , 2019, ICLR.

[2]  Jure Leskovec,et al.  GraphRNN: A Deep Generative Model for Graphs , 2018, ICML 2018.

[3]  Philip S. Yu,et al.  DynGraphGAN: Dynamic Graph Embedding via Generative Adversarial Networks , 2019, DASFAA.

[4]  Richard S. Zemel,et al.  Gated Graph Sequence Neural Networks , 2015, ICLR.

[5]  Fei Wang,et al.  Patient Subtyping via Time-Aware LSTM Networks , 2017, KDD.

[6]  Jure Leskovec,et al.  Predicting Dynamic Embedding Trajectory in Temporal Interaction Networks , 2019, KDD.

[7]  Max Welling,et al.  Semi-Supervised Classification with Graph Convolutional Networks , 2016, ICLR.

[8]  Tianqi Chen,et al.  Net2Net: Accelerating Learning via Knowledge Transfer , 2015, ICLR.

[9]  Xiang Ren,et al.  Characterizing and Forecasting User Engagement with In-App Action Graph: A Case Study of Snapchat , 2019, KDD.

[10]  Hui Li,et al.  A Deep Learning Approach to Link Prediction in Dynamic Networks , 2014, SDM.

[11]  Gong Zhang,et al.  GCN-GAN: A Non-linear Temporal Link Prediction Model for Weighted Dynamic Networks , 2019, IEEE INFOCOM 2019 - IEEE Conference on Computer Communications.

[12]  Kevin Lee,et al.  A review of dynamic network models with latent variables. , 2017, Statistics surveys.

[13]  Pietro Liò,et al.  Graph Attention Networks , 2017, ICLR.

[14]  Philip S. Yu,et al.  Deep Dynamic Network Embedding for Link Prediction , 2018, IEEE Access.

[15]  Pablo Jensen,et al.  Revealing evolutions in dynamical networks , 2017, ArXiv.

[16]  Jie Chen,et al.  EvolveGCN: Evolving Graph Convolutional Networks for Dynamic Graphs , 2020, AAAI.

[17]  Jure Leskovec,et al.  Inductive Representation Learning on Large Graphs , 2017, NIPS.

[18]  Bernard Ghanem,et al.  DeepGCNs: Can GCNs Go As Deep As CNNs? , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[19]  Jürgen Schmidhuber,et al.  Learning Precise Timing with LSTM Recurrent Networks , 2003, J. Mach. Learn. Res..

[20]  O. Aalen,et al.  Survival and Event History Analysis: A Process Point of View , 2008 .

[21]  Tamara G. Kolda,et al.  Temporal Link Prediction Using Matrix and Tensor Factorizations , 2010, TKDD.

[22]  Alfred O. Hero,et al.  Dynamic Stochastic Blockmodels for Time-Evolving Social Networks , 2014, IEEE Journal of Selected Topics in Signal Processing.

[23]  Liang Gou,et al.  DySAT: Deep Neural Representation Learning on Dynamic Graphs via Self-Attention Networks , 2020, WSDM.

[24]  Xavier Bresson,et al.  Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering , 2016, NIPS.

[25]  Mingyuan Zhou,et al.  Semi-Implicit Variational Inference , 2018, ICML.

[26]  Albert,et al.  Emergence of scaling in random networks , 1999, Science.

[27]  Da Xu,et al.  Generative Graph Convolutional Network for Growing Graphs , 2019, ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[28]  Pascal Poupart,et al.  Relational Representation Learning for Dynamic (Knowledge) Graphs: A Survey , 2019, ArXiv.

[29]  Edoardo M. Airoldi,et al.  A Survey of Statistical Network Models , 2009, Found. Trends Mach. Learn..

[30]  Michalis Vazirgiannis,et al.  EvoNet: A Neural Network for Predicting the Evolution of Dynamic Graphs , 2020, ICANN.

[31]  Zhanxing Zhu,et al.  Spatio-temporal Graph Convolutional Neural Network: A Deep Learning Framework for Traffic Forecasting , 2017, IJCAI.

[32]  Dit-Yan Yeung,et al.  Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting , 2015, NIPS.

[33]  Jason Eisner,et al.  The Neural Hawkes Process: A Neurally Self-Modulating Multivariate Point Process , 2016, NIPS.

[34]  Palash Goyal,et al.  dyngraph2vec: Capturing Network Dynamics using Dynamic Graph Representation Learning , 2018, Knowl. Based Syst..

[35]  Jinyin Chen,et al.  GC-LSTM: Graph Convolution Embedded LSTM for Dynamic Link Prediction , 2018, ArXiv.

[36]  Eric P. Xing,et al.  Discrete Temporal Models of Social Networks , 2006, SNA@ICML.

[37]  Andrew McGregor,et al.  Graph stream algorithms: a survey , 2014, SGMD.

[38]  Petter Holme,et al.  Modern temporal network theory: a colloquium , 2015, The European Physical Journal B.

[39]  Zhiyuan Liu,et al.  Graph Neural Networks: A Review of Methods and Applications , 2018, AI Open.

[40]  R. Zemel,et al.  Neural Relational Inference for Interacting Systems , 2018, ICML.

[41]  Jure Leskovec,et al.  Representation Learning on Graphs: Methods and Applications , 2017, IEEE Data Eng. Bull..

[42]  Kevin S. Xu Stochastic Block Transition Models for Dynamic Networks , 2014, AISTATS.

[43]  Tom A. B. Snijders,et al.  Introduction to stochastic actor-based models for network dynamics , 2010, Soc. Networks.

[44]  Palash Goyal,et al.  DynamicGEM: A Library for Dynamic Graph Embedding Methods , 2018, ArXiv.

[45]  Apurva Narayan,et al.  Learning Graph Dynamics using Deep Neural Networks , 2018 .

[46]  Giulio Rossetti,et al.  Community Discovery in Dynamic Networks , 2017, ACM Comput. Surv..

[47]  Fernando Berzal Galiano,et al.  A Survey of Link Prediction in Complex Networks , 2016, ACM Comput. Surv..

[48]  Alessandro Rozza,et al.  Dynamic Graph Convolutional Networks , 2017, Pattern Recognit..

[49]  Yuxiao Dong,et al.  DeepInf: Social Influence Prediction with Deep Learning , 2018, KDD.

[50]  Stephan Günnemann,et al.  Graph Hawkes Network for Reasoning on Temporal Knowledge Graphs , 2020, ArXiv.

[51]  Nicola Santoro,et al.  Time-varying graphs and dynamic networks , 2010, Int. J. Parallel Emergent Distributed Syst..

[52]  Qi Xuan,et al.  E-LSTM-D: A Deep Learning Framework for Dynamic Network Link Prediction , 2019, IEEE Transactions on Systems, Man, and Cybernetics: Systems.

[53]  Boris Knyazev,et al.  Learning temporal attention in dynamic graphs with bilinear interactions , 2019, PloS one.

[54]  Wenwu Zhu,et al.  Structural Deep Network Embedding , 2016, KDD.

[55]  Jure Leskovec,et al.  Graph Convolutional Neural Networks for Web-Scale Recommender Systems , 2018, KDD.

[56]  Palash Goyal,et al.  Graph Embedding Techniques, Applications, and Performance: A Survey , 2017, Knowl. Based Syst..

[57]  Jayanta K. Ghosh,et al.  Survival and Event History Analysis: A Process Point of View by Odd O. Aalen, Ørnulf Borgan, Håkon K. Gjessing , 2009 .

[58]  Max Welling,et al.  Modeling Relational Data with Graph Convolutional Networks , 2017, ESWC.

[59]  Yonghong Yan,et al.  Restricted Boltzmann Machine-Based Approaches for Link Prediction in Dynamic Networks , 2018, IEEE Access.

[60]  Naoki Masuda,et al.  A Guide to Temporal Networks , 2016, Series on Complexity Science.

[61]  A. Barabasi,et al.  Evolution of the social network of scientific collaborations , 2001, cond-mat/0104162.

[62]  Le Song,et al.  Know-Evolve: Deep Temporal Reasoning for Dynamic Knowledge Graphs , 2017, ICML.

[63]  Nitesh V. Chawla,et al.  Evaluating link prediction methods , 2014, Knowledge and Information Systems.

[64]  Kathleen M. Carley,et al.  Toward an interoperable dynamic network analysis toolkit , 2007, Decis. Support Syst..

[65]  Philip S. Yu,et al.  A Comprehensive Survey on Graph Neural Networks , 2019, IEEE Transactions on Neural Networks and Learning Systems.

[66]  Silvio Savarese,et al.  Structural-RNN: Deep Learning on Spatio-Temporal Graphs , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[67]  Xavier Bresson,et al.  Structured Sequence Modeling with Graph Convolutional Recurrent Networks , 2016, ICONIP.

[68]  Katarzyna Musial,et al.  Simulation and Augmentation of Social Networks for Building Deep Learning Models , 2019, ArXiv.

[69]  Carter T. Butts,et al.  A Relational Event Approach to Modeling Behavioral Dynamics , 2017, 1707.09902.

[70]  Charu C. Aggarwal,et al.  Evolutionary Network Analysis , 2014, ACM Comput. Surv..

[71]  Paul G. Spirakis,et al.  Elements of the theory of dynamic networks , 2018, Commun. ACM.

[72]  Xiaoning Qian,et al.  Variational Graph Recurrent Neural Networks , 2019, NeurIPS.

[73]  Faïez Gargouri,et al.  Modeling dynamics of social networks: A survey , 2014, 2014 6th International Conference on Computational Aspects of Social Networks.

[74]  Jian Zhang,et al.  A Survey on Streaming Algorithms for Massive Graphs , 2010, Managing and Mining Graph Data.

[75]  Z. Wang,et al.  The structure and dynamics of multilayer networks , 2014, Physics Reports.

[76]  Linyuan Lu,et al.  Link Prediction in Complex Networks: A Survey , 2010, ArXiv.

[77]  Mathias Niepert,et al.  Learning Convolutional Neural Networks for Graphs , 2016, ICML.

[78]  Lukasz Kaiser,et al.  Attention is All you Need , 2017, NIPS.

[79]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[80]  Jari Saramäki,et al.  Temporal Networks , 2011, Encyclopedia of Social Network Analysis and Mining.

[81]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[82]  S. Strogatz Exploring complex networks , 2001, Nature.

[83]  Vijay S. Pande,et al.  Molecular graph convolutions: moving beyond fingerprints , 2016, Journal of Computer-Aided Molecular Design.

[84]  Wenwu Zhu,et al.  Deep Learning on Graphs: A Survey , 2018, IEEE Transactions on Knowledge and Data Engineering.

[85]  Aynaz Taheri,et al.  Learning to Represent the Evolution of Dynamic Graphs with Recurrent Models , 2019, WWW.

[86]  Per Block,et al.  Change we can believe in: Comparing longitudinal network models on consistency, interpretability and predictive power , 2018, Soc. Networks.

[87]  Pedro A. Szekely,et al.  Recurrent Event Network : Global Structure Inference Over Temporal Knowledge Graph , 2019 .

[88]  Vijay K. Devabhaktuni,et al.  Evaluating Link Prediction Accuracy in Dynamic Networks with Added and Removed Edges , 2016, 2016 IEEE International Conferences on Big Data and Cloud Computing (BDCloud), Social Computing and Networking (SocialCom), Sustainable Computing and Communications (SustainCom) (BDCloud-SocialCom-SustainCom).

[89]  Germain Forestier,et al.  Deep learning for time series classification: a review , 2018, Data Mining and Knowledge Discovery.

[90]  Anuraj Mohan,et al.  Temporal Link Prediction: A Survey , 2019, New Generation Computing.

[91]  Hong Cheng,et al.  Predicting Path Failure In Time-Evolving Graphs , 2019, KDD.

[92]  Piotr Indyk,et al.  Maintaining Stream Statistics over Sliding Windows , 2002, SIAM J. Comput..

[93]  Max Welling,et al.  Variational Graph Auto-Encoders , 2016, ArXiv.

[94]  Tomas E. Ward,et al.  Generative Adversarial Networks: A Survey and Taxonomy , 2019, ArXiv.

[95]  Jiliang Tang,et al.  Streaming Graph Neural Networks , 2018, SIGIR.

[96]  Alán Aspuru-Guzik,et al.  Convolutional Networks on Graphs for Learning Molecular Fingerprints , 2015, NIPS.

[97]  Xavier Bresson,et al.  Geometric Matrix Completion with Recurrent Multi-Graph Neural Networks , 2017, NIPS.

[98]  Christos Faloutsos,et al.  Graphs over time: densification laws, shrinking diameters and possible explanations , 2005, KDD '05.

[99]  Cornelius Fritz,et al.  Tempus volat, hora fugit: A survey of tie‐oriented dynamic network models in discrete and continuous time , 2019, Statistica Neerlandica.

[100]  Pascal Poupart,et al.  Representation Learning for Dynamic Graphs: A Survey , 2020, J. Mach. Learn. Res..