Variational Graph Recurrent Neural Networks

Representation learning over graph structured data has been mostly studied in static graph settings while efforts for modeling dynamic graphs are still scant. In this paper, we develop a novel hierarchical variational model that introduces additional latent random variables to jointly model the hidden states of a graph recurrent neural network (GRNN) to capture both topology and node attribute changes in dynamic graphs. We argue that the use of high-level latent random variables in this variational GRNN (VGRNN) can better capture potential variability observed in dynamic graphs as well as the uncertainty of node latent representation. With semi-implicit variational inference developed for this new VGRNN architecture (SI-VGRNN), we show that flexible non-Gaussian latent representations can further help dynamic graph analytic tasks. Our experiments with multiple real-world dynamic graph datasets demonstrate that SI-VGRNN and VGRNN consistently outperform the existing baseline and state-of-the-art methods by a significant margin in dynamic link prediction.

[1]  Lise Getoor,et al.  Collective Classification in Network Data , 2008, AI Mag..

[2]  Xiaoning Qian,et al.  Semi-Implicit Graph Variational Auto-Encoders , 2019, NeurIPS.

[3]  Yoshua Bengio,et al.  Z-Forcing: Training Stochastic Recurrent Networks , 2017, NIPS.

[4]  Alfred O. Hero,et al.  Dynamic Stochastic Blockmodels for Time-Evolving Social Networks , 2014, IEEE Journal of Selected Topics in Signal Processing.

[5]  Hamidreza Amindavar,et al.  Maneuvering target tracking based on combined stochastic differential equations and garch process , 2012, 2012 11th International Conference on Information Science, Signal Processing and their Applications (ISSPA).

[6]  Jure Leskovec,et al.  node2vec: Scalable Feature Learning for Networks , 2016, KDD.

[7]  Luca Antiga,et al.  Automatic differentiation in PyTorch , 2017 .

[8]  Max Welling,et al.  Semi-Supervised Classification with Graph Convolutional Networks , 2016, ICLR.

[9]  Jure Leskovec,et al.  {SNAP Datasets}: {Stanford} Large Network Dataset Collection , 2014 .

[10]  Jian Pei,et al.  TIMERS: Error-Bounded SVD Restart on Dynamic Networks , 2017, AAAI.

[11]  Mohammad Al Hasan,et al.  Link Prediction in Dynamic Networks Using Graphlet , 2016, ECML/PKDD.

[12]  Palash Goyal,et al.  dyngraph2vec: Capturing Network Dynamics using Dynamic Graph Representation Learning , 2018, Knowl. Based Syst..

[13]  Xiaoning Qian,et al.  Differential Expression Analysis of Dynamical Sequencing Count Data with a Gamma Markov Chain , 2018, 1803.02527.

[14]  Jianhua Z. Huang,et al.  Robust Negative Sampling for Network Embedding , 2019, AAAI.

[15]  Jure Leskovec,et al.  Inductive Representation Learning on Large Graphs , 2017, NIPS.

[16]  Yueting Zhuang,et al.  Dynamic Network Embedding by Modeling Triadic Closure Process , 2018, AAAI.

[17]  Yoshua Bengio,et al.  A Recurrent Latent Variable Model for Sequential Data , 2015, NIPS.

[18]  Yan Liu,et al.  DynGEM: Deep Embedding Method for Dynamic Graphs , 2018, ArXiv.

[19]  Jure Leskovec,et al.  Learning Structural Node Embeddings via Diffusion Wavelets , 2017, KDD.

[20]  Yoshua Bengio,et al.  Variational Bi-LSTMs , 2017, ArXiv.

[21]  Mingzhe Wang,et al.  LINE: Large-scale Information Network Embedding , 2015, WWW.

[22]  Wei Zhang,et al.  Dynamic Graph Representation Learning via Self-Attention Networks , 2018, ArXiv.

[23]  Xavier Bresson,et al.  Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering , 2016, NIPS.

[24]  Hongyuan Zha,et al.  DyRep: Learning Representations over Dynamic Graphs , 2019, ICLR.

[25]  David J. Marchette,et al.  Scan Statistics on Enron Graphs , 2005, Comput. Math. Organ. Theory.

[26]  Xiaoning Qian,et al.  Bayesian multi-domain learning for cancer subtype discovery from next-generation sequencing count data , 2018, NeurIPS.

[27]  Stephan Günnemann,et al.  Deep Gaussian Embedding of Graphs: Unsupervised Inductive Learning via Ranking , 2017, ICLR.

[28]  Huan Liu,et al.  Attributed Network Embedding for Learning in a Dynamic Environment , 2017, CIKM.

[29]  Arman Hasanzadeh,et al.  Piecewise Stationary Modeling of Random Processes Over Graphs With an Application to Traffic Prediction , 2017, 2019 IEEE International Conference on Big Data (Big Data).

[30]  Ole Winther,et al.  Sequential Neural Models with Stochastic Layers , 2016, NIPS.

[31]  Krishna P. Gummadi,et al.  On the evolution of user interaction in Facebook , 2009, WOSN '09.

[32]  Seyyed Hamed Fouladi,et al.  Stochastic Differential Equations for Modeling of High Maneuvering Target Tracking , 2013 .

[33]  Max Welling,et al.  Variational Graph Auto-Encoders , 2016, ArXiv.

[34]  Le Song,et al.  Know-Evolve: Deep Temporal Reasoning for Dynamic Knowledge Graphs , 2017, ICML.

[35]  Edward R. Dougherty,et al.  Scalable optimal Bayesian classification of single-cell trajectories under regulatory model uncertainty , 2018, BMC Genomics.

[36]  Dmitry P. Vetrov,et al.  Doubly Semi-Implicit Variational Inference , 2018, AISTATS.

[37]  Xavier Bresson,et al.  Structured Sequence Modeling with Graph Convolutional Recurrent Networks , 2016, ICONIP.

[38]  Jon M. Kleinberg,et al.  Overview of the 2003 KDD Cup , 2003, SKDD.

[39]  Arman Hasanzadeh,et al.  A Graph Signal Processing Approach For Real-Time Traffic Prediction In Transportation Networks , 2017, 1711.06954.

[40]  Steven Skiena,et al.  DeepWalk: online learning of social representations , 2014, KDD.

[41]  Hamidreza Amindavar,et al.  Maneuvering target tracking based on SDE driven by garch volatility , 2012, 2012 IEEE Statistical Signal Processing Workshop (SSP).

[42]  Daniel R. Figueiredo,et al.  struc2vec: Learning Node Representations from Structural Identity , 2017, KDD.

[43]  Mingyuan Zhou,et al.  Semi-Implicit Variational Inference , 2018, ICML.