Deep Learning on Graphs: A Survey

Deep learning has been shown to be successful in a number of domains, ranging from acoustics, images, to natural language processing. However, applying deep learning to the ubiquitous graph data is non-trivial because of the unique characteristics of graphs. Recently, substantial research efforts have been devoted to applying deep learning methods to graphs, resulting in beneficial advances in graph analysis techniques. In this survey, we comprehensively review the different types of deep learning methods on graphs. We divide the existing methods into five categories based on their model architectures and training strategies: graph recurrent neural networks, graph convolutional networks, graph autoencoders, graph reinforcement learning, and graph adversarial methods. We then provide a comprehensive overview of these methods in a systematic manner mainly by following their development history. We also analyze the differences and compositions of different methods. Finally, we briefly outline the applications in which they have been used and discuss potential future research directions.

[1]  Dan Wang,et al.  Adversarial Network Embedding , 2017, AAAI.

[2]  Yixin Chen,et al.  An End-to-End Deep Learning Architecture for Graph Classification , 2018, AAAI.

[3]  Stanford,et al.  Learning to Discover Social Circles in Ego Networks , 2012 .

[4]  Inderjit S. Dhillon,et al.  Weighted Graph Cuts without Eigenvectors A Multilevel Approach , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[5]  Wenwu Zhu,et al.  Deep Variational Network Embedding in Wasserstein Space , 2018, KDD.

[6]  W. A. Kirk,et al.  An Introduction to Metric Spaces and Fixed Point Theory , 2001 .

[7]  Yanfang Ye,et al.  Heterogeneous Graph Attention Network , 2019, WWW.

[8]  Demis Hassabis,et al.  Mastering the game of Go without human knowledge , 2017, Nature.

[9]  László Babai,et al.  Graph isomorphism in quasipolynomial time [extended abstract] , 2015, STOC.

[10]  Jian Pei,et al.  Community Preserving Network Embedding , 2017, AAAI.

[11]  Ziwei Zhang.  A Note on Spectral Clustering and SVD of Graph Data , 2018, ArXiv.

[12]  Razvan Pascanu,et al.  A simple neural network module for relational reasoning , 2017, NIPS.

[13]  F. Scarselli,et al.  A new model for learning in graph domains , 2005, Proceedings. 2005 IEEE International Joint Conference on Neural Networks, 2005..

[14]  Zhaochun Ren,et al.  Dynamic Graph Neural Networks , 2018, ArXiv.

[15]  Le Song,et al.  Stochastic Training of Graph Convolutional Networks with Variance Reduction , 2017, ICML.

[16]  Enhong Chen,et al.  Learning Deep Representations for Graph Clustering , 2014, AAAI.

[17]  Stephan Günnemann,et al.  Predict then Propagate: Graph Neural Networks meet Personalized PageRank , 2018, ICLR.

[18]  Tara N. Sainath,et al.  FUNDAMENTAL TECHNOLOGIES IN MODERN SPEECH RECOGNITION Digital Object Identifier 10.1109/MSP.2012.2205597 , 2012 .

[19]  Xiao Wang,et al.  Hierarchical Taxonomy Aware Network Embedding , 2018, KDD.

[20]  Pierre Vandergheynst,et al.  A Multiscale Pyramid Transform for Graph Signals , 2013, IEEE Transactions on Signal Processing.

[21]  Ah Chung Tsoi,et al.  The Graph Neural Network Model , 2009, IEEE Transactions on Neural Networks.

[22]  Albert,et al.  Emergence of scaling in random networks , 1999, Science.

[23]  Guillermo Sapiro,et al.  Deep Neural Networks with Random Gaussian Weights: A Universal Classification Strategy? , 2015, IEEE Transactions on Signal Processing.

[24]  S. S. Vallender.  Calculation of the Wasserstein Distance Between Probability Distributions on the Line , 1974 .

[25]  Rajeev Motwani,et al.  The PageRank Citation Ranking : Bringing Order to the Web , 1999, WWW 1999.

[26]  Jiliang Tang,et al.  Signed Graph Convolutional Networks , 2018, 2018 IEEE International Conference on Data Mining (ICDM).

[27]  Jeffrey C Grossman,et al.  Crystal Graph Convolutional Neural Networks for an Accurate and Interpretable Prediction of Material Properties. , 2017, Physical review letters.

[28]  Jian Pei,et al.  Arbitrary-Order Proximity Preserved Network Embedding , 2018, KDD.

[29]  Charu C. Aggarwal,et al.  Heterogeneous Network Embedding via Deep Architectures , 2015, KDD.

[30]  Takanori Maehara,et al.  Revisiting Graph Neural Networks: All We Have is Low-Pass Filters , 2019, ArXiv.

[31]  Jure Leskovec,et al.  Hierarchical Graph Representation Learning with Differentiable Pooling , 2018, NeurIPS.

[32]  Kurt Mehlhorn,et al.  Weisfeiler-Lehman Graph Kernels , 2011, J. Mach. Learn. Res..

[33]  Fei Wang,et al.  Structural Deep Embedding for Hyper-Networks , 2017, AAAI.

[34]  Pineda,et al.  Generalization of back-propagation to recurrent neural networks. , 1987, Physical review letters.

[35]  Yoshua Bengio,et al.  Towards Gene Expression Convolutions using Gene Interaction Graphs , 2018, ArXiv.

[36]  L. Asz.  Random Walks on Graphs: a Survey , 2022 .

[37]  Svetha Venkatesh,et al.  Column Networks for Collective Classification , 2016, AAAI.

[38]  Siegfried Gottwald,et al.  Fuzzy Sets and Fuzzy Logic , 1993 .

[39]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[40]  M. J. D. Powell,et al.  An efficient method for finding the minimum of a function of several variables without calculating derivatives , 1964, Comput. J..

[41]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[42]  Ulrike von Luxburg,et al.  A tutorial on spectral clustering , 2007, Stat. Comput..

[43]  Hongxia Yang,et al.  Learning Disentangled Representations for Recommendation , 2019, NeurIPS.

[44]  Jure Leskovec,et al.  Inductive Representation Learning on Large Graphs , 2017, NIPS.

[45]  Pietro Liò,et al.  Graph Attention Networks , 2017, ICLR.

[46]  Wenwu Zhu,et al.  Power up! Robust Graph Convolutional Network against Evasion Attacks based on Graph Powering , 2019, ArXiv.

[47]  Qiang Ma,et al.  Dual Graph Convolutional Networks for Graph-Based Semi-Supervised Classification , 2018, WWW.

[48]  Alessandro Rozza,et al.  Dynamic Graph Convolutional Networks , 2017, Pattern Recognit..

[49]  Jan Eric Lenssen,et al.  Fast Graph Representation Learning with PyTorch Geometric , 2019, ArXiv.

[50]  Fei Wang,et al.  Patient Subtyping via Time-Aware LSTM Networks , 2017, KDD.

[51]  Pierre Vandergheynst,et al.  Wavelets on Graphs via Spectral Graph Theory , 2009, ArXiv.

[52]  Yishay Mansour,et al.  Policy Gradient Methods for Reinforcement Learning with Function Approximation , 1999, NIPS.

[53]  Lise Getoor,et al.  Collective Classification in Network Data , 2008, AI Mag..

[54]  Zhiyuan Liu,et al.  Graph Neural Networks: A Review of Methods and Applications , 2018, AI Open.

[55]  L. B. Almeida.  A learning rule for asynchronous perceptrons with feedback in a combinatorial environment , 1990 .

[56]  Lina Yao,et al.  Adversarially Regularized Graph Autoencoder , 2018, IJCAI.

[57]  Nitish Srivastava,et al.  Dropout: a simple way to prevent neural networks from overfitting , 2014, J. Mach. Learn. Res..

[58]  Alexander J. Smola,et al.  Go for a Walk and Arrive at the Answer: Reasoning Over Paths in Knowledge Bases using Reinforcement Learning , 2017, ICLR.

[59]  Stephan Günnemann,et al.  Deep Gaussian Embedding of Graphs: Unsupervised Inductive Learning via Ranking , 2017, ICLR.

[60]  Samy Bengio,et al.  Order Matters: Sequence to sequence for sets , 2015, ICLR.

[61]  Lukasz Kaiser,et al.  Attention is All you Need , 2017, NIPS.

[62]  Stephan Günnemann,et al.  NetGAN: Generating Graphs via Random Walks , 2018, ICML.

[63]  Nicola De Cao,et al.  MolGAN: An implicit generative model for small molecular graphs , 2018, ArXiv.

[64]  Junzhou Huang,et al.  Adaptive Sampling Towards Fast Graph Representation Learning , 2018, NeurIPS.

[65]  Mingzhe Wang,et al.  LINE: Large-scale Information Network Embedding , 2015, WWW.

[66]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[67]  Wei Lu,et al.  Deep Neural Networks for Learning Graph Representations , 2016, AAAI.

[68]  Mikhail Belkin,et al.  Laplacian Eigenmaps and Spectral Techniques for Embedding and Clustering , 2001, NIPS.

[69]  Stephan Günnemann,et al.  Pitfalls of Graph Neural Network Evaluation , 2018, ArXiv.

[70]  Jure Leskovec,et al.  Graph Convolutional Neural Networks for Web-Scale Recommender Systems , 2018, KDD.

[71]  Mario Vento,et al.  A (sub)graph isomorphism algorithm for matching large graphs , 2004, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[72]  Luís C. Lamb,et al.  Learning to Solve NP-Complete Problems - A Graph Neural Network for the Decision TSP , 2018, AAAI.

[73]  Joan Bruna,et al.  Community Detection with Graph Neural Networks , 2017, 1705.08415.

[74]  Xavier Bresson,et al.  CayleyNets: Graph Convolutional Neural Networks With Complex Rational Spectral Filters , 2017, IEEE Transactions on Signal Processing.

[75]  Mathias Niepert,et al.  Learning Convolutional Neural Networks for Graphs , 2016, ICML.

[76]  Nikos Komodakis,et al.  Dynamic Edge-Conditioned Filters in Convolutional Neural Networks on Graphs , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[77]  Cyrus Shahabi,et al.  Diffusion Convolutional Recurrent Neural Network: Data-Driven Traffic Forecasting , 2017, ICLR.

[78]  Le Song,et al.  Discriminative Embeddings of Latent Variable Models for Structured Data , 2016, ICML.

[79]  Joan Bruna,et al.  Deep Convolutional Networks on Graph-Structured Data , 2015, ArXiv.

[80]  Philip S. Yu,et al.  Adversarial Attack and Defense on Graph Data: A Survey , 2018 .

[81]  Ryan A. Rossi,et al.  Attention Models in Graphs , 2018, ACM Trans. Knowl. Discov. Data.

[82]  Xavier Bresson,et al.  Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering , 2016, NIPS.

[83]  Zhi-Li Zhang,et al.  Stability and Generalization of Graph Convolutional Neural Networks , 2019, KDD.

[84]  Stephan Günnemann,et al.  Adversarial Attacks on Neural Networks for Graph Data , 2018, KDD.

[85]  Minyi Guo,et al.  GraphGAN: Graph Representation Learning with Generative Adversarial Nets , 2017, AAAI.

[86]  Alessandro Sperduti,et al.  A general framework for adaptive processing of data structures , 1998, IEEE Trans. Neural Networks.

[87]  Samuel S. Schoenholz,et al.  Neural Message Passing for Quantum Chemistry , 2017, ICML.

[88]  Wenwu Zhu,et al.  Structural Deep Network Embedding , 2016, KDD.

[89]  Pierre Vandergheynst,et al.  Graph Signal Processing: Overview, Challenges, and Applications , 2017, Proceedings of the IEEE.

[90]  Rob Fergus,et al.  Learning Multiagent Communication with Backpropagation , 2016, NIPS.

[91]  Alex Smola,et al.  Deep Graph Library: Towards Efficient and Scalable Deep Learning on Graphs , 2019, ArXiv.

[92]  Ryan A. Rossi,et al.  Graph Classification using Structural Attention , 2018, KDD.

[93]  Max Welling,et al.  Auto-Encoding Variational Bayes , 2013, ICLR.

[94]  Razvan Pascanu,et al.  Interaction Networks for Learning about Objects, Relations and Physics , 2016, NIPS.

[95]  Pascal Frossard,et al.  The emerging field of signal processing on graphs: Extending high-dimensional data analysis to networks and other irregular domains , 2012, IEEE Signal Processing Magazine.

[96]  Philip S. Yu,et al.  Deep Recursive Network Embedding with Regular Equivalence , 2018, KDD.

[97]  Wenwu Zhu,et al.  Robust Graph Convolutional Networks Against Adversarial Attacks , 2019, KDD.

[98]  Jonathan Masci,et al.  Geometric Deep Learning on Graphs and Manifolds Using Mixture Model CNNs , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[99]  Jure Leskovec,et al.  Modeling Polypharmacy Side Effects with Graph Convolutional Networks , 2018 .

[100]  Yoshua Bengio,et al.  Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation , 2014, EMNLP.

[101]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[102]  Yedid Hoshen,et al.  VAIN: Attentional Multi-agent Predictive Modeling , 2017, NIPS.

[103]  Jiliang Tang,et al.  Signed Graph Convolutional Network , 2018, ArXiv.

[104]  Zhuwen Li,et al.  Combinatorial Optimization with Graph Convolutional Networks and Guided Tree Search , 2018, NeurIPS.

[105]  Jure Leskovec,et al.  Graph Convolutional Policy Network for Goal-Directed Molecular Graph Generation , 2018, NeurIPS.

[106]  Cao Xiao,et al.  FastGCN: Fast Learning with Graph Convolutional Networks via Importance Sampling , 2018, ICLR.

[107]  Jure Leskovec,et al.  How Powerful are Graph Neural Networks? , 2018, ICLR.

[108]  Richard S. Zemel,et al.  Gated Graph Sequence Neural Networks , 2015, ICLR.

[109]  Yuxiao Dong,et al.  DeepInf : Modeling Influence Locality in Large Social Networks , 2018 .

[110]  Philip S. Yu,et al.  A Comprehensive Survey on Graph Neural Networks , 2019, IEEE Transactions on Neural Networks and Learning Systems.

[111]  Donald F. Towsley,et al.  Diffusion-Convolutional Neural Networks , 2015, NIPS.

[112]  Alán Aspuru-Guzik,et al.  Convolutional Networks on Graphs for Learning Molecular Fingerprints , 2015, NIPS.

[113]  Xavier Bresson,et al.  Geometric Matrix Completion with Recurrent Multi-Graph Neural Networks , 2017, NIPS.

[114]  Max Welling,et al.  Modeling Relational Data with Graph Convolutional Networks , 2017, ESWC.

[115]  George J. Klir,et al.  Fuzzy sets and fuzzy logic , 1995 .

[116]  Xueqi Cheng,et al.  Graph Wavelet Neural Network , 2019, ICLR.

[117]  Joan Bruna,et al.  Few-Shot Learning with Graph Neural Networks , 2017, ICLR.

[118]  Jie Zhang,et al.  Semi-supervised Learning on Graphs with Generative Adversarial Nets , 2018, CIKM.

[119]  Jure Leskovec,et al.  GNN Explainer: A Tool for Post-hoc Explanation of Graph Neural Networks , 2019, ArXiv.

[120]  Ah Chung Tsoi,et al.  The Vapnik-Chervonenkis dimension of graph and recursive neural networks , 2018, Neural Networks.

[121]  Ben Glocker,et al.  Spectral Graph Convolutions for Population-based Disease Prediction , 2017, MICCAI.

[122]  Yuxin Chen,et al.  Learning to Decipher the Heap for Program Verification , 2016 .

[123]  Wenwu Zhu,et al.  DepthLGP: Learning Embeddings of Out-of-Sample Nodes in Dynamic Networks , 2018, AAAI.

[124]  Geoffrey E. Hinton,et al.  Learning representations by back-propagating errors , 1986, Nature.

[125]  Song-Chun Zhu,et al.  Learning Human-Object Interactions by Graph Parsing Neural Networks , 2018, ECCV.

[126]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[127]  Jure Leskovec,et al.  Representation Learning on Graphs: Methods and Applications , 2017, IEEE Data Eng. Bull..

[128]  Vijay S. Pande,et al.  Molecular graph convolutions: moving beyond fingerprints , 2016, Journal of Computer-Aided Molecular Design.

[129]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[130]  Svetha Venkatesh,et al.  Graph Transformation Policy Network for Chemical Reaction Prediction , 2018, KDD.

[131]  Wenhan Xiong,et al.  DeepPath: A Reinforcement Learning Method for Knowledge Graph Reasoning , 2017, EMNLP.

[132]  Jiliang Tang,et al.  Streaming Graph Neural Networks , 2018, SIGIR.

[133]  Razvan Pascanu,et al.  Learning Deep Generative Models of Graphs , 2018, ICLR 2018.

[134]  Hanghang Tong,et al.  Graph Convolutional Networks: Algorithms, Applications and Open Challenges , 2018, CSoNet.

[135]  Zhanxing Zhu,et al.  Spatio-temporal Graph Convolutional Neural Network: A Deep Learning Framework for Traffic Forecasting , 2017, IJCAI.

[136]  Max Welling,et al.  Variational Graph Auto-Encoders , 2016, ArXiv.

[137]  Silvio Savarese,et al.  Structural-RNN: Deep Learning on Spatio-Temporal Graphs , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[138]  Abhinav Gupta,et al.  The More You Know: Using Knowledge Graphs for Image Classification , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[139]  Xiao-Ming Wu,et al.  Deeper Insights into Graph Convolutional Networks for Semi-Supervised Learning , 2018, AAAI.

[140]  Ben Glocker,et al.  Distance Metric Learning Using Graph Convolutional Networks: Application to Functional Brain Networks , 2017, MICCAI.

[141]  Omer Levy,et al.  Neural Word Embedding as Implicit Matrix Factorization , 2014, NIPS.

[142]  Khalil Sima'an,et al.  Graph Convolutional Encoders for Syntax-aware Neural Machine Translation , 2017, EMNLP.

[143]  Razvan Pascanu,et al.  Relational inductive biases, deep learning, and graph networks , 2018, ArXiv.

[144]  Jian Pei,et al.  A Survey on Network Embedding , 2017, IEEE Transactions on Knowledge and Data Engineering.

[145]  Pietro Liò,et al.  Deep Graph Infomax , 2018, ICLR.

[146]  Hao Ma,et al.  GaAN: Gated Attention Networks for Learning on Large and Spatiotemporal Graphs , 2018, UAI.

[147]  Sanja Fidler,et al.  3D Graph Neural Networks for RGBD Semantic Segmentation , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[148]  Brendan D. McKay,et al.  Practical graph isomorphism, II , 2013, J. Symb. Comput..

[149]  Yoshua Bengio,et al.  Neural Machine Translation by Jointly Learning to Align and Translate , 2014, ICLR.

[150]  Fu Jie Huang,et al.  A Tutorial on Energy-Based Learning , 2006 .

[151]  Yoshua Bengio,et al.  Extracting and composing robust features with denoising autoencoders , 2008, ICML '08.

[152]  Regina Barzilay,et al.  Convolutional Embedding of Attributed Molecular Graphs for Physical Property Prediction , 2017, J. Chem. Inf. Model..

[153]  Le Song,et al.  Adversarial Attack on Graph Structured Data , 2018, ICML.

[154]  Christos Faloutsos,et al.  Beyond Sigmoids: The NetTide Model for Social Network Growth, and Its Applications , 2016, KDD.

[155]  Pierre Vandergheynst,et al.  Geometric Deep Learning: Going beyond Euclidean data , 2016, IEEE Signal Process. Mag..

[156]  Charu C. Aggarwal,et al.  Learning Deep Network Representations with Adversarially Regularized Autoencoders , 2018, KDD.

[157]  Jian Pei,et al.  AutoNE: Hyperparameter Optimization for Massive Network Embedding , 2019, KDD.

[158]  Stephan Günnemann,et al.  Adversarial Attacks on Graph Neural Networks via Meta Learning , 2019, ICLR.

[159]  Steven Skiena,et al.  DeepWalk: online learning of social representations , 2014, KDD.

[160]  Diego Marcheggiani,et al.  Encoding Sentences with Graph Convolutional Networks for Semantic Role Labeling , 2017, EMNLP.

[161]  Zhengyang Wang,et al.  Large-Scale Learnable Graph Convolutional Networks , 2018, KDD.

[162]  Kilian Q. Weinberger,et al.  Simplifying Graph Convolutional Networks , 2019, ICML.

[163]  R. A. Leibler,et al.  On Information and Sufficiency , 1951 .

[164]  Marc Brockschmidt,et al.  Learning to Represent Programs with Graphs , 2017, ICLR.

[165]  Joan Bruna,et al.  Spectral Networks and Locally Connected Networks on Graphs , 2013, ICLR.

[166]  Robert Tibshirani,et al.  The Elements of Statistical Learning: Data Mining, Inference, and Prediction, 2nd Edition , 2001, Springer Series in Statistics.

[167]  J. MacQueen.  Some methods for classification and analysis of multivariate observations , 1967 .

[168]  Jure Leskovec,et al.  GraphRNN: Generating Realistic Graphs with Deep Auto-regressive Models , 2018, ICML.

[169]  D. Ruppert.  The Elements of Statistical Learning: Data Mining, Inference, and Prediction , 2004 .

[170]  Max Welling,et al.  Graph Convolutional Matrix Completion , 2017, ArXiv.

[171]  Max Welling,et al.  Semi-Supervised Classification with Graph Convolutional Networks , 2016, ICLR.

[172]  Stephen Lin,et al.  Graph Embedding and Extensions: A General Framework for Dimensionality Reduction , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[173]  Ken-ichi Kawarabayashi,et al.  Representation Learning on Graphs with Jumping Knowledge Networks , 2018, ICML.