A Gentle Introduction to Deep Learning for Graphs

The adaptive processing of graph data is a long-standing research topic that has been lately consolidated as a theme of major interest in the deep learning community. The snap increase in the amount and breadth of related research has come at the price of little systematization of knowledge and attention to earlier literature. This work is a tutorial introduction to the field of deep learning for graphs. It favors a consistent and progressive presentation of the main concepts and architectural aspects over an exposition of the most recent literature, for which the reader is referred to available surveys. The paper takes a top-down view of the problem, introducing a generalized formulation of graph representation learning based on a local and iterative approach to structured information processing. Moreover, it introduces the basic building blocks that can be combined to design novel and effective neural models for graphs. We complement the methodological exposition with a discussion of interesting research challenges and applications in the field.

[1]  Lise Getoor,et al.  Collective Classification in Network Data , 2008, AI Mag..

[2]  Nicola De Cao,et al.  MolGAN: An implicit generative model for small molecular graphs , 2018, ArXiv.

[3]  Joan Bruna,et al.  Spectral Networks and Locally Connected Networks on Graphs , 2013, ICLR.

[4]  Horst Bunke,et al.  Self-organizing maps for learning the edit costs in graph matching , 2005, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[5]  Lise Getoor,et al.  Query-driven Active Surveying for Collective Classification , 2012 .

[6]  Davide Bacciu,et al.  Graph generation by sequential edge prediction , 2019, ESANN.

[7]  Yue Gao,et al.  Dynamic Hypergraph Neural Networks , 2019, IJCAI.

[8]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[9]  Tat-Seng Chua,et al.  Graph Adversarial Training: Dynamically Regularizing Based on Graph Structure , 2019, IEEE Transactions on Knowledge and Data Engineering.

[10]  Alessio Micheli,et al.  Neural Network for Graphs: A Contextual Constructive Approach , 2009, IEEE Transactions on Neural Networks.

[11]  Abhinav Gupta,et al.  Videos as Space-Time Region Graphs , 2018, ECCV.

[12]  Alessio Micheli,et al.  Contextual processing of structured data by recursive cascade correlation , 2004, IEEE Transactions on Neural Networks.

[13]  Alessio Micheli,et al.  Universal Approximation Capability of Cascade Correlation for Structures , 2005, Neural Computation.

[14]  M. T. Le,et al.  Towards Graph Pooling by Edge Contraction , 2019 .

[15]  Lukasz Kaiser,et al.  Attention is All you Need , 2017, NIPS.

[16]  Alessio Micheli,et al.  Recursive self-organizing network models , 2004, Neural Networks.

[17]  Ah Chung Tsoi,et al.  A self-organizing map for adaptive processing of structured data , 2003, IEEE Trans. Neural Networks.

[18]  Yue Gao,et al.  Hypergraph Neural Networks , 2018, AAAI.

[19]  Diego Marcheggiani,et al.  Exploiting Semantics in Neural Machine Translation with Graph Convolutional Networks , 2018, NAACL.

[20]  Jonathon S. Hare,et al.  Deep Cascade Learning , 2018, IEEE Transactions on Neural Networks and Learning Systems.

[21]  Claudio Gallicchio,et al.  Fast and Deep Graph Neural Networks , 2019, AAAI.

[22]  Shuiwang Ji,et al.  Graph U-Nets , 2019, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[23]  Samuel S. Schoenholz,et al.  Neural Message Passing for Quantum Chemistry , 2017, ICML.

[24]  Alessandro Sperduti,et al.  Supervised neural networks for the classification of structures , 1997, IEEE Trans. Neural Networks.

[25]  Davide Bacciu,et al.  Deep Tree Transductions - A Short Survey , 2019, INNSBDDL.

[26]  Michael A. Osborne,et al.  On the Limitations of Representing Functions on Sets , 2019, ICML.

[27]  Xiao-Ming Wu,et al.  Deeper Insights into Graph Convolutional Networks for Semi-Supervised Learning , 2018, AAAI.

[28]  George Karypis,et al.  Comparison of descriptor spaces for chemical compound retrieval and classification , 2006, Sixth International Conference on Data Mining (ICDM'06).

[29]  Jure Leskovec,et al.  Graph Convolutional Neural Networks for Web-Scale Recommender Systems , 2018, KDD.

[30]  Andrew L. Maas.  Rectifier Nonlinearities Improve Neural Network Acoustic Models , 2013 .

[31]  Ah Chung Tsoi,et al.  The Graph Neural Network Model , 2009, IEEE Transactions on Neural Networks.

[32]  B. Bollobás.  The evolution of random graphs , 1984 .

[33]  Pietro Liò,et al.  Deep Graph Infomax , 2018, ICLR.

[34]  Davide Bacciu,et al.  Compositional Generative Mapping for Tree-Structured Data—Part I: Bottom-Up Probabilistic Modeling of Trees , 2012, IEEE Transactions on Neural Networks and Learning Systems.

[35]  Dongsup Kim,et al.  FP2VEC: a new molecular featurizer for learning molecular properties , 2019, Bioinform..

[36]  Luc De Raedt,et al.  kLog: A Language for Logical and Relational Learning with Kernels (Extended Abstract) , 2012, IJCAI.

[37]  Trevor Hastie,et al.  The Elements of Statistical Learning , 2001 .

[38]  Yue Wang,et al.  Dynamic Graph CNN for Learning on Point Clouds , 2018, ACM Trans. Graph..

[39]  Stefano Ermon,et al.  Graphite: Iterative Generative Modeling of Graphs , 2018, ICML.

[40]  Fabio Roli,et al.  Wild Patterns: Ten Years After the Rise of Adversarial Machine Learning , 2018, CCS.

[41]  Christopher D. Manning,et al.  Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networks , 2015, ACL.

[42]  Daniele Calandriello,et al.  Improved Large-Scale Graph Learning through Ridge Spectral Sparsification , 2018, ICML.

[43]  Claudio Giuliano,et al.  SocialLink: exploiting graph embeddings to link DBpedia entities to Twitter profiles , 2018, Progress in Artificial Intelligence.

[44]  Alessio Micheli,et al.  An introduction to recursive neural networks and kernel methods for cheminformatics. , 2007, Current pharmaceutical design.

[45]  Jure Leskovec,et al.  Modeling Polypharmacy Side Effects with Graph Convolutional Networks , 2018 .

[46]  Ashwin Srinivasan,et al.  The Predictive Toxicology Challenge 2000-2001 , 2001, Bioinform..

[47]  Pierre Vandergheynst,et al.  Wavelets on Graphs via Spectral Graph Theory , 2009, ArXiv.

[48]  Andrew Y. Ng,et al.  Parsing Natural Scenes and Natural Language with Recursive Neural Networks , 2011, ICML.

[49]  Ben Poole,et al.  Categorical Reparametrization with Gumble-Softmax , 2017, ICLR 2017.

[50]  Kan Li,et al.  A deeper graph neural network for recommender systems , 2019, Knowl. Based Syst..

[51]  Bernhard Schölkopf,et al.  Wasserstein Auto-Encoders , 2017, ICLR.

[52]  Stephan Günnemann,et al.  Pitfalls of Graph Neural Network Evaluation , 2018, ArXiv.

[53]  So Young Sohn,et al.  Graph convolutional network approach applied to predict hourly bike-sharing demands considering spatial, temporal, and global effects , 2019, PloS one.

[54]  George Cybenko,et al.  Approximation by superpositions of a sigmoidal function , 1989, Math. Control. Signals Syst..

[55]  Pierre Baldi,et al.  Graph kernels for chemical informatics , 2005, Neural Networks.

[56]  Alessio Micheli,et al.  A general framework for unsupervised processing of structured data , 2004, Neurocomputing.

[57]  Edmondo Trentin,et al.  Recursive Neural Networks for Density Estimation Over Generalized Random Graphs , 2018, IEEE Transactions on Neural Networks and Learning Systems.

[58]  Steven Skiena,et al.  DeepWalk: online learning of social representations , 2014, KDD.

[59]  Diego Marcheggiani,et al.  Encoding Sentences with Graph Convolutional Networks for Semantic Role Labeling , 2017, EMNLP.

[60]  Richard S. Zemel,et al.  Gated Graph Sequence Neural Networks , 2015, ICLR.

[61]  P. Dobson,et al.  Distinguishing enzyme structures from non-enzymes without alignments. , 2003, Journal of molecular biology.

[62]  Antje Chang,et al.  BRENDA , the enzyme database : updates and major new developments , 2003 .

[63]  Regina Barzilay,et al.  Junction Tree Variational Autoencoder for Molecular Graph Generation , 2018, ICML.

[64]  Yoshua Bengio,et al.  GMNN: Graph Markov Neural Networks , 2019, ICML.

[65]  Nikos Komodakis,et al.  Dynamic Edge-Conditioned Filters in Convolutional Neural Networks on Graphs , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[66]  Pinar Yanardag,et al.  Deep Graph Kernels , 2015, KDD.

[67]  Max Welling,et al.  Modeling Relational Data with Graph Convolutional Networks , 2017, ESWC.

[68]  Jure Leskovec,et al.  Representation Learning on Graphs: Methods and Applications , 2017, IEEE Data Eng. Bull..

[69]  Edmondo Trentin,et al.  A Maximum-Likelihood Connectionist Model for Unsupervised Learning over Graphical Domains , 2009, ICANN.

[70]  Davide Bacciu,et al.  Edge-based sequential graph generation with recurrent neural networks , 2020, Neurocomputing.

[71]  Pietro Liò,et al.  Graph Attention Networks , 2017, ICLR.

[72]  Béla Bollobás,et al.  Random Graphs , 1985 .

[73]  Minyi Guo,et al.  GraphGAN: Graph Representation Learning with Generative Adversarial Nets , 2017, AAAI.

[74]  O. Chapelle,et al.  Semi-Supervised Learning (Chapelle, O. et al., Eds.; 2006) [Book reviews] , 2009, IEEE Transactions on Neural Networks.

[75]  Xiaochun Cao,et al.  Topology Optimization based Graph Convolutional Network , 2019, IJCAI.

[76]  Jonathan Blackledge.  Chapter 2 – 2D Fourier Theory , 2005 .

[77]  Matt J. Kusner,et al.  A Model to Search for Synthesizable Molecules , 2019, NeurIPS.

[78]  Lorenzo Livi,et al.  Concept Drift and Anomaly Detection in Graph Streams , 2017, IEEE Transactions on Neural Networks and Learning Systems.

[79]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[80]  László Lovász,et al.  Random Walks on Graphs: A Survey , 1993 .

[81]  Davide Bacciu,et al.  Contextual Graph Markov Model: A Deep and Generative Approach to Graph Processing , 2018, ICML.

[82]  Gholamreza Haffari,et al.  Graph-to-Sequence Learning using Gated Graph Neural Networks , 2018, ACL.

[83]  Kurt Mehlhorn,et al.  Efficient graphlet kernels for large graph comparison , 2009, AISTATS.

[84]  Edmondo Trentin,et al.  Nonparametric small random networks for graph-structured pattern recognition , 2018, Neurocomputing.

[85]  Alexander J. Smola,et al.  Deep Sets , 2017, 1703.06114.

[86]  Jan Eric Lenssen,et al.  Fast Graph Representation Learning with PyTorch Geometric , 2019, ArXiv.

[87]  Yoshua Bengio,et al.  Convolutional networks for images, speech, and time series , 1998 .

[88]  Xiaojin Zhu,et al.  Semi-Supervised Learning , 2010, Encyclopedia of Machine Learning.

[89]  Cao Xiao,et al.  FastGCN: Fast Learning with Graph Convolutional Networks via Importance Sampling , 2018, ICLR.

[90]  Jure Leskovec,et al.  Hierarchical Graph Representation Learning with Differentiable Pooling , 2018, NeurIPS.

[91]  Stephan Günnemann,et al.  NetGAN: Generating Graphs via Random Walks , 2018, ICML.

[92]  Ulrike von Luxburg,et al.  A tutorial on spectral clustering , 2007, Stat. Comput..

[93]  Daniel R. Figueiredo,et al.  struc2vec: Learning Node Representations from Structural Identity , 2017, KDD.

[94]  Nikos Komodakis,et al.  GraphVAE: Towards Generation of Small Graphs Using Variational Autoencoders , 2018, ICANN.

[95]  Hongwei Jin,et al.  Latent Adversarial Training of Graph Convolution Networks , 2019 .

[96]  Giacomo Iadarola.  Graph-based classification for detecting instances of bug patterns , 2018 .

[97]  Ken-ichi Kawarabayashi,et al.  Representation Learning on Graphs with Jumping Knowledge Networks , 2018, ICML.

[98]  J. Bobadilla,et al.  Recommender systems survey , 2013, Knowl. Based Syst..

[99]  Stephan Günnemann,et al.  Adversarial Attacks on Neural Networks for Graph Data , 2018, KDD.

[100]  Bernhard Schölkopf,et al.  Learning with Hypergraphs: Clustering, Classification, and Embedding , 2006, NIPS.

[101]  Zhanxing Zhu,et al.  Spatio-temporal Graph Convolutional Neural Network: A Deep Learning Framework for Traffic Forecasting , 2017, IJCAI.

[102]  Michael I. Jordan,et al.  Mixed Memory Markov Models: Decomposing Complex Stochastic Processes as Mixtures of Simpler Ones , 1999, Machine Learning.

[103]  Jure Leskovec,et al.  Inductive Representation Learning on Large Graphs , 2017, NIPS.

[104]  Ah Chung Tsoi,et al.  Self-Organizing Maps for cyclic and unbounded graphs , 2008, ESANN.

[105]  Kurt Mehlhorn,et al.  Weisfeiler-Lehman Graph Kernels , 2011, J. Mach. Learn. Res..

[106]  Christian Lebiere,et al.  The Cascade-Correlation Learning Architecture , 1989, NIPS.

[107]  Jure Leskovec,et al.  How Powerful are Graph Neural Networks? , 2018, ICLR.

[108]  Sepp Hochreiter,et al.  Untersuchungen zu dynamischen neuronalen Netzen , 1991 .

[109]  Claudio Gallicchio,et al.  Graph Echo State Networks , 2010, The 2010 International Joint Conference on Neural Networks (IJCNN).

[110]  Yu-Xiang Wang,et al.  Graph Sparsification Approaches for Laplacian Smoothing , 2016, AISTATS.

[111]  Wenwu Zhu,et al.  Deep Learning on Graphs: A Survey , 2018, IEEE Transactions on Knowledge and Data Engineering.

[112]  Yoshua Bengio,et al.  Learning long-term dependencies with gradient descent is difficult , 1994, IEEE Trans. Neural Networks.

[113]  Razvan Pascanu,et al.  Learning Deep Generative Models of Graphs , 2018, ICLR 2018.

[114]  Alessandro Sperduti,et al.  A general framework for adaptive processing of data structures , 1998, IEEE Trans. Neural Networks.

[115]  Niloy Ganguly,et al.  NeVAE: A Deep Generative Model for Molecular Graphs , 2018, AAAI.

[116]  Giuseppe Antonio Di Luna,et al.  SAFE: Self-Attentive Function Embeddings for Binary Similarity , 2018, DIMVA.

[117]  Jaewoo Kang,et al.  Self-Attention Graph Pooling , 2019, ICML.

[118]  Qi Liu,et al.  Constrained Graph Variational Autoencoders for Molecule Design , 2018, NeurIPS.

[119]  Jure Leskovec,et al.  GraphRNN: Generating Realistic Graphs with Deep Auto-regressive Models , 2018, ICML.

[120]  Inderjit S. Dhillon,et al.  Weighted Graph Cuts without Eigenvectors A Multilevel Approach , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[121]  Davide Bacciu,et al.  A Non-Negative Factorization approach to node pooling in Graph Convolutional Neural Networks , 2019, AI*IA.

[122]  Alán Aspuru-Guzik,et al.  Convolutional Networks on Graphs for Learning Molecular Fingerprints , 2015, NIPS.

[123]  Xavier Bresson,et al.  Geometric Matrix Completion with Recurrent Multi-Graph Neural Networks , 2017, NIPS.

[124]  A. Debnath,et al.  Structure-activity relationship of mutagenic aromatic and heteroaromatic nitro compounds. Correlation with molecular orbital energies and hydrophobicity. , 1991, Journal of medicinal chemistry.

[125]  Jure Leskovec,et al.  node2vec: Scalable Feature Learning for Networks , 2016, KDD.

[126]  Si Zhang,et al.  Graph convolutional networks: a comprehensive review , 2019, Computational Social Networks.

[127]  Corinna Cortes,et al.  Support-Vector Networks , 1995, Machine Learning.

[128]  Philip S. Yu,et al.  A Comprehensive Survey on Graph Neural Networks , 2019, IEEE Transactions on Neural Networks and Learning Systems.

[129]  Yixin Chen,et al.  An End-to-End Deep Learning Architecture for Graph Classification , 2018, AAAI.

[130]  Foster J. Provost,et al.  Classification in Networked Data: a Toolkit and a Univariate Case Study , 2007, J. Mach. Learn. Res..

[131]  Sergey Ivanov,et al.  Anonymous Walk Embeddings , 2018, ICML.

[132]  Hans-Peter Kriegel,et al.  Protein function prediction via graph kernels , 2005, ISMB.

[133]  Alexander Zien,et al.  Semi-Supervised Learning , 2006 .

[134]  Pierre Vandergheynst,et al.  Geometric Deep Learning: Going beyond Euclidean data , 2016, IEEE Signal Process. Mag..

[135]  Helen Yannakoudakis,et al.  Neural Character-based Composition Models for Abuse Detection , 2018, ALW.

[136]  Yue Gao,et al.  Dynamic Hypergraph Structure Learning , 2018, IJCAI.

[137]  Razvan Pascanu,et al.  Relational inductive biases, deep learning, and graph networks , 2018, ArXiv.

[138]  Davide Bacciu,et al.  A Fair Comparison of Graph Neural Networks for Graph Classification , 2020, ICLR.

[139]  Max Welling,et al.  Auto-Encoding Variational Bayes , 2013, ICLR.

[140]  Seokho Kang,et al.  Efficient learning of non-autoregressive graph variational autoencoders for molecular graph generation , 2019, Journal of Cheminformatics.

[141]  Teuvo Kohonen,et al.  The self-organizing map , 1990 .

[142]  Alessio Micheli,et al.  Application of Cascade Correlation Networks for Structures to Chemistry , 2004, Applied Intelligence.

[143]  S. V. N. Vishwanathan,et al.  Graph kernels , 2007 .

[144]  Alexander J. Smola,et al.  Deep Graph Library: Towards Efficient and Scalable Deep Learning on Graphs , 2019, ArXiv.

[145]  Max Welling,et al.  Variational Graph Auto-Encoders , 2016, ArXiv.

[146]  Max Welling,et al.  Semi-Supervised Classification with Graph Convolutional Networks , 2016, ICLR.