Nested Graph Neural Networks

Graph neural network (GNN)’s success in graph classification is closely related to the Weisfeiler-Lehman (1-WL) algorithm. By iteratively aggregating neighboring node features to a center node, both 1-WL and GNN obtain a node representation that encodes a rooted subtree around the center node. These rooted subtree representations are then pooled into a single representation to represent the whole graph. However, rooted subtrees are of limited expressiveness to represent a nontree graph. To address it, we propose Nested Graph Neural Networks (NGNNs). NGNN represents a graph with rooted subgraphs instead of rooted subtrees, so that two graphs sharing many identical subgraphs (rather than subtrees) tend to have similar representations. The key is to make each node representation encode a subgraph around it more than a subtree. To achieve this, NGNN extracts a local subgraph around each node and applies a base GNN to each subgraph to learn a subgraph representation. The whole-graph representation is then obtained by pooling these subgraph representations. We provide a rigorous theoretical analysis showing that NGNN is strictly more powerful than 1-WL. In particular, we proved that NGNN can discriminate almost all r-regular graphs, where 1-WL always fails. Moreover, unlike other more powerful GNNs, NGNN only introduces a constantfactor higher time complexity than standard GNNs. NGNN is a plug-and-play framework that can be combined with various base GNNs. We test NGNN with different base GNNs on several benchmark datasets. NGNN uniformly improves their performance and shows highly competitive performance on all datasets.

[1]  Joan Bruna,et al.  Spectral Networks and Locally Connected Networks on Graphs , 2013, ICLR.

[2]  Yaron Lipman,et al.  On the Universality of Invariant Networks , 2019, ICML.

[3]  Karsten M. Borgwardt,et al.  The graphlet spectrum , 2009, ICML '09.

[4]  Peter J. Cameron,et al.  Strongly regular graphs , 2003 .

[5]  M. Randic,et al.  Resistance distance , 1993 .

[6]  Max Welling,et al.  Semi-Supervised Classification with Graph Convolutional Networks , 2016, ICLR.

[7]  Le Song,et al.  Discriminative Embeddings of Latent Variable Models for Structured Data , 2016, ICML.

[8]  Katsuhiko Ishiguro,et al.  Graph Warp Module: an Auxiliary Module for Boosting the Power of Graph Neural Networks , 2019, ArXiv.

[9]  Vinayak A. Rao,et al.  Relational Pooling for Graph Representations , 2019, ICML.

[11]  Stephan Günnemann,et al.  Directional Message Passing for Molecular Graphs , 2020, ICLR.

[12]  Alán Aspuru-Guzik,et al.  Convolutional Networks on Graphs for Learning Molecular Fingerprints , 2015, NIPS.

[13]  Xavier Bresson,et al.  Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering , 2016, NIPS.

[14]  Kurt Mehlhorn,et al.  Efficient graphlet kernels for large graph comparison , 2009, AISTATS.

[15]  Dominique Beaini,et al.  Directional Graph Networks , 2020, ICML.

[16]  Gabriel Peyré,et al.  Universal Invariant and Equivariant Graph Neural Networks , 2019, NeurIPS.

[17]  Kristina Lerman,et al.  MixHop: Higher-Order Graph Convolutional Architectures via Sparsified Neighborhood Mixing , 2019, ICML.

[18]  Andreas Loukas,et al.  What graph neural networks cannot learn: depth vs width , 2019, ICLR.

[19]  Yixin Chen,et al.  Link Prediction Based on Graph Neural Networks , 2018, NeurIPS.

[20]  Vijay S. Pande,et al.  MoleculeNet: a benchmark for molecular machine learning , 2017, Chemical science.

[21]  Roman Garnett,et al.  Propagation kernels: efficient graph kernels from propagated information , 2015, Machine Learning.

[22]  Stefanos Zafeiriou,et al.  Improving Graph Neural Network Expressivity via Subgraph Isomorphism Counting , 2020, ArXiv.

[23]  Pavlo O. Dral,et al.  Quantum chemistry structures and properties of 134 kilo molecules , 2014, Scientific Data.

[24]  Antje Chang,et al.  BRENDA , the enzyme database : updates and major new developments , 2003 .

[25]  J. Leskovec,et al.  Open Graph Benchmark: Datasets for Machine Learning on Graphs , 2020, NeurIPS.

[26]  Davide Bacciu,et al.  A Fair Comparison of Graph Neural Networks for Graph Classification , 2020, ICLR.

[27]  Yaron Lipman,et al.  Provably Powerful Graph Networks , 2019, NeurIPS.

[28]  Richard S. Zemel,et al.  Gated Graph Sequence Neural Networks , 2015, ICLR.

[29]  R'emy Brossard,et al.  Graph convolutions that can finally model local structure , 2020, ArXiv.

[30]  Joan Bruna,et al.  On the equivalence between graph isomorphism testing and function approximation with GNNs , 2019, NeurIPS.

[31]  P. Dobson,et al.  Distinguishing enzyme structures from non-enzymes without alignments. , 2003, Journal of molecular biology.

[32]  Jure Leskovec,et al.  Inductive Representation Learning on Large Graphs , 2017, NIPS.

[33]  Yixin Chen,et al.  An End-to-End Deep Learning Architecture for Graph Classification , 2018, AAAI.

[34]  Steven Skiena,et al.  DeepWalk: online learning of social representations , 2014, KDD.

[35]  Frederick R. Manby,et al.  OrbNet: Deep Learning for Quantum Chemistry Using Symmetry-Adapted Atomic-Orbital Features , 2020, The Journal of chemical physics.

[36]  Yixin Chen,et al.  Inductive Graph Pattern Learning for Recommender Systems Based on a Graph Neural Network , 2019, ArXiv.

[37]  Pietro Liò,et al.  Principal Neighbourhood Aggregation for Graph Nets , 2020, NeurIPS.

[38]  Jure Leskovec,et al.  Hierarchical Graph Representation Learning with Differentiable Pooling , 2018, NeurIPS.

[39]  Jan Eric Lenssen,et al.  Fast Graph Representation Learning with PyTorch Geometric , 2019, ArXiv.

[40]  A. Debnath,et al.  Structure-activity relationship of mutagenic aromatic and heteroaromatic nitro compounds. Correlation with molecular orbital energies and hydrophobicity. , 1991, Journal of medicinal chemistry.

[41]  Yixin Chen,et al.  Weisfeiler-Lehman Neural Machine for Link Prediction , 2017, KDD.

[42]  V. Prasanna,et al.  Deep Graph Neural Networks with Shallow Subgraph Samplers , 2020, ArXiv.

[43]  Christopher Morris,et al.  Weisfeiler and Leman go sparse: Towards scalable higher-order graph embeddings , 2020, NeurIPS.

[44]  Michalis Vazirgiannis,et al.  k-hop Graph Neural Networks , 2019, Neural Networks.

[45]  Jure Leskovec,et al.  How Powerful are Graph Neural Networks? , 2018, ICLR.

[46]  Jure Leskovec,et al.  Distance Encoding -- Design Provably More Powerful GNNs for Structural Representation Learning , 2020 .

[47]  S. V. N. Vishwanathan,et al.  Graph kernels , 2007 .

[48]  Samuel S. Schoenholz,et al.  Neural Message Passing for Quantum Chemistry , 2017, ICML.

[49]  Risi Kondor,et al.  Cormorant: Covariant Molecular Neural Networks , 2019, NeurIPS.

[50]  Thomas Lukasiewicz,et al.  The Surprising Power of Graph Neural Networks with Random Node Initialization , 2020, IJCAI.

[51]  Andreas Loukas,et al.  Building powerful and equivariant graph neural networks with structural message-passing , 2020, NeurIPS.

[52]  Bernard Ghanem,et al.  DeeperGCN: All You Need to Train Deeper GCNs , 2020, ArXiv.

[53]  Marc Lelarge,et al.  Characterizing the Expressive Power of Invariant and Equivariant Graph Neural Networks , 2020, ArXiv.

[54]  Ludovic Dos Santos,et al.  Coloring graph neural networks for node disambiguation , 2019, IJCAI.

[55]  David Haussler,et al.  Convolution kernels on discrete structures , 1999 .

[56]  Ah Chung Tsoi,et al.  The Graph Neural Network Model , 2009, IEEE Transactions on Neural Networks.

[57]  Jure Leskovec,et al.  Identity-aware Graph Neural Networks , 2021, AAAI.

[58]  Kurt Mehlhorn,et al.  Weisfeiler-Lehman Graph Kernels , 2011, J. Mach. Learn. Res..

[59]  Alexander J. Smola,et al.  Deep Sets , 2017, 1703.06114.

[60]  Hannu Toivonen,et al.  Statistical evaluation of the predictive toxicology challenge , 2000 .

[61]  Max Welling,et al.  Natural Graph Networks , 2020, NeurIPS.

[62]  Djork-Arn'e Clevert,et al.  Parameterized Hypercomplex Graph Neural Networks for Graph Classification , 2021, ICANN.

[63]  Hisashi Kashima,et al.  Random Features Strengthen Graph Neural Networks , 2020, SDM.

[64]  Joan Bruna,et al.  Can graph neural networks count substructures? , 2020, NeurIPS.

[65]  Ashwin Srinivasan,et al.  Statistical Evaluation of the Predictive Toxicology Challenge 2000-2001 , 2003, Bioinform..

[66]  Hans-Peter Kriegel,et al.  Shortest-path kernels on graphs , 2005, Fifth IEEE International Conference on Data Mining (ICDM'05).

[67]  Yaron Lipman,et al.  Invariant and Equivariant Graph Networks , 2018, ICLR.

[68]  Martin Grohe,et al.  Weisfeiler and Leman Go Neural: Higher-order Graph Neural Networks , 2018, AAAI.