Natural Graph Networks

Conventional neural message passing algorithms are invariant under permutation of the messages and hence forget how the information flows through the network. Studying the local symmetries of graphs, we propose a more general algorithm that uses different kernels on different edges, making the network equivariant to local and global graph isomorphisms and hence more expressive. Using elementary category theory, we formalize many distinct equivariant neural networks as natural networks, and show that their kernels are 'just' a natural transformation between two functors. We give one practical instantiation of a natural network on graphs which uses a equivariant message network parameterization, yielding good performance on several benchmarks.

[1]  S. V. N. Vishwanathan,et al.  Graph kernels , 2007 .

[2]  Max Welling,et al.  Semi-Supervised Classification with Graph Convolutional Networks , 2016, ICLR.

[3]  Kurt Mehlhorn,et al.  Weisfeiler-Lehman Graph Kernels , 2011, J. Mach. Learn. Res..

[4]  Max Welling,et al.  Group Equivariant Convolutional Networks , 2016, ICML.

[5]  Roman Garnett,et al.  Propagation kernels: efficient graph kernels from propagated information , 2015, Machine Learning.

[6]  Siamak Ravanbakhsh,et al.  Incidence Networks for Geometric Deep Learning , 2019, ICML 2020.

[7]  Stefanos Zafeiriou,et al.  Improving Graph Neural Network Expressivity via Subgraph Isomorphism Counting , 2020, ArXiv.

[8]  Nikos Komodakis,et al.  Dynamic Edge-Conditioned Filters in Convolutional Neural Networks on Graphs , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[9]  Kevin Leyton-Brown,et al.  Deep Models of Interactions Across Sets , 2018, ICML.

[10]  Max Welling,et al.  Modeling Relational Data with Graph Convolutional Networks , 2017, ESWC.

[11]  Martin Grohe,et al.  Weisfeiler and Leman Go Neural: Higher-order Graph Neural Networks , 2018, AAAI.

[12]  Axel Poigné,et al.  Basic category theory , 1993, LICS 1993.

[13]  Urs Schreiber,et al.  Parallel Transport and Functors , 2007, 0705.0452.

[14]  David I. Spivak,et al.  Seven Sketches in Compositionality: An Invitation to Applied Category Theory , 2018, 1803.05316.

[15]  Jure Leskovec,et al.  How Powerful are Graph Neural Networks? , 2018, ICLR.

[16]  Mathias Niepert,et al.  Learning Convolutional Neural Networks for Graphs , 2016, ICML.

[17]  Nathanael Perraudin,et al.  DeepSphere: Efficient spherical Convolutional Neural Network with HEALPix sampling for cosmological applications , 2018, Astron. Comput..

[18]  Samuel S. Schoenholz,et al.  Neural Message Passing for Quantum Chemistry , 2017, ICML.

[19]  Zhi-Li Zhang,et al.  Hunt For The Unique, Stable, Sparse And Fast Feature Learning On Graphs , 2017, NIPS.

[20]  Gabriel Peyré,et al.  Universal Invariant and Equivariant Graph Neural Networks , 2019, NeurIPS.

[21]  Risi Kondor,et al.  Covariant Compositional Networks For Learning Graphs , 2018, ICLR.

[22]  Max Welling,et al.  Gauge Equivariant Convolutional Networks and the Icosahedral CNN 1 , 2019 .

[23]  F. Scarselli,et al.  A new model for learning in graph domains , 2005, Proceedings. 2005 IEEE International Joint Conference on Neural Networks, 2005..

[24]  van den Berg,et al.  UvA-DARE (Digital Academic Modeling Relational Data with Graph Convolutional Networks Modeling Relational Data with Graph Convolutional Networks , 2017 .

[25]  Philip S. Yu,et al.  A Comprehensive Survey on Graph Neural Networks , 2019, IEEE Transactions on Neural Networks and Learning Systems.

[26]  Donald F. Towsley,et al.  Diffusion-Convolutional Neural Networks , 2015, NIPS.

[27]  Pinar Yanardag,et al.  Deep Graph Kernels , 2015, KDD.

[28]  Xavier Bresson,et al.  Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering , 2016, NIPS.

[29]  Yixin Chen,et al.  An End-to-End Deep Learning Architecture for Graph Classification , 2018, AAAI.

[30]  Joan Bruna,et al.  Spectral Networks and Locally Connected Networks on Graphs , 2013, ICLR.

[31]  Max Welling,et al.  Graph Convolutional Matrix Completion , 2017, ArXiv.

[32]  Kurt Mehlhorn,et al.  Efficient graphlet kernels for large graph comparison , 2009, AISTATS.

[33]  Jure Leskovec,et al.  Hierarchical Graph Representation Learning with Differentiable Pooling , 2018, NeurIPS.

[34]  Yaron Lipman,et al.  Provably Powerful Graph Networks , 2019, NeurIPS.

[35]  Sergey Ivanov,et al.  Anonymous Walk Embeddings , 2018, ICML.

[36]  Yaron Lipman,et al.  Invariant and Equivariant Graph Networks , 2018, ICLR.