Graph Neural Networks with Learnable Structural and Positional Representations

Graph neural networks (GNNs) have become the standard learning architectures for graphs. GNNs have been applied to numerous domains ranging from quantum chemistry, recommender systems to knowledge graphs and natural language processing. A major issue with arbitrary graphs is the absence of canonical positional information of nodes, which decreases the representation power of GNNs to distinguish e.g. isomorphic nodes and other graph symmetries. An approach to tackle this issue is to introduce Positional Encoding (PE) of nodes, and inject it into the input layer, like in Transformers. Possible graph PE are Laplacian eigenvectors. In this work, we propose to decouple structural and positional representations to make easy for the network to learn these two essential properties. We introduce a novel generic architecture which we call LSPE (Learnable Structural and Positional Encodings). We investigate several sparse and fully-connected (Transformer-like) GNNs, and observe a performance increase for molecular datasets, from 1.79% up to 64.14% when considering learnable PE for both GNN classes. 1

[1]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[2]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.

[3]  Mikhail Belkin,et al.  Laplacian Eigenmaps for Dimensionality Reduction and Data Representation , 2003, Neural Computation.

[4]  M. Kaufmann.  What Can Be Computed Locally ? , 2003 .

[5]  Jitendra Malik,et al.  Spectral grouping using the Nystrom method , 2004, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[6]  Alex Krizhevsky,et al.  Learning Multiple Layers of Features from Tiny Images , 2009 .

[7]  Ryan G. Coleman,et al.  ZINC: A Free Tool to Discover Chemistry for Biology , 2012, J. Chem. Inf. Model..

[8]  Evan Bolton,et al.  PubChem's BioAssay Database , 2011, Nucleic Acids Res..

[9]  Rongjie Lai,et al.  A Splitting Method for Orthogonality Constrained Problems , 2014, J. Sci. Comput..

[10]  Sergey Ioffe,et al.  Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift , 2015, ICML.

[11]  Rob Fergus,et al.  Learning Multiagent Communication with Backpropagation , 2016, NIPS.

[12]  Xavier Bresson,et al.  Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering , 2016, NIPS.

[13]  Lukasz Kaiser,et al.  Attention is All you Need , 2017, NIPS.

[14]  Max Welling,et al.  Semi-Supervised Classification with Graph Convolutional Networks , 2016, ICLR.

[15]  Jure Leskovec,et al.  Inductive Representation Learning on Large Graphs , 2017, NIPS.

[16]  Xavier Bresson,et al.  Residual Gated Graph ConvNets , 2017, ArXiv.

[17]  Vijay S. Pande,et al.  MoleculeNet: a benchmark for molecular machine learning , 2017, Chemical science.

[18]  Samuel S. Schoenholz,et al.  Neural Message Passing for Quantum Chemistry , 2017, ICML.

[19]  Jonathan Masci,et al.  Geometric Deep Learning on Graphs and Manifolds Using Mixture Model CNNs , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[20]  Pietro Liò,et al.  Graph Attention Networks , 2017, ICLR.

[21]  A. A. LEMAN,et al.  THE REDUCTION OF A GRAPH TO CANONICAL FORM AND THE ALGEBRA WHICH APPEARS THEREIN , 2018 .

[22]  Regina Barzilay,et al.  Junction Tree Variational Autoencoder for Molecular Graph Generation , 2018, ICML.

[23]  Yaron Lipman,et al.  Provably Powerful Graph Networks , 2019, NeurIPS.

[24]  Davide Eynard,et al.  Fake News Detection on Social Media using Geometric Deep Learning , 2019, ArXiv.

[25]  Natalia Gimelshein,et al.  PyTorch: An Imperative Style, High-Performance Deep Learning Library , 2019, NeurIPS.

[26]  Bernard Ghanem,et al.  DeepGCNs: Can GCNs Go As Deep As CNNs? , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[27]  Joan Bruna,et al.  On the equivalence between graph isomorphism testing and function approximation with GNNs , 2019, NeurIPS.

[28]  Jure Leskovec,et al.  How Powerful are Graph Neural Networks? , 2018, ICLR.

[29]  Yaron Lipman,et al.  Invariant and Equivariant Graph Networks , 2018, ICLR.

[30]  Jure Leskovec,et al.  Position-aware Graph Neural Networks , 2019, ICML.

[31]  Martin Grohe,et al.  Weisfeiler and Leman Go Neural: Higher-order Graph Neural Networks , 2018, AAAI.

[32]  Vinayak A. Rao,et al.  Relational Pooling for Graph Representations , 2019, ICML.

[33]  P. Battaglia,et al.  Learning Symbolic Physics with Graph Networks , 2019, ArXiv.

[34]  H. Kashima,et al.  Approximation Ratios of Graph Neural Networks for Combinatorial Problems , 2019, NeurIPS.

[35]  Alex Smola,et al.  Deep Graph Library: Towards Efficient and Scalable Deep Learning on Graphs , 2019, ArXiv.

[36]  Jure Leskovec,et al.  PinnerSage: Multi-Modal User Embedding Framework for Recommendations at Pinterest , 2020, KDD.

[37]  Dominique Beaini,et al.  Principal Neighbourhood Aggregation for Graph Nets , 2020, NeurIPS.

[38]  Kristian Kersting,et al.  TUDataset: A collection of benchmark datasets for learning with graphs , 2020, ArXiv.

[39]  Jure Leskovec,et al.  Distance Encoding: Design Provably More Powerful Neural Networks for Graph Representation Learning , 2020, NeurIPS.

[40]  Yoshua Bengio,et al.  Benchmarking Graph Neural Networks , 2023, J. Mach. Learn. Res..

[41]  Balasubramaniam Srinivasan,et al.  On the Equivalence between Positional Node Embeddings and Structural Graph Representations , 2019, ICLR.

[42]  J. Leskovec,et al.  Open Graph Benchmark: Datasets for Machine Learning on Graphs , 2020, NeurIPS.

[43]  Ryoma Sato,et al.  A Survey on The Expressive Power of Graph Neural Networks , 2020, ArXiv.

[44]  Andreas Loukas,et al.  What graph neural networks cannot learn: depth vs width , 2019, ICLR.

[45]  Xavier Bresson,et al.  A Generalization of Transformer Networks to Graphs , 2020, ArXiv.

[46]  Bernard Ghanem,et al.  DeeperGCN: All You Need to Train Deeper GCNs , 2020, ArXiv.

[47]  Emma J. Chory,et al.  A Deep Learning Approach to Antibiotic Discovery , 2020, Cell.

[48]  Jeremy B. R. Hayter,et al.  Utilising Graph Machine Learning within Drug Discovery and Development , 2020, ArXiv.

[49]  Buyue Qian,et al.  Graph Neural Network-Based Diagnosis Prediction , 2020, Big Data.

[50]  Pushmeet Kohli,et al.  Unveiling the predictive power of static structure in glassy systems , 2020 .

[51]  Sen Jia,et al.  How Much Position Information Do Convolutional Neural Networks Encode? , 2020, ICLR.

[52]  Zhiyuan Liu,et al.  Graph Neural Networks: A Review of Methods and Applications , 2018, AI Open.

[53]  Marc Lelarge,et al.  Expressive Power of Invariant and Equivariant Graph Neural Networks , 2020, ICLR.

[54]  Eran Yahav,et al.  On the Bottleneck of Graph Neural Networks and its Practical Implications , 2020, ICLR.

[55]  Yoshua Bengio,et al.  Machine Learning for Combinatorial Optimization: a Methodological Tour d'Horizon , 2018, Eur. J. Oper. Res..

[56]  Elias Boutros Khalil,et al.  Combinatorial optimization and reasoning with graph neural networks , 2021, IJCAI.

[57]  Di He,et al.  Do Transformers Really Perform Bad for Graph Representation? , 2021, ArXiv.

[58]  Djork-Arn'e Clevert,et al.  Parameterized Hypercomplex Graph Neural Networks for Graph Classification , 2021, ICANN.

[59]  Dominique Beaini,et al.  Rethinking Graph Transformers with Spectral Attention , 2021, NeurIPS.

[60]  Pan Li,et al.  On Positional and Structural Node Features for Graph Neural Networks on Non-attributed Graphs , 2021, ArXiv.

[61]  Martin Schmitt,et al.  Position Information in Transformers: An Overview , 2021, Computational Linguistics.

[62]  Dominique Beaini,et al.  Directional Graph Networks , 2020, ICML.

[63]  Julien Mairal,et al.  GraphiT: Encoding Graph Structure in Transformers , 2021, ArXiv.

[64]  M. Bronstein,et al.  Improving Graph Neural Network Expressivity via Subgraph Isomorphism Counting , 2020, IEEE Transactions on Pattern Analysis and Machine Intelligence.