A Collective Learning Framework to Boost GNN Expressiveness

Graph Neural Networks (GNNs) have recently been used for node and graph classification tasks with great success, but GNNs model dependencies among the attributes of nearby neighboring nodes rather than dependencies among observed node labels. In this work, we consider the task of inductive node classification using GNNs in supervised and semi-supervised settings, with the goal of incorporating label dependencies. Because current GNNs are not universal (i.e., most-expressive) graph representations, we propose a general collective learning approach to increase the representation power of any existing GNN. Our framework combines ideas from collective classification with self-supervised learning, and uses a Monte Carlo approach to sampling embeddings for inductive learning across graphs. We evaluate performance on five real-world network datasets and demonstrate consistent, significant improvement in node classification accuracy, for a variety of state-of-the-art GNNs.

[1]  Jennifer Neville,et al.  Iterative Classification in Relational Data , 2000 .

[2]  Stefanos Zafeiriou,et al.  Improving Graph Neural Network Expressivity via Subgraph Isomorphism Counting , 2020, ArXiv.

[3]  Joan Bruna,et al.  On the equivalence between graph isomorphism testing and function approximation with GNNs , 2019, NeurIPS.

[4]  Michael S. Bernstein,et al.  Designing and deploying online field experiments , 2014, WWW.

[5]  Bruno Ribeiro,et al.  Stochastic Gradient Descent for Relational Logistic Regression via Partial Network Crawls , 2017, ArXiv.

[6]  Balaraman Ravindran,et al.  HOPF: Higher Order Propagation Framework for Deep Collective Classification , 2018, ArXiv.

[7]  Jennifer Neville,et al.  Deep Collective Inference , 2017, AAAI.

[8]  Yoshua Bengio,et al.  GMNN: Graph Markov Neural Networks , 2019, ICML.

[9]  Patrick Pérez,et al.  Boosting Few-Shot Visual Learning With Self-Supervision , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[10]  Jennifer Neville,et al.  Learning relational probability trees , 2003, KDD '03.

[11]  Yaron Lipman,et al.  On the Universality of Invariant Networks , 2019, ICML.

[12]  Lise Getoor,et al.  Collective Classification in Network Data , 2008, AI Mag..

[13]  Bert Huang,et al.  Recurrent collective classification , 2017, Knowledge and Information Systems.

[14]  Qian Xu,et al.  Graph Random Neural Networks for Semi-Supervised Learning on Graphs , 2020, NeurIPS.

[15]  Martial Hebert,et al.  Shuffle and Learn: Unsupervised Learning Using Temporal Order Verification , 2016, ECCV.

[16]  Andrew McCallum,et al.  Introduction to Statistical Relational Learning , 2007 .

[17]  Jean-Loup Guillaume,et al.  Fast unfolding of communities in large networks , 2008, 0803.0476.

[18]  H. Robbins.  A Stochastic Approximation Method , 1951 .

[19]  Hans-Peter Kriegel,et al.  Protein function prediction via graph kernels , 2005, ISMB.

[20]  Yaron Lipman,et al.  Provably Powerful Graph Networks , 2019, NeurIPS.

[21]  Ming-Wei Chang,et al.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding , 2019, NAACL.

[22]  Marc Lelarge,et al.  Characterizing the Expressive Power of Invariant and Equivariant Graph Neural Networks , 2020, ArXiv.

[23]  Yaron Lipman,et al.  Invariant and Equivariant Graph Networks , 2018, ICLR.

[24]  Martin Grohe,et al.  Weisfeiler and Leman Go Neural: Higher-order Graph Neural Networks , 2018, AAAI.

[25]  Svetha Venkatesh,et al.  Column Networks for Collective Classification , 2016, AAAI.

[26]  Jennifer Neville,et al.  Pseudolikelihood EM for Within-network Relational Learning , 2008, 2008 Eighth IEEE International Conference on Data Mining.

[27]  Jure Leskovec,et al.  Inductive Representation Learning on Large Graphs , 2017, NIPS.

[28]  Ming-Hsuan Yang,et al.  Unsupervised Representation Learning by Sorting Sequences , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[29]  L. Getoor,et al.  Link-Based Classification , 2003, Encyclopedia of Machine Learning and Data Mining.

[30]  James A. Reggia,et al.  Recurrent Neural Collective Classification , 2013, IEEE Transactions on Neural Networks and Learning Systems.

[31]  Max Welling,et al.  Semi-Supervised Classification with Graph Convolutional Networks , 2016, ICLR.

[32]  Austin R. Benson,et al.  Residual Correlation in Graph Neural Network Regression , 2020, KDD.

[33]  Jennifer Neville,et al.  Simple estimators for relational Bayesian classifiers , 2003, Third IEEE International Conference on Data Mining.

[34]  David M. Pennock,et al.  Towards Structural Logistic Regression: Combining Relational and Statistical Learning , 2002 .

[35]  Alexei A. Efros,et al.  Unsupervised Visual Representation Learning by Context Prediction , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[36]  Paolo Favaro,et al.  Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles , 2016, ECCV.

[37]  Andreas Loukas,et al.  Building powerful and equivariant graph neural networks with structural message-passing , 2020, NeurIPS.

[38]  Jennifer Neville,et al.  Why collective inference improves relational classification , 2004, KDD.

[39]  Balasubramaniam Srinivasan,et al.  On the Equivalence between Node Embeddings and Structural Graph Representations , 2019, ICLR 2020.

[40]  Ali Razavi,et al.  Data-Efficient Image Recognition with Contrastive Predictive Coding , 2019, ICML.

[41]  Paul N. Bennett,et al.  Overcoming Relational Learning Biases to Accurately Predict Preferences in Large Scale Networks , 2015, WWW.

[42]  Doina Precup,et al.  Break the Ceiling: Stronger Multi-scale Deep Graph Convolutional Networks , 2019, NeurIPS.

[43]  Jure Leskovec,et al.  How Powerful are Graph Neural Networks? , 2018, ICLR.

[44]  Vinayak A. Rao,et al.  Relational Pooling for Graph Representations , 2019, ICML.

[45]  Bruno Ribeiro,et al.  Are Graph Neural Networks Miscalibrated? , 2019, ArXiv.

[46]  Dominique Beaini,et al.  Directional Graph Networks , 2020, ICML.