A Collective Learning Framework to Boost GNN Expressiveness

Graph Neural Networks (GNNs) have recently been used for node and graph classification tasks with great success, but GNNs model dependencies among the attributes of nearby neighboring nodes rather than dependencies among observed node labels. In this work, we consider the task of inductive node classification using GNNs in supervised and semi-supervised settings, with the goal of incorporating label dependencies. Because current GNNs are not universal (i.e., most-expressive) graph representations, we propose a general collective learning approach to increase the representation power of any existing GNN. Our framework combines ideas from collective classification with self-supervised learning, and uses a Monte Carlo approach to sampling embeddings for inductive learning across graphs. We evaluate performance on five real-world network datasets and demonstrate consistent, significant improvement in node classification accuracy, for a variety of state-of-the-art GNNs.

[1]  Bert Huang,et al.  Recurrent collective classification , 2017, Knowledge and Information Systems.

[2]  Paolo Favaro,et al.  Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles , 2016, ECCV.

[3]  Jennifer Neville,et al.  Simple estimators for relational Bayesian classifiers , 2003, Third IEEE International Conference on Data Mining.

[4]  Lise Getoor,et al.  Collective Classification in Network Data , 2008, AI Mag..

[5]  Ali Razavi,et al.  Data-Efficient Image Recognition with Contrastive Predictive Coding , 2019, ICML.

[6]  Paul N. Bennett,et al.  Overcoming Relational Learning Biases to Accurately Predict Preferences in Large Scale Networks , 2015, WWW.

[7]  Yaron Lipman,et al.  On the Universality of Invariant Networks , 2019, ICML.

[8]  Yoshua Bengio,et al.  GMNN: Graph Markov Neural Networks , 2019, ICML.

[9]  Jennifer Neville,et al.  Deep Collective Inference , 2017, AAAI.

[10]  Patrick Pérez,et al.  Boosting Few-Shot Visual Learning With Self-Supervision , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[11]  Doina Precup,et al.  Break the Ceiling: Stronger Multi-scale Deep Graph Convolutional Networks , 2019, NeurIPS.

[12]  Jennifer Neville,et al.  Learning relational probability trees , 2003, KDD '03.

[13]  David M. Pennock,et al.  Towards Structural Logistic Regression: Combining Relational and Statistical Learning , 2002 .

[14]  Alexei A. Efros,et al.  Unsupervised Visual Representation Learning by Context Prediction , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[15]  L. Getoor,et al.  Link-Based Classification , 2003, Encyclopedia of Machine Learning and Data Mining.

[16]  Svetha Venkatesh,et al.  Column Networks for Collective Classification , 2016, AAAI.

[17]  Joan Bruna,et al.  On the equivalence between graph isomorphism testing and function approximation with GNNs , 2019, NeurIPS.

[18]  Martin Grohe,et al.  Weisfeiler and Leman Go Neural: Higher-order Graph Neural Networks , 2018, AAAI.

[19]  Jure Leskovec,et al.  How Powerful are Graph Neural Networks? , 2018, ICLR.

[20]  Hans-Peter Kriegel,et al.  Protein function prediction via graph kernels , 2005, ISMB.

[21]  Bruno Ribeiro,et al.  Stochastic Gradient Descent for Relational Logistic Regression via Partial Network Crawls , 2017, ArXiv.

[22]  Vinayak A. Rao,et al.  Relational Pooling for Graph Representations , 2019, ICML.

[23]  Austin R. Benson,et al.  Residual Correlation in Graph Neural Network Regression , 2020, KDD.

[24]  Bruno Ribeiro,et al.  Are Graph Neural Networks Miscalibrated? , 2019, ArXiv.

[25]  James A. Reggia,et al.  Recurrent Neural Collective Classification , 2013, IEEE Transactions on Neural Networks and Learning Systems.

[26]  Jure Leskovec,et al.  Inductive Representation Learning on Large Graphs , 2017, NIPS.

[27]  Jennifer Neville,et al.  Why collective inference improves relational classification , 2004, KDD.

[28]  Max Welling,et al.  Semi-Supervised Classification with Graph Convolutional Networks , 2016, ICLR.

[29]  Ming-Wei Chang,et al.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding , 2019, NAACL.

[30]  H. Robbins.  A Stochastic Approximation Method , 1951 .

[31]  Ming-Hsuan Yang,et al.  Unsupervised Representation Learning by Sorting Sequences , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[32]  Jennifer Neville,et al.  Pseudolikelihood EM for Within-network Relational Learning , 2008, 2008 Eighth IEEE International Conference on Data Mining.

[33]  Jennifer Neville,et al.  Iterative Classification in Relational Data , 2000 .

[34]  Balasubramaniam Srinivasan,et al.  On the Equivalence between Node Embeddings and Structural Graph Representations , 2019, ICLR 2020.

[35]  Andrew McCallum,et al.  Introduction to Statistical Relational Learning , 2007 .

[36]  Martial Hebert,et al.  Shuffle and Learn: Unsupervised Learning Using Temporal Order Verification , 2016, ECCV.