Graph Convolution for Semi-Supervised Classification: Improved Linear Separability and Out-of-Distribution Generalization

Recently there has been increased interest in semisupervised classification in the presence of graphical information. A new class of learning models has emerged that relies, at its most basic level, on classifying the data after first applying a graph convolution. To understand the merits of this approach, we study the classification of a mixture of Gaussians, where the data corresponds to the node attributes of a stochastic block model. We show that graph convolution extends the regime in which the data is linearly separable by a factor of roughly 1/ √ D, where D is the expected degree of a node, as compared to the mixture model data on its own. Furthermore, we find that the linear classifier obtained by minimizing the crossentropy loss after the graph convolution generalizes to out-of-distribution data where the unseen data can have different intraand inter-class edge probabilities from the training data.

[1]  Stephan Günnemann,et al.  Predict then Propagate: Combining neural networks with personalized pagerank for classification on graphs , 2018, ICLR 2018.

[2]  Olgica Milenkovic,et al.  Joint Adaptive Feature Smoothing and Topology Extraction via Generalized PageRank GNNs , 2020, ArXiv.

[3]  Max Welling,et al.  Semi-Supervised Classification with Graph Convolutional Networks , 2016, ICLR.

[4]  Joshua T. Vogelstein,et al.  Covariate-assisted spectral clustering , 2014, Biometrika.

[5]  Jure Leskovec,et al.  Community Detection in Networks with Node Attributes , 2013, 2013 IEEE 13th International Conference on Data Mining.

[6]  Roman Vershynin,et al.  High-Dimensional Probability , 2018 .

[7]  Jure Leskovec,et al.  How Powerful are Graph Neural Networks? , 2018, ICLR.

[8]  Laurent Massoulié,et al.  Community detection thresholds and the weak Ramanujan property , 2013, STOC.

[9]  Olgica Milenkovic,et al.  Optimizing Generalized PageRank Methods for Seed-Expansion Community Detection , 2019, NeurIPS.

[10]  Stefanie Jegelka,et al.  Generalization and Representational Limits of Graph Neural Networks , 2020, ICML.

[11]  Hong Cheng,et al.  Clustering Large Attributed Graphs: A Balance between Structural and Attribute Similarities , 2011, TKDD.

[12]  Andrea Montanari,et al.  Asymptotic Mutual Information for the Two-Groups Stochastic Block Model , 2015, ArXiv.

[13]  Elchanan Mossel,et al.  A Proof of the Block Model Threshold Conjecture , 2013, Combinatorica.

[14]  Andrea Montanari,et al.  Semidefinite programs on sparse random graphs and their application to community detection , 2015, STOC.

[15]  Jon M. Kleinberg,et al.  Block models and personalized PageRank , 2016, Proceedings of the National Academy of Sciences.

[16]  Jure Leskovec,et al.  Graph Convolutional Neural Networks for Web-Scale Recommender Systems , 2018, KDD.

[17]  Samuel S. Schoenholz,et al.  Neural Message Passing for Quantum Chemistry , 2017, ICML.

[18]  Lawrence Carin,et al.  Stochastic Blockmodels meet Graph Neural Networks , 2019, ICML.

[19]  Kathryn B. Laskey,et al.  Stochastic blockmodels: First steps , 1983 .

[20]  Cristopher Moore,et al.  Asymptotic analysis of the stochastic block model for modular networks and its algorithmic applications , 2011, Physical review. E, Statistical, nonlinear, and soft matter physics.

[21]  Ernest Valveny,et al.  Graph embedding in vector spaces by node attribute statistics , 2012, Pattern Recognit..

[22]  Thomas Seidl,et al.  Spectral Subspace Clustering for Graphs with Feature Vectors , 2013, 2013 IEEE 13th International Conference on Data Mining.

[23]  William L. Hamilton.  Graph Representation Learning , 2020, Synthesis Lectures on Artificial Intelligence and Machine Learning.

[24]  Ah Chung Tsoi,et al.  The Graph Neural Network Model , 2009, IEEE Transactions on Neural Networks.

[25]  Weixiong Zhang,et al.  Graph Convolutional Networks Meet Markov Random Fields: Semi-Supervised Community Detection in Attribute Networks , 2019, AAAI.

[26]  Joan Bruna,et al.  Community Detection with Graph Neural Networks , 2017, 1705.08415.

[27]  Stephen P. Boyd,et al.  Graph Implementations for Nonsmooth Convex Programs , 2008, Recent Advances in Learning and Control.

[28]  Emmanuel Abbe,et al.  Exact Recovery in the Stochastic Block Model , 2014, IEEE Transactions on Information Theory.

[29]  Mark Heimann,et al.  Generalizing Graph Neural Networks Beyond Homophily , 2020, ArXiv.

[30]  Danai Koutra,et al.  Two Sides of the Same Coin: Heterophily and Oversmoothing in Graph Convolutional Neural Networks , 2021, ArXiv.

[31]  Emmanuel Viennet,et al.  Community Detection based on Structural and Attribute Similarities , 2012, ICDS 2012.

[32]  Andreas Loukas,et al.  What graph neural networks cannot learn: depth vs width , 2019, ICLR.

[33]  Emmanuel Abbe,et al.  Proof of the Achievability Conjectures for the General Stochastic Block Model , 2018 .

[34]  Elchanan Mossel,et al.  Consistency Thresholds for the Planted Bisection Model , 2014, STOC.

[35]  Jure Leskovec,et al.  Inductive Representation Learning on Large Graphs , 2017, NIPS.

[36]  Jan Eric Lenssen,et al.  Fast Graph Representation Learning with PyTorch Geometric , 2019, ArXiv.

[37]  Andreas Loukas,et al.  How hard is to distinguish graphs with graph neural networks? , 2020, NeurIPS.

[38]  O. Papaspiliopoulos.  High-Dimensional Probability: An Introduction with Applications in Data Science , 2020 .

[39]  Cristopher Moore,et al.  The Computer Science and Physics of Community Detection: Landscapes, Phase Transitions, and Hardness , 2017, Bull. EATCS.

[40]  Emmanuel Abbe,et al.  Community Detection in General Stochastic Block models: Fundamental Limits and Efficient Algorithms for Recovery , 2015, 2015 IEEE 56th Annual Symposium on Foundations of Computer Science.

[41]  Razvan Pascanu,et al.  Interaction Networks for Learning about Objects, Relations and Physics , 2016, NIPS.

[42]  Andrea Montanari,et al.  Contextual Stochastic Block Models , 2018, NeurIPS.

[43]  Emmanuel Abbe,et al.  Community detection and stochastic block models: recent developments , 2017, Found. Trends Commun. Inf. Theory.

[44]  Chen Lu,et al.  Contextual Stochastic Block Model: Sharp Thresholds and Contiguity , 2020, J. Mach. Learn. Res..

[45]  Laurent Massoulié,et al.  Non-backtracking Spectrum of Random Graphs: Community Detection and Non-regular Ramanujan Graphs , 2014, 2015 IEEE 56th Annual Symposium on Foundations of Computer Science.

[46]  Jess Banks,et al.  Information-theoretic thresholds for community detection in sparse networks , 2016, COLT.