Two Sides of the Same Coin: Heterophily and Oversmoothing in Graph Convolutional Neural Networks

In node classification tasks, graph convolutional neural networks (GCNs) have demonstrated competitive performance over traditional methods on diverse graph data. However, it is known that the performance of GCNs degrades with increasing number of layers (oversmoothing problem) and recent studies have also shown that GCNs may perform worse in heterophilous graphs, where neighboring nodes tend to belong to different classes (heterophily problem). These two problems are usually viewed as unrelated, and thus are studied independently, often at the graph filter level from a spectral perspective.We are the first to take a unified perspective to jointly explain the oversmoothing and heterophily problems at the node level. Specifically, we profile the nodes via two quantitative metrics: the relative degree of a node (compared to its neighbors) and the node-level heterophily. Our theory shows that the interplay of these two profiling metrics defines three cases of node behaviors, which explain the oversmoothing and heterophily problems jointly and can predict the performance of GCNs. Based on insights from our theory, we show theoretically and empirically the effectiveness of two strategies: structure-based edge correction, which learns corrected edge weights from structural properties (i.e., degrees), and feature-based edge correction, which learns signed edge weights from node features. Compared to other approaches, which tend to handle well either heterophily or oversmoothing, we show that our model, GGCN, which incorporates the two strategies performs well in both problems. We provide a longer version of this paper in [1] and codes on https://github.com/YujunYan/Heterophily_and_oversmoothing.

[1]  Jie Chen,et al.  Memory-Based Message Passing: Decoupling the Message for Propagation from Discrimination , 2022, ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[2]  Michael T. Schaub,et al.  How does Heterophily Impact the Robustness of Graph Neural Networks?: Theoretical Connections and Practical Implications , 2021, KDD.

[3]  Jiliang Tang,et al.  Is Homophily a Necessity for Graph Neural Networks? , 2021, ICLR.

[4]  Shuiwang Ji,et al.  Non-Local Graph Neural Networks , 2020, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[5]  Omkar Bhalerao,et al.  Large Scale Learning on Non-Homophilous Graphs: New Benchmarks and Strong Simple Methods , 2021, NeurIPS.

[6]  Doina Precup,et al.  Is Heterophily A Real Nightmare For Graph Neural Networks To Do Node Classification? , 2021, ArXiv.

[7]  K. Fountoulakis,et al.  Graph Convolution for Semi-Supervised Classification: Improved Linear Separability and Out-of-Distribution Generalization , 2021, ICML.

[8]  Xiao Wang,et al.  Beyond Low-frequency Information in Graph Convolutional Networks , 2021, AAAI.

[9]  Ryan A. Rossi,et al.  Graph Neural Networks with Heterophily , 2020, AAAI.

[10]  Olgica Milenkovic,et al.  Adaptive Universal Generalized PageRank Graph Neural Network , 2020, ICLR.

[11]  Bryan Hooi,et al.  Understanding and Resolving Performance Degradation in Deep Graph Convolutional Networks , 2020, CIKM.

[12]  Rik Sarkar,et al.  Multi-scale Attributed Node Embedding , 2019, J. Complex Networks.

[13]  Yaliang Li,et al.  Simple and Deep Graph Convolutional Networks , 2020, ICML.

[14]  L. Akoglu,et al.  Beyond Homophily in Graph Neural Networks: Current Limitations and Effective Designs , 2020, NeurIPS.

[15]  Razvan Pascanu,et al.  Pointer Graph Networks , 2020, NeurIPS.

[16]  Kevin Swersky,et al.  Neural Execution Engines: Learning to Execute Subroutines , 2020, NeurIPS.

[17]  Kevin Chen-Chuan Chang,et al.  Geom-GCN: Geometric Graph Convolutional Networks , 2020, ICLR.

[18]  L. Akoglu,et al.  PairNorm: Tackling Oversmoothing in GNNs , 2019, ICLR.

[19]  Xu Sun,et al.  Measuring and Relieving the Over-smoothing Problem for Graph Neural Networks from the Topological View , 2019, AAAI.

[20]  Ryan A. Rossi,et al.  On Proximity and Structural Role-based Embeddings in Networks , 2019, ACM Trans. Knowl. Discov. Data.

[21]  Junzhou Huang,et al.  DropEdge: Towards Deep Graph Convolutional Networks on Node Classification , 2019, ICLR.

[22]  Daniel Tarlow,et al.  Learning Execution through Neural Code Fusion , 2019, ICLR.

[23]  Taiji Suzuki,et al.  Graph Neural Networks Exponentially Lose Expressive Power for Node Classification , 2019, ICLR.

[24]  Jure Leskovec,et al.  Improving Graph Attention Networks with Large Margin-based Constraints , 2019, ArXiv.

[25]  Danai Koutra,et al.  GroupINN: Grouping-based Interpretable Neural Network for Classification of Limited, Noisy Brain Data , 2019, KDD.

[26]  Dan Goldwasser,et al.  Encoding Social Information with Graph Convolutional Networks forPolitical Perspective Detection in News Media , 2019, ACL.

[27]  Bernard Ghanem,et al.  DeepGCNs: Can GCNs Go As Deep As CNNs? , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[28]  Kilian Q. Weinberger,et al.  Simplifying Graph Convolutional Networks , 2019, ICML.

[29]  Jure Leskovec,et al.  How Powerful are Graph Neural Networks? , 2018, ICLR.

[30]  Stephan Günnemann,et al.  Pitfalls of Graph Neural Network Evaluation , 2018, ArXiv.

[31]  Ken-ichi Kawarabayashi,et al.  Representation Learning on Graphs with Jumping Knowledge Networks , 2018, ICML.

[32]  Xiao-Ming Wu,et al.  Deeper Insights into Graph Convolutional Networks for Semi-Supervised Learning , 2018, AAAI.

[33]  Joan Bruna,et al.  Few-Shot Learning with Graph Neural Networks , 2017, ICLR.

[34]  Marc Brockschmidt,et al.  Learning to Represent Programs with Graphs , 2017, ICLR.

[35]  Pietro Liò,et al.  Graph Attention Networks , 2017, ICLR.

[36]  Stephan Günnemann,et al.  Deep Gaussian Embedding of Graphs: Unsupervised Inductive Learning via Ranking , 2017, ICLR.

[37]  Jure Leskovec,et al.  Inductive Representation Learning on Large Graphs , 2017, NIPS.

[38]  Samuel S. Schoenholz,et al.  Neural Message Passing for Quantum Chemistry , 2017, ICML.

[39]  Max Welling,et al.  Semi-Supervised Classification with Graph Convolutional Networks , 2016, ICLR.

[40]  Xavier Bresson,et al.  Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering , 2016, NIPS.

[41]  Danai Koutra,et al.  Linearized and Single-Pass Belief Propagation , 2014, Proc. VLDB Endow..

[42]  Lise Getoor,et al.  Query-driven Active Surveying for Collective Classification , 2012 .

[43]  Jimeng Sun,et al.  Social influence analysis in large-scale networks , 2009, KDD.

[44]  Lise Getoor,et al.  Collective Classification in Network Data , 2008, AI Mag..

[45]  Christos Faloutsos,et al.  Netprobe: a fast and scalable system for fraud detection in online auction networks , 2007, WWW '07.

[46]  M. Newman.  Assortative mixing in networks. , 2002, Physical review letters.