CurvDrop: A Ricci Curvature Based Approach to Prevent Graph Neural Networks from Over-Smoothing and Over-Squashing

Graph neural networks (GNNs) are powerful models to handle graph data and can achieve state-of-the-art in many critical tasks including node classification and link prediction. However, existing graph neural networks still face both challenges of over-smoothing and over-squashing based on previous literature. To this end, we propose a new Curvature-based topology-aware Dropout sampling technique named CurvDrop, in which we integrate the Discrete Ricci Curvature into graph neural networks to enable more expressive graph models. Also, this work can improve graph neural networks by quantifying connections in graphs and using structural information such as community structures in graphs. As a result, our method can tackle the both challenges of over-smoothing and over-squashing with theoretical justification. Also, numerous experiments on public datasets show the effectiveness and robustness of our proposed method. The code and data are released in https://github.com/liu-yang-maker/Curvature-based-Dropout.

[1]  Yongchao Liu,et al.  Heterogeneous Graph Neural Architecture Search , 2021, 2021 IEEE International Conference on Data Mining (ICDM).

[2]  Francesco Di Giovanni,et al.  Understanding over-squashing and bottlenecks on graphs via curvature , 2021, ICLR.

[3]  Guohua Wu,et al.  Curvature Graph Neural Network , 2021, Inf. Sci..

[4]  Marco Serafini,et al.  Scalable Graph Neural Network Training , 2021, ACM SIGOPS Oper. Syst. Rev..

[5]  Jiuxin Cao,et al.  Multi-level Hyperedge Distillation for Social Linking Prediction on Sparsely Observed Networks , 2021, WWW.

[6]  V. Prasanna,et al.  Deep Graph Neural Networks with Shallow Subgraph Samplers , 2020, ArXiv.

[7]  Chuan Zhou,et al.  Graph Geometry Interaction Learning , 2020, NeurIPS.

[8]  Bo Liu,et al.  Heterogeneous Hypergraph Embedding for Graph Classification , 2020, WSDM.

[9]  J. Leskovec,et al.  Multi-hop Attention Graph Neural Networks , 2020, IJCAI.

[10]  Rana Forsati,et al.  Minimal Variance Sampling with Provable Guarantees for Fast Training of Graph Neural Networks , 2020, KDD.

[11]  Eran Yahav,et al.  On the Bottleneck of Graph Neural Networks and its Practical Implications , 2020, ICLR.

[12]  Jun Wang,et al.  Adaptive Structural Fingerprints for Graph Attention Networks , 2020, ICLR.

[13]  Chao Chen,et al.  Curvature Graph Network , 2020, ICLR.

[14]  Weijing Shi,et al.  Point-GNN: Graph Neural Network for 3D Object Detection in a Point Cloud , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[15]  Linyuan Lu,et al.  On the sum of Ricci-curvatures for weighted graphs , 2020, Pure and Applied Mathematics Quarterly.

[16]  Xu Sun,et al.  Measuring and Relieving the Over-smoothing Problem for Graph Neural Networks from the Topological View , 2019, AAAI.

[17]  Farhad Kamangar,et al.  Non-smooth Optimization over Stiefel Manifolds with Applications to Dimensionality Reduction and Graph Clustering , 2019, IJCAI.

[18]  Junzhou Huang,et al.  DropEdge: Towards Deep Graph Convolutional Networks on Node Classification , 2019, ICLR.

[19]  Feng Luo,et al.  Community Detection on Networks with Ricci Flow , 2019, Scientific Reports.

[20]  Dan Goldwasser,et al.  Encoding Social Information with Graph Convolutional Networks forPolitical Perspective Detection in News Media , 2019, ACL.

[21]  Mark Kempton,et al.  Large scale Ricci curvature on graphs , 2019, Calculus of Variations and Partial Differential Equations.

[22]  Taiji Suzuki,et al.  On Asymptotic Behaviors of Graph CNNs from Dynamical Systems Perspective , 2019, ArXiv.

[23]  Xu Chen,et al.  Actional-Structural Graph Convolutional Networks for Skeleton-Based Action Recognition , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[24]  Shadi Albarqouni,et al.  InceptionGCN: Receptive Field Aware Graph Convolutional Network for Disease Prediction , 2019, IPMI.

[25]  Stephan Günnemann,et al.  Predict then Propagate: Graph Neural Networks meet Personalized PageRank , 2018, ICLR.

[26]  Junzhou Huang,et al.  Adaptive Sampling Towards Fast Graph Representation Learning , 2018, NeurIPS.

[27]  Ken-ichi Kawarabayashi,et al.  Representation Learning on Graphs with Jumping Knowledge Networks , 2018, ICML.

[28]  Stephan Günnemann,et al.  NetGAN: Generating Graphs via Random Walks , 2018, ICML.

[29]  Cao Xiao,et al.  FastGCN: Fast Learning with Graph Convolutional Networks via Importance Sampling , 2018, ICLR.

[30]  Xiao-Ming Wu,et al.  Deeper Insights into Graph Convolutional Networks for Semi-Supervised Learning , 2018, AAAI.

[31]  Alex Fout,et al.  Protein Interface Prediction using Graph Convolutional Networks , 2017, NIPS.

[32]  Jure Leskovec,et al.  Inductive Representation Learning on Large Graphs , 2017, NIPS.

[33]  Samuel S. Schoenholz,et al.  Neural Message Passing for Quantum Chemistry , 2017, ICML.

[34]  S. Yau,et al.  Ricci curvature of graphs , 2011 .

[35]  Y. Ollivier.  Ricci curvature of Markov chains on metric spaces , 2007, math/0701886.

[36]  Jilong Wang,et al.  Graph Stochastic Neural Networks for Semi-supervised Learning , 2020, NeurIPS.

[37]  S. Yau,et al.  Ricci curvature and eigenvalue estimate on locally finite graphs , 2010 .

[38]  J. Delvenne,et al.  Random walks on graphs , 2004 .