Graph-CAT: Graph Co-Attention Networks via local and global attribute augmentations

Graph neural networks have achieved tremendous success in semi-supervised node classification. In this paper, we firstly analyse the propagation strategies in two milestone methods, Graph Convolutional Network (GCN) and Graph Attention Network (GAT), to reveal their underlying philosophies. According to our analysis, the propagations in GAT can be interpreted as learnable and asymmetric local attribute augmentations, while that of GCN can be interpreted as fixed and symmetric local attribute smoothing. Unfortunately, the local attribute augmentations in GAT is not adequate in certain circumstances, because the nodes tend to possess similar attributes in local neighbourhoods. With a toy experiment, we manage to demonstrate the necessity to incorporate global information. Therefore, we propose a novel Graph Co-ATtention Network (Graph-CAT), which performs both the local and global attribute augmentations based on two different yet complementary attention schemes. Extensive experiments in both the transductive and inductive tasks demonstrate the superiority of our Graph-CAT compared to the state-of-the-art methods. © 2021 Elsevier B.V. All rights reserved.

[1]  Jonathan Masci,et al.  Geometric Deep Learning on Graphs and Manifolds Using Mixture Model CNNs , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[2]  Chong Wang,et al.  Attention-based Graph Neural Network for Semi-supervised Learning , 2018, ArXiv.

[3]  Philip S. Yu,et al.  A Survey on Knowledge Graphs: Representation, Acquisition, and Applications , 2020, IEEE Transactions on Neural Networks and Learning Systems.

[4]  Chengqi Zhang,et al.  Learning Graph Embedding With Adversarial Training Methods , 2019, IEEE Transactions on Cybernetics.

[5]  Philip S. Yu,et al.  A Comprehensive Survey on Graph Neural Networks , 2019, IEEE Transactions on Neural Networks and Learning Systems.

[6]  Alán Aspuru-Guzik,et al.  Convolutional Networks on Graphs for Learning Molecular Fingerprints , 2015, NIPS.

[7]  Xueqi Cheng,et al.  Graph Wavelet Neural Network , 2019, ICLR.

[8]  Max Welling,et al.  Semi-Supervised Classification with Graph Convolutional Networks , 2016, ICLR.

[9]  Shirui Pan,et al.  DAGCN: Dual Attention Graph Convolutional Networks , 2019, 2019 International Joint Conference on Neural Networks (IJCNN).

[10]  Ruslan Salakhutdinov,et al.  Revisiting Semi-Supervised Learning with Graph Embeddings , 2016, ICML.

[11]  L. Getoor,et al.  Link-Based Classification , 2003, Encyclopedia of Machine Learning and Data Mining.

[12]  Wen-Chih Peng,et al.  Exploiting Centrality Information with Graph Convolutions for Network Representation Learning , 2019, 2019 IEEE 35th International Conference on Data Engineering (ICDE).

[13]  Mikhail Belkin,et al.  Manifold Regularization: A Geometric Framework for Learning from Labeled and Unlabeled Examples , 2006, J. Mach. Learn. Res..

[14]  Yun Fu,et al.  Adversarial Graph Embedding for Ensemble Clustering , 2019, IJCAI.

[15]  G. G. Stokes.  "J." , 1890, The New Yale Book of Quotations.

[16]  Gorjan Alagic,et al.  #p , 2019, Quantum information & computation.

[17]  Ken-ichi Kawarabayashi,et al.  Representation Learning on Graphs with Jumping Knowledge Networks , 2018, ICML.

[18]  Jun Wang,et al.  Adaptive Structural Fingerprints for Graph Attention Networks , 2020, ICLR.

[19]  Qimai Li,et al.  Label Efficient Semi-Supervised Learning via Graph Filtering , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[20]  Taiji Suzuki,et al.  Graph Neural Networks Exponentially Lose Expressive Power for Node Classification , 2019, ICLR.

[21]  Ryan A. Rossi,et al.  Graph Convolutional Networks with Motif-based Attention , 2019, CIKM.

[22]  Joan Bruna,et al.  Spectral Networks and Locally Connected Networks on Graphs , 2013, ICLR.

[23]  Jure Leskovec,et al.  Predicting multicellular function through multi-layer tissue networks , 2017, Bioinform..

[24]  Miss A.O. Penney.  (b) , 1974, The New Yale Book of Quotations.

[25]  Shuiwang Ji,et al.  Graph U-Nets , 2019, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[26]  Kevin Chen-Chuan Chang,et al.  Geom-GCN: Geometric Graph Convolutional Networks , 2020, ICLR.

[27]  Yanfang Ye,et al.  Heterogeneous Graph Attention Network , 2019, WWW.

[28]  Jure Leskovec,et al.  Improving Graph Attention Networks with Large Margin-based Constraints , 2019, ArXiv.

[29]  Estevam R. Hruschka,et al.  Toward an Architecture for Never-Ending Language Learning , 2010, AAAI.

[30]  Leman Akoglu,et al.  PairNorm: Tackling Oversmoothing in GNNs , 2020, ICLR.

[31]  Lukasz Kaiser,et al.  Attention is All you Need , 2017, NIPS.

[32]  Dacheng Tao,et al.  SPAGAN: Shortest Path Graph Attention Network , 2019, IJCAI.

[33]  Sepp Hochreiter,et al.  Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs) , 2015, ICLR.

[34]  Jure Leskovec,et al.  How Powerful are Graph Neural Networks? , 2018, ICLR.

[35]  Jure Leskovec,et al.  Inductive Representation Learning on Large Graphs , 2017, NIPS.

[36]  Pietro Liò,et al.  Graph Attention Networks , 2017, ICLR.

[37]  Pablo Tamayo,et al.  Gene set enrichment analysis: A knowledge-based approach for interpreting genome-wide expression profiles , 2005, Proceedings of the National Academy of Sciences of the United States of America.

[38]  Xiao-Ming Wu,et al.  Deeper Insights into Graph Convolutional Networks for Semi-Supervised Learning , 2018, AAAI.

[39]  Pietro Liò,et al.  Deep Graph Infomax , 2018, ICLR.

[40]  Zoubin Ghahramani,et al.  Combining active learning and semi-supervised learning using Gaussian fields and harmonic functions , 2003, ICML 2003.

[41]  Tingyang Xu,et al.  DropEdge: Towards Deep Graph Convolutional Networks on Node Classification , 2020, ICLR.

[42]  Hao Ma,et al.  GaAN: Gated Attention Networks for Learning on Large and Spatiotemporal Graphs , 2018, UAI.

[43]  Bernard Ghanem,et al.  DeepGCNs: Can GCNs Go As Deep As CNNs? , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[44]  Qianqian Wang,et al.  Multi-View Attribute Graph Convolution Networks for Clustering , 2020, IJCAI.

[45]  Jason Weston,et al.  Deep learning via semi-supervised embedding , 2008, ICML '08.

[46]  Katarzyna Musial,et al.  Multi-level Graph Convolutional Networks for Cross-platform Anchor Link Prediction , 2020, KDD.

[47]  Steven Skiena,et al.  DeepWalk: online learning of social representations , 2014, KDD.

[48]  Zhengyang Wang,et al.  Large-Scale Learnable Graph Convolutional Networks , 2018, KDD.

[49]  Xavier Bresson,et al.  Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering , 2016, NIPS.

[50]  Chen Gong,et al.  Hyperspectral Image Classification With Context-Aware Dynamic Graph Convolutional Network , 2019, IEEE Trans. Geosci. Remote. Sens..

[51]  Lise Getoor,et al.  Collective Classification in Network Data , 2008, AI Mag..

[52]  Xiaolong Li,et al.  GeniePath: Graph Neural Networks with Adaptive Receptive Paths , 2018, AAAI.

[53]  Chengqi Zhang,et al.  CFOND: Consensus Factorization for Co-Clustering Networked Data , 2019, IEEE Transactions on Knowledge and Data Engineering.

[54]  Stephan Günnemann,et al.  Predict then Propagate: Graph Neural Networks meet Personalized PageRank , 2018, ICLR.

[55]  Ah Chung Tsoi,et al.  The Graph Neural Network Model , 2009, IEEE Transactions on Neural Networks.

[56]  Paul Honeine,et al.  Bridging the Gap Between Spectral and Spatial Domains in Graph Neural Networks , 2020, ArXiv.